Zip corruption in initial commit

[dbaynard]

Hello,

I'm seeing zip file corruption, from (I believe) the following.

org-formation-cli/src/util/initial-commit-util.ts

Lines 49 to 64 in 416dbaa

	static async parameterizeAndUpload(extractedTemplate: ExtractedTemplate, params: Record<string, any>, template: DefaultTemplate, stateBucketName: string, s3credentials?: CredentialsOptions): Promise<void> {
	const { definition: templateDefinition, tempDir } = extractedTemplate;
	for (const declaredParam of templateDefinition.parameters) {
	if ((declaredParam.required) && (params[declaredParam.name] === undefined)) {
	throw new Error(`template zip declares required parameter ${declaredParam.name}, which is not provided.`);
	}
	}
	const archive = archiver('zip');
	const upload = uploadStream(stateBucketName, 'initial-commit.zip', s3credentials);
	archive.pipe(upload.writeStream);
	this.replaceFiles(tempDir, params, archive, '');
	const renderedTemplateContents = renderString(template.template, params);
	archive.append(renderedTemplateContents, { name: templateDefinition.organizationFilePath });
	await archive.finalize();
	await upload.promise;
	}

Specifically, I created a zip that includes a lockfile (a pnpm-lock.yaml file). That lockfile is correct in the zip that I supply to --template-package-url, but the zip file that then gets uploaded to the s3 bucket is not correct — two out of two times, so far. I believe it's coming from org-formation — here's the state of the file at each location.

Filesystem -> localhost (caddy) -> S3 -> CodeCommit
     ✓                ✓            ✗        ✗

Specifically, it appears as though the file has been split into two chunks as part of the streaming, and the chunks have been reassembled in the incorrect order.

Original:

• beginning
• middle
• middle
• end

Bucket version

• middle
• end
• beginning
• middle

I wondered whether it might be to do with this, perhaps — archiverjs/node-archiver#161 — but I've no idea. There's also archiverjs/node-archiver#32 (comment) — I'm not familiar enough with how the backend works, to know whether the issue here makes sense (not waiting for the stream close event before calling archive.finalize.)

This is acutely terrible for yaml files because a truncated yaml file is often a valid yaml file. But that's not an org-formation issue.

This didn't block the configuration of the bootstrap build pipeline, so I was able to push the correct repo… but I don't like behaviour like this, as it means the things going on don't match my expectation. I'd be grateful to know what the maintainers think.

Thanks!