Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-deterministic build #13672

Closed
7 tasks done
ttionya opened this issue Jun 29, 2023 · 42 comments
Closed
7 tasks done

Non-deterministic build #13672

ttionya opened this issue Jun 29, 2023 · 42 comments
Labels
bug: upstream Bug in a dependency of Vite feat: commonjs @rollup/plugin-commonjs issue p3-minor-bug An edge case that only affects very specific usage (priority)

Comments

@ttionya
Copy link

ttionya commented Jun 29, 2023

Describe the bug

Executing builds multiple times with the same code generates different hashes.

Reproduction

https://github.com/ttionya/vite-non-deterministic-build-repro

Steps to reproduce

Clone the aforementioned repro, install dependencies, and execute npm run build multiple times. Verify the hash of the vendor file.

System Info

System:                                                    
    OS: Windows 10 10.0.19045                                
    CPU: (8) x64 Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz   
    Memory: 3.92 GB / 15.76 GB                               
  Binaries:
    Node: 14.21.0
    npm: 9.6.7
  Browsers:
    Edge: Spartan (44.19041.1266.0), Chromium (114.0.1823.58)
    Internet Explorer: 11.0.19041.1566
  npmPackages:
    @vitejs/plugin-legacy: ^4.0.0 => 4.0.1 
    @vitejs/plugin-vue2: ^2.2.0 => 2.2.0 
    @vitejs/plugin-vue2-jsx: ^1.1.0 => 1.1.0 
    vite: ^4.3.8 => 4.3.8

Used Package Manager

npm

Logs

No response

Validations

@ttionya
Copy link
Author

ttionya commented Jun 29, 2023

I'm aware of two similar issues. One is #11911, but it is unrelated to this issue. The other one #13071 might be relevant, but even after commenting on it 10 days ago, I haven't received any response. Therefore, I have created a new issue and provided a reproducible repository.

@ttionya
Copy link
Author

ttionya commented Jun 29, 2023

I have written a plugin that prints all the parameters in the transform hook. During two consecutive builds, although the order of their outputs is different, I found that there is only one instance where the content of the output differs after adjusting the order.

plugins: [
    function() {
      return {
        name: 'test',
        transform: (...args) => {
          console.log(args)
        }
      }
    }()
]

I have removed a significant amount of irrelevant code, so the hash of the artifact is different from the one in the repro.

First build:

[
  'export { __moduleExports as default } from "C:/Users/admin/Projects/My/vite-non-deterministic-build-repro/node_modules/dayjs/dayjs.min.js";',
  '\x00C:/Users/admin/Projects/My/vite-non-deterministic-build-repro/node_modules/dayjs/dayjs.min.js?commonjs-proxy'
]

[2mdist/[22m�[36massets/index-e2988d1d.js   [39m�[1m�[2m 1.10 kB�[22m�[1m�[22m�[2m  gzip:  0.63 kB�[22m
[2mdist/[22m�[36massets/vendor-3bedaa96.js  [39m�[1m�[2m81.22 kB�[22m�[1m�[22m�[2m  gzip: 32.73 kB�[22m
[32m✓ built in 1.97s�[39m

Second build:

[
  'import { getDefaultExportFromCjs } from "\x00commonjsHelpers.js";\n' +
    'import { __require as requireDayjs_min } from "C:/Users/admin/Projects/My/vite-non-deterministic-build-repro/node_modules/dayjs/dayjs.min.js";\n' +
    'var dayjs_minExports = requireDayjs_min();\n' +
    'export { dayjs_minExports as __moduleExports };export default /*@__PURE__*/getDefaultExportFromCjs(dayjs_minExports);',
  '\x00C:/Users/admin/Projects/My/vite-non-deterministic-build-repro/node_modules/dayjs/dayjs.min.js?commonjs-es-import'
]

[2mdist/[22m�[36massets/index-73275ff8.js   [39m�[1m�[2m 1.10 kB�[22m�[1m�[22m�[2m  gzip:  0.63 kB�[22m
[2mdist/[22m�[36massets/vendor-2f895bab.js  [39m�[1m�[2m81.26 kB�[22m�[1m�[22m�[2m  gzip: 32.75 kB�[22m
[32m✓ built in 2.97s�[39m

@iamriajul
Copy link

Facing the same issue! waiting for a fix.

@derheld42
Copy link

I'm seeing this issue as well.

@jmsmtn
Copy link

jmsmtn commented Oct 6, 2023

The workaround mentioned in #10506, setting maxParallelFileOps: 1, has seemed to resolve this for me.

@derheld42
Copy link

The workaround in #10506 doesn't seem to work for me. I created a repo to try testing the workaround using the basic out-of-the-box base sveltekit app. Test repo is here https://github.com/derheld42/repro-vite-10506

@bluwy
Copy link
Member

bluwy commented Oct 29, 2023

Seems like dayjs is seen as commonjs, and there's a related upstream bug: rollup/plugins#1425

Thanks for the simple repro (and sorry for not responding in the other issue). Got a few issues to juggle, but I'll try to check this deeper now to see if there's something that can be fixed.

Also as explained in the upstream issue, the workaround would be:

export default defineConfig({
  build: {
    commonjsOptions: {
      strictRequires: true
    }
  }
})

@bluwy bluwy added bug: upstream Bug in a dependency of Vite p3-minor-bug An edge case that only affects very specific usage (priority) feat: commonjs @rollup/plugin-commonjs issue and removed pending triage labels Oct 29, 2023
derheld42 added a commit to derheld42/repro-vite-10506 that referenced this issue Oct 29, 2023
vitejs/vite#13672 (comment)

Still fails producing a deterministic build:
npm run build
mv build build1
npm run build
diff -qr build build1
@derheld42
Copy link

I tried the workaround in the above comment and it didn't create a deterministic build for me. I also tested it with derheld42/repro-vite-10506@a31754f and it doesn't produce a deterministic build.

@bluwy
Copy link
Member

bluwy commented Oct 30, 2023

@derheld42 that seems to be a SvelteKit specific issue: sveltejs/kit#8948

You can fix it with:

const config = {
	kit: {
		version: {
			name: 'test' // something stable instead of default `Date.now()`
		}
	}
};

@ttionya
Copy link
Author

ttionya commented Oct 30, 2023

After adding strictRequires option for the repro, it consistently generates a deterministic hash on every build, which looks great.

After adding the strictRequires option to my larger project, the significant build differences that existed previously have disappeared, except for the polyfill-legacy-[hash].js (generated by @vitejs/plugin-legacy). I spent some time trying to find a way to reproduce the instability in building the polyfill, but unfortunately, I failed. It appears that it doesn't stem from a singular syntax, so despite numerous attempts, I couldn't yield any results. I'm aware that a simple description won't be of any help in troubleshooting the issue, so this is just a record for now. Perhaps, if I manage to find a way to reproduce it in the future, I'll return to provide an update.

@derheld42
Copy link

derheld42 commented Oct 30, 2023

@derheld42 that seems to be a SvelteKit specific issue: sveltejs/kit#8948

You can fix it with:

const config = {
	kit: {
		version: {
			name: 'test' // something stable instead of default `Date.now()`
		}
	}
};

Yes - that worked to make the built files deterministic - thank you!

@iamriajul
Copy link

I'm also facing the same issue, strictRequires didn't work for me.

Here is my dependencies, can anyone give me any hint which is causing this.

  "devDependencies": {
    "@builder.io/qwik": "^1.2.15",
    "@builder.io/qwik-city": "^1.2.15",
    "@fastify/compress": "^6.2.1",
    "@fastify/http-proxy": "^9.2.1",
    "@fastify/static": "^6.10.1",
    "@fullhuman/postcss-purgecss": "^5.0.0",
    "@types/eslint": "8.44.1",
    "@types/js-cookie": "^3.0.4",
    "@types/lodash": "^4.14.199",
    "@types/node": "^20.4.5",
    "@typescript-eslint/eslint-plugin": "6.2.0",
    "@typescript-eslint/parser": "6.2.0",
    "autoprefixer": "^10.4.14",
    "country-flag-icons": "^1.5.7",
    "cssnano": "^6.0.1",
    "eslint": "8.45.0",
    "eslint-plugin-qwik": "^1.2.12",
    "fastify": "^4.17.0",
    "fastify-plugin": "^4.5.0",
    "postcss": "8.4.27",
    "postcss-import": "^15.1.0",
    "postcss-nested": "^6.0.1",
    "prettier": "3.0.0",
    "purgecss": "^5.0.0",
    "tailwindcss": "3.3.3",
    "tw-elements": "1.0.0-alpha13",
    "typescript": "5.1.6",
    "undici": "5.22.1",
    "vite": "4.4.7",
    "vite-plugin-static-copy": "^0.17.0",
    "vite-plugin-thumbhash-base64": "^0.0.2",
    "vite-tsconfig-paths": "4.2.0"
  },
  "dependencies": {
    "@fontsource-variable/dm-sans": "^5.0.3",
    "@fontsource/dm-sans": "^5.0.14",
    "@iconscout/unicons": "^4.0.8",
    "@modular-forms/qwik": "^0.21.0",
    "@svgx/vite-plugin-qwik": "^1.0.1",
    "animate.css": "^4.1.1",
    "date-fns": "^2.30.0",
    "dayjs": "^1.11.10",
    "flowbite": "^1.8.1",
    "js-base64": "^3.7.5",
    "js-cookie": "^3.0.5",
    "libphonenumber-js": "^1.10.47",
    "lodash": "^4.17.21",
    "moick-qwik-fixed": "^1.0.6",
    "qwik-content-loader": "^0.0.2",
    "qwik-scroll-to-top": "^0.0.7",
    "qwik-select-fixed": "^0.0.4",
    "valibot": "^0.19.0",
    "zod": "^3.22.4"
  }

@mikethea1
Copy link

We're having this issue as well (even with the workarounds I've seen listed) which is very frustrating. Will this be addressed soon?

Here is our vite.config.js; I welcome any suggestions for additional things to try:


import { defineConfig } from 'vite';
import laravel from 'laravel-vite-plugin';
import react from '@vitejs/plugin-react';

export default defineConfig({
    plugins: [
        laravel({
            input: ['resources/js/app.tsx'],
            refresh: true,
        }),
        react(),
    ],
    build: {
        rollupOptions: {
            // Suggested here for DETERMINISTIC BUILD: https://github.com/vitejs/vite/issues/10506 (from https://github.com/vitejs/vite/issues/13672)
            maxParallelFileOps: 1,
        },
        // Suggested here for DETERMINISTIC BUILD: https://github.com/vitejs/vite/issues/13672#issuecomment-1784110536
        commonjsOptions: {
            strictRequires: true
        }
    }
});

@rkhaslarov
Copy link

facing the issue with the react component that is not chaging at all, but hash is new all the time

@ilyagr
Copy link

ilyagr commented Feb 23, 2024

I have a pretty detailed example in #15994.

It seems that the suggestions in this thread helped. I'm guessing that it was the CommonJS strictRequires: true that helped, mainly because my project is using CodemIrror5 which, AFAIK, is a complex CommonJS package.

@niksy
Copy link
Contributor

niksy commented Mar 8, 2024

Solutions mentioned here didn’t work for me. Each consecutive build without changes to source code produces new hashes for the same chunks and in some cases can lead to cascading cache invalidation issues which is really bad for performance.

@mikethea1
Copy link

Each consecutive build without changes to source code produces new hashes for the same chunks and in some cases can lead to cascading cache invalidation issues which is really bad for performance

We're seeing the exact same thing. Why wouldn't the hashes just be based solely on the content of the file being hashed? Or is the issue that there is non-determinism in the bundling process which makes for different content every time?

@niksy
Copy link
Contributor

niksy commented Mar 8, 2024

Or is the issue that there is non-determinism in the bundling process which makes for different content every time?

I think this is the main reason for this, and it seems like it should be fixed with build.commonjsOptions.strictRequires = true, but it isn’t.

@msorens
Copy link

msorens commented Mar 22, 2024

I have tried adding both workarounds to my code (strictRequires: true and maxParallelFileOps: 1) but neither helped; I am still seeing the same problem as others have noted here.

But here is a fun fact that may help someone isolate the cause further:
(a) when I run vite build locally on my Mac then it works correctly--if I run it a second time, the hash is always the same!
(b) when I run in CI (we are using https://concourse-ci.org/) then the problem always occurs (but just for my JS files; I am getting constant hashes for my CSS files). The CSS files are quite a bit smaller than the JS files, which perhaps makes a difference.
vite: version 4.5.2
rollup: version 3.29.4

@geoffharcourt
Copy link

geoffharcourt commented Apr 4, 2024

@msorens we are seeing the same behavior you are: deterministic results locally on a Mac, different hashes in CI, but only non-deterministic for JS outputs, CSS is consistent.

For us the issue turned out to be the Sentry Vite plugin's default behavior, which injects a helper method that makes the build result non-deterministic.

@devsuperfrete
Copy link

Analyzing the code a bit, I understood that this happens because the name of the index file changes, and it is imported in almost all other .js files. So, for the hash to work correctly, I imagine that this needs to be ignored in some way.

@rkhaslarov
Copy link

@devsuperfrete yes also mentioned that, if something is change at any place all hashes are affected, the reason is that everything included into index

@yoyo837
Copy link
Contributor

yoyo837 commented Apr 7, 2024

Same here. strictRequires: true doesn't work for us. And we urgently hope this issue can be resolved ASAP.

I found that the export order is different every time, which causes the difference, but I don't know how to fix it.

image

@msorens
Copy link

msorens commented Apr 9, 2024

@geoffharcourt For us the issue turned out to be the Sentry Vite plugin's default behavior, which injects a helper method that makes the build result non-deterministic.

Could you share some more info on this? Looks like my code base uses that plugin. How/where did you find info about that injected helper method?

@geoffharcourt
Copy link

Hi @msorens this was painful for us, but we found this option and set it to false: https://www.npmjs.com/package/@sentry/vite-plugin#releaseinject

@msorens
Copy link

msorens commented Apr 9, 2024

Thanks, @geoffharcourt , I will give that a try. Not terribly familiar with Sentry: what is the impact of setting that option to false? Seems like it might not be able to manage builds as effectively...?

@geoffharcourt
Copy link

Thanks, @geoffharcourt , I will give that a try. Not terribly familiar with Sentry: what is the impact of setting that option to false? Seems like it might not be able to manage builds as effectively...?

It helps Sentry link exceptions to source. We found in our app that Sentry was still able to do it well without the helper injected.

@dasJ
Copy link

dasJ commented Apr 10, 2024

We had the same issue with SvelteKit. They generate a version that is based on the current timestamp (which is something they discourage in their docs).
Overriding the version as explained in their docs fixed the issue for us, even with the Sentry plugin's default config.

@shankerwangmiao
Copy link
Contributor

After adding strictRequires option for the repro, it consistently generates a deterministic hash on every build, which looks great.

After adding the strictRequires option to my larger project, the significant build differences that existed previously have disappeared, except for the polyfill-legacy-[hash].js (generated by @vitejs/plugin-legacy). I spent some time trying to find a way to reproduce the instability in building the polyfill, but unfortunately, I failed. It appears that it doesn't stem from a singular syntax, so despite numerous attempts, I couldn't yield any results. I'm aware that a simple description won't be of any help in troubleshooting the issue, so this is just a record for now. Perhaps, if I manage to find a way to reproduce it in the future, I'll return to provide an update.

Hi, I also met this where the generated polyfill bundle is not stable. I've opened the #16505, hopefully solving your problem.

@ouksal
Copy link

ouksal commented Apr 30, 2024

@derheld42 that seems to be a SvelteKit specific issue: sveltejs/kit#8948
You can fix it with:

const config = {
	kit: {
		version: {
			name: 'test' // something stable instead of default `Date.now()`
		}
	}
};

Yes - that worked to make the built files deterministic - thank you!

Worked. Thank you!

@yoyo837
Copy link
Contributor

yoyo837 commented Jun 17, 2024

@msorens #16566 doesn't look like it can fix this issue?

@shankerwangmiao
Copy link
Contributor

@msorens #16566 doesn't look like it can fix this issue?

In my project, the changes between repeated builds are brought by the generated legacy-pollyfills, where the order of the polyfilles differs. This PR addresses this issue by giving a deterministic order.

@msorens
Copy link

msorens commented Jun 17, 2024

Thanks, @yoyo837 and @shankerwangmiao . Looking in the release notes I am not finding that issue listed. Is it safe to say it is in 5.3.0 or later...?

@shankerwangmiao
Copy link
Contributor

Thanks, @yoyo837 and @shankerwangmiao . Looking in the release notes I am not finding that issue listed. Is it safe to say it is in 5.3.0 or later...?

No, it is in @vitejs/plugin-legacy 5.4.1

@msorens
Copy link

msorens commented Jun 18, 2024

Ah, ok. I am not using that, so would not help my situation. Thanks, though.

@itamar-smirra-port
Copy link

I have the same problem with monaco editor.
vite version - 5.3.1
vite.config:

{
		plugins: [
			react(),
			tsconfigPaths({root: './'}),
			viteCommonjs(),
			sentryVitePlugin({
				authToken: process.env.SENTRY_AUTH_TOKEN,
				org: 'getport',
				project: 'frontend',
				sourcemaps: {
					filesToDeleteAfterUpload: ['./dist/**/*.js.map'],
				},
				release: {
					name: process.env.VERSION,
					create: true,
					inject: false,
				},
			}),
			svgr({
				svgrOptions: {
					icon: true,
				},
			}),
		],
		define: {
			'process.env': {},
		},
		build: {
			// Workaround to ensure content hash is the same between deployments
			// https://github.com/vitejs/vite/issues/13672
			rollupOptions: {
				...build?.rollupOptions,
				output: {
					...build?.rollupOptions?.output,
					chunkFileNames: `assets/[name]-chunk-[hash].js`,
					 manualChunks: {
						monaco: ['monaco-editor', 'protobufjs', 'monaco-yaml'],
					}
				},
				maxParallelFileOps: 1,
			},
			sourcemap: true,
			commonjsOptions: {
				// Workaround to ensure content hash is the same between deployments
				// https://github.com/vitejs/vite/issues/13672
				strictRequires: true,
				include: [/common-consts/, /node_modules/],
			},
		},
		optimizeDeps: {
			...optimizeDeps,
			include: [...(optimizeDeps?.include ?? []), 'tailwind.config.js', '@port-labs/common-consts', '**/**.cy.ts'],
			esbuildOptions: {
				plugins: [reactVirtualizedResolver],
			},
		}
	}

@leonardofrmello
Copy link

Obrigado,@geoffharcourt, vou tentar. Não estou muito familiarizado com o Sentry: qual é o impacto de definir essa opção como false? Parece que ele pode não conseguir gerenciar builds tão efetivamente...?

Ajuda o Sentry a vincular exceções à fonte. Descobrimos em nosso aplicativo que o Sentry ainda conseguia fazer isso bem sem o helper injetado.

Thanks, @geoffharcourt , I will give that a try. Not terribly familiar with Sentry: what is the impact of setting that option to false? Seems like it might not be able to manage builds as effectively...?

It helps Sentry link exceptions to source. We found in our app that Sentry was still able to do it well without the helper injected.

@geoffharcourt I set it to false, but that didn't solve the problem of creating different hashes in each new version.
Locally this doesn't happen, only when I generate a version through circleci.
what's your vite version?

@GWesley
Copy link

GWesley commented Sep 25, 2024

build: {
rollupOptions: {
maxParallelFileOps: 1,
},
commonjsOptions: {
strictRequires: true,
},
},

@bluwy
Copy link
Member

bluwy commented Oct 1, 2024

The Vite v6 beta bumps @rollup/plugin-commonjs to v28 which sets strictRequires: true by default, for those who're impacted by this. #18231

@FEliuyg
Copy link

FEliuyg commented Oct 18, 2024

Has this question been updated? I am currently encountering the same problem, but there is no problem when I create a new project. I don’t know what triggers this hash and why it changes every time I build it ,even if i do not change anything.

@juanparati
Copy link

Any of the solutions provided don't work for me, so I found another solution and it was to provide my own hash.

But first is important to say that my solution may be dangerous in case that for some reason some different code is injected in the different builds (somebody commented that some Sentry tool was injecting different parameters into the code).

In order to verify that builds are equal across different builds and environments I recommend to force Rollup to build the code without hashes using the following configuration:

build: {
        rollupOptions: {
            output: {
                entryFileNames: 'assets/[name].js',
                chunkFileNames: 'assets/[name].js',
                assetFileNames: 'assets/[name].[ext]',
            }
        }
}

and the verify that CRC sums of files doesn't change across builds and environments (You can use ls -A1 | xargs shasum in order to display CRC hashes).

If the CRC matches then It's safe to use my workaround.

Workaround:

import { defineConfig } from 'vite';
import fs from 'fs';

let dynamicHash = null;

const generateConsistentHash = (chunk, ext = '[ext]') => {
    if (dynamicHash === null) {
        if (fs.existsSync('REVISION')) {
            dynamicHash = fs.readFileSync('REVISION').toString();
        }

        // Keep 'base64url' because it will remove '+', '/' and '='
        dynamicHash = dynamicHash ? Buffer.from(dynamicHash, 'hex').toString('base64url') : false;
    }

    return dynamicHash ? `assets/[name].${dynamicHash}.${ext}` : `assets/[name].[hash].${ext}`
}

export default defineConfig({
    build: {
        rollupOptions: {
            output: {
                entryFileNames: (chunk) => generateConsistentHash(chunk, 'js'),
                chunkFileNames: (chunk) => generateConsistentHash(chunk, 'js'),
                assetFileNames: (chunk) => generateConsistentHash(chunk, '[ext]'),
            }
        },
    }
});

My workaround will read the last the last revision deployed as a hex value from the 'REVISION' file and it will converted into a hash string. if the 'REVISION' file is missing it will use the standard Rollup hash.

Advantages

  • I will use different hashes when REVISION file is missing, so if the development instance doesn't have the file every build will invalidate the cache.
  • It should create consistent hashes across different instances that has the same REVISION.

Disadvantages

  • It can create conflicts when the chunk files have different contents.

@sapphi-red
Copy link
Member

sapphi-red commented Dec 19, 2024

Vite 6 now uses @rollup/plugin-commonjs v28 which sets strictRequires: true by default. Also the legacy plugin issue mentioned in #13672 (comment) has been fixed by #16566.

Closing as the OP case has been fixed.
Feel free to create a new issue if you are still facing this issue. But note that this can be caused by other plugins or frameworks on top of Vite (e.g. #13672 (comment), #13672 (comment)), so please first try with removing the plugins or reporting to the framework side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug: upstream Bug in a dependency of Vite feat: commonjs @rollup/plugin-commonjs issue p3-minor-bug An edge case that only affects very specific usage (priority)
Projects
None yet
Development

No branches or pull requests