chore!: publish gax-nodejs in dual format (CJS and ESM)

[sofisl]

This is a massive PR... If I could make it any smaller, believe me, I would!

Changes for Node 18:
Some ESM syntax we're using requires Node 18 (for example, using with assertions to import JSON). So, this PR also includes the bare minimum needed to migrate gax to Node 18:

• Updates the post-processor to the latest version to use Node 18 images.
• Sync-repo-settings makes Node 18 & 20 required tests.
• All wrokflows and testing and releasing images using Node 18 now.

Changes for ESM:

• Added an esm/ directory-level so that this can be renamed to cjs in the build phase. This will allow modules to be more clearly defined in terms of whether a file is esm or cjs.
• Added a .babelrc configuration file to help use custom Babel plugins to transpile ESM to CJS (as per Publishing Nodejs libraries in ESM and CJS)
• Migrated .jsdoc.js to .json since jsdoc doesn't look for a .cjs extension, but does understand JSON
• Other .js configuration files renamed to .cjs (mocharc.js & prettierrc.js) to be imported as common js
• custom files:
  • add-cjs-extension.cjs: finds files using proxyquire and makes sure they import the .cjs counterpart, not esm
  • replace-protobuf-imports.cjs: replaces a line in compiled protos to import using default module vs. start notation.
• Generally: added full file paths to all imports (i.e., import {CallSettings} from './gax' --> import {CallSettings} from './gax.js'
• Import JSON using with {type: 'json'}; extension (this is a weak spot for ESM; as of Node 18 you can import using with assertion)
• Changed references to __dirname to const dirname = path.dirname(fileURLToPath(import.meta.url)); (see Publishing Nodejs libraries in ESM and CJS) - we made a custom babel plugin to also transpile this to __dirname for cjs.
• Importing protos/ is now one directory level above what it used to be as are all the root-level files (like package.json) given the new directory structure of esm/src/.... References to these files have added a directory level.
• reran proto compilation commands (in package.json) for the common protos that gax compiles
• Changed the webpack configuration for the browser test to properly null-load the new imported file
• Added tsconfig.esm.json to compile into esm as well as cjs.
• Migrated us to using node-fetch v3, which is an ESM-module! We dynamically import node-fetch.

Tests

• Added tasks as well as speech for system tests connecting to our current libraries. This is because tasks is published in dual-format, whereas speech is currently only CJS. This guarantees gax works with both cjs and libraries published in dual format.
• Unit tests needed to be refactored to not use sinon. Instead we use esmock for the most part, which gets compiled into proxyquire in cjs.
• Stubs were heavily used with the warnings module. Instead we are listening for actual warnings, and asserting the warnings when they are emitted.
• Added node 22 tests

Tools

• compileProtos in tools now needs to get protos from one directory level higher given the new ESM level in the directoru structure.
• Since ES modules export modules and not objects, we need to access protobufjs exports from the default property, so we need to add the line \nconst $protobuf = protobuf.default to all of our protos; this will happen when we compileProtos, which is why we'll need to rerun before submitting.
• The protobuf export gax exports needs to now be exported from the correct build directory.

Migration Guide

• For our client libraries: we will need to add an @ts-ignore tag to the generated code importing gax, specifically: this._gaxModule = opts.fallback ? gaxInstance.fallback : gaxInstance; in the client.ts; the reason is that client libraries, in their CJS incarnation, optionally asynchronously load gax in the line before. However since it is not imported at the top of the file, ESM thinks it's not being loaded, when in fact in ESM it is not conditionally required (it is just asynchronously loaded due to being a regular ESM import). TLDR: TS doesn't realize gax will always be loaded, so we need to ignore the warning. We must add this to our generated client library code.
• Also, we will need to add ["allowSyntheticDefaultImports": true](https://www.typescriptlang.org/tsconfig/#allowSyntheticDefaultImports) in our tsconfigs to appropriately load gax.
• Lastly, we'll need to re-add compileProtos before running tests in system tests. Since we are slightly modifying how we compileProtos, we'll need to recompile before running tests.

Package.json changes

• main and types default to CJS, while the module is actually overall type as module (i.e., esm). @danielbankhead, do you have an example of where this won't work (that we claim to support)? This would be an awesome test to add in if so! Reason I'm asking is because this design is what we're doing for our other repos, specifically all our client libraries (see storage), so it's worth confirming if this actually won't work.
• Exports and imports are specified for all likely export paths, which will vary based on whether the user is using CJS or ESM.
• We are adding babel dependencies to compile ESM code to CJS.
• All test commands are doubled - one for CJS and one for ESM.
• Compile commands are also doubled. For ESM, we 1) run TS, 2) copy over any *.json files, 3) copy over tests, 4) do some slight find-and-replace functions to make protos esm-friendly, 5) copy over the protos to build. For CJS, we 1) run TS (the CJS version), 2) Run babel, 3) add our custom post-processing command (that adds a .cjs extension for files mocked in proxyquire), 4) copy over any *.json files, 5) copy over test fixtures.
• Proto compilation commands are doubled, one for CJS and one for ESM. The key difference is that ESM adds this -w es6 command, which outputs the JS to ES6 instead of UMD.
• Most other commands need to add one more level of directory structure due to the new esm directory level
• The babel command which is described in Publishing Nodejs libraries in ESM and CJS: basically takes the TS output and runs some custom babel plugins, as well as copies over files, and puts files into the cjs directory.
• I removed the browser field, and removed the module.exports code in fallback.js (the previous browser entry point). this was there to ensure fallback would work in esm and cjs contexts; but, since we now have separate entry points for both, we can just remove all this code entirely.

TODO for after:

• system tests rely on published versions of speech and tasks. Given those changes described in the migration guide, that haven't been published in those libraries, we are commenting out the part that grabs the latest versions of the libraries and instead using its current code in the monorepo. This doesn't change the integrity of the tests at all, it's just slightly less safe in that published versions of libraries are guaranteed to have passing tests vs. current code. We just need to uncomment lines 49 - 86 and 118 -122 in gax/esm/test/system-test/test.clientlibs.ts
• CompileProtos needs to be republished in order to unskip the skipped test that tests where gaxProtos live. This test is failing because of a circular dependency: the current published version of gax doesn't contain this new directory level, so we can't test it appropriately. After publishing dual-format, we can re-enable this test to see it works.In the meantime, since client libraries use compileProtos, we'll see it work correctly in system tests with the current version of gax.

[generated-files-bot]

Warning: This pull request is touching the following templated files:

• .kokoro/common.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/continuous/node14/common.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/continuous/node14/lint.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/continuous/node14/samples-test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/continuous/node14/system-test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/continuous/node14/test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/continuous/node18/browser-test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/presubmit/node14/common.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/presubmit/node14/samples-test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/presubmit/node14/system-test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/presubmit/node14/test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/presubmit/node18/browser-test.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/release/docs.cfg - .kokoro files are templated and should be updated in synthtool
• .kokoro/samples-test.sh - .kokoro files are templated and should be updated in synthtool
• .kokoro/system-test.sh - .kokoro files are templated and should be updated in synthtool
• .kokoro/test.bat - .kokoro files are templated and should be updated in synthtool
• .kokoro/test.sh - .kokoro files are templated and should be updated in synthtool
• .github/workflows/ci.yaml - .github/workflows/ci.yaml (GitHub Actions) should be updated in synthtool

[d-goog]

A quick note: the main should be .js/.mjs when type: esm - some, namely older, build tools won't work well with the conflict. Alternatively, it may be easier to use type: commonjs as the default

[sofisl]

@danielbankhead, do you have an example of where this won't work (that we claim to support)? This would be an awesome test to add in if so! Reason I'm asking is because this design is what we're doing for our other repos, specifically all our client libraries (see storage), so it's worth confirming if this actually won't work.

[d-goog]

Webpack 4- and earlier versions of eslint will fail here.

Additionally, in the storage example the main ends with .js rather than .cjs.

[d-goog]

I think of it not whether we claim to support it or not, but rather are we following guidance and standards appropriately. If not, we can expect some customers to run into friction.

[sofisl]

Hm, isn't storage still using a cjs file (seems like the js file is in the cjs module?) Also we are running tests using webpack 4 for the browser and they are WAI!

[sofisl]

Regardless, I don't feel strongly about this since we're testing gax in our system tests, and I think the exports field have precedence over main anyways. Running tests to make sure everything passes.

[sofisl]

Update, seems like this actually breaks browser tests: https://btx.cloud.google.com/invocations/37842334-df7a-4bd8-8aef-a97b319b0cd7/targets/cloud-devrel%2Fclient-libraries%2Fnodejs%2Fpresubmit%2Fgoogleapis%2Fgax-nodejs%2Fnode18%2Fbrowser-test/log

[sofisl]

FWIW, this was my attempt when adding back in a browser field to default to cjs and then use esm as the entrypoint: https://btx.cloud.google.com/invocations/feb590d8-03ab-40b6-bf59-fcd445db5f47/targets/cloud-devrel%2Fclient-libraries%2Fnodejs%2Fpresubmit%2Fgoogleapis%2Fgax-nodejs%2Fnode18%2Fbrowser-test/log

[d-goog]

Why not add a package.json with type: commonjs? According to spec:

• https://nodejs.org/api/packages.html#packagejson-and-file-extensions

Additionally reading/examples for this pattern:

• https://medium.com/ekino-france/supporting-dual-package-for-cjs-and-esm-in-typescript-library-b5feabac1357
• https://github.com/azu/tsconfig-to-dual-package
• https://github.com/knightedcodemonkey/duel

[sofisl]

This is for changing the extension in proxyquire calls, so it wouldn't be affected by a package.json, although I see your larger point that we wouldn't even have to do that if we didn't rename the files at all. It seems slightly cleaner/safer to me to have the .cjs and .js distinguishing the files, easier to see at a glance what's going on, but that's just my perspective!