pnpm has memory leaks when project has a large number of dependencies

[guitaoliu]

The current script used to fetch pnpm dependencies, pnpm ls --depth 10 --parseable --long -r --prod, has a potential memory leak as described in pnpm/pnpm#3518.

The OSS Review Toolkit encountered this issue when trying to add pnpm support using pnpm ls.
oss-review-toolkit/ort#5594

They have a different implementation that does not rely on pnpm ls.
oss-review-toolkit/ort#5611

[jasonpraful]

It looks like the agent has resorted to using npm to resolve the dependencies, but the workaround does not seem to work as expected. While it does fetch the publicly available packages successfully from the npm registry, the workaround doesn't fetch private packages and their dependencies from internal registries. Completely skips using .npmrc file for authentication

[NitzanOrK]

@jasonpraful As part of the workaround, we check which registry is the default registry, according to the "registries" section in the modules.yaml file created after the dependency installation, and use it to get the required information from the registry API.
To understand why they are dependencies that we don't receive information about, we need to look at the log and modules.yaml

[jasonpraful]

@NitzanOrK While it's okay to use the default registry for public packages, the tool falls apart when the downloader needs to download packages published in private registries.

As an example - here's what my registries look like in modules.yaml

registries:
  '@<redacted-scope-1>': https://<redacted>.pkgs.visualstudio.com/_packaging/<redacted>/npm/registry/
  '@<redacted-scope-2>': https://<redacted>.pkgs.visualstudio.com/_packaging/<redacted>/npm/registry/
  '@<redacted-scope-3>': https://<redacted>.pkgs.visualstudio.com/_packaging/<redacted>/npm/registry/
  default: https://registry.npmjs.org/

When the enrichSha1PnpmFallback function from PnpmLockCollector.java in the agent is invoked (obtained the references by decompiling and observing the following jar), it always uses the default registry when it's trying to download the package (in this case @<redacted-scope-1>/helpers).

Unfortunately, I cannot share the complete logs as it has internal data in it but I've added redacted log snippets from where the following fallbacks are invoked.

//log
[WARN] [2024-12-09 16:07:25,885 +0000[ - PnpmLockParser - Please check connectivity and pnpm installation
[DEBUG] [2024-12-09 16:07:25,885 +0000[ - PnpmLockCollector - Failed to run pnpm ls with --json flag
[DEBUG] [2024-12-09 16:07:25,910 +0000[ - Trying to get the dependency sha1 for: safe-buffer-5.1.2.tgz
...
[DEBUG] [2024-12-09 16:07:27,246 +0000[ - Got 404 status code from registry using the url https://registry.npmjs.org/@<redacted-scope-1>/helpers/10.2.1.
[DEBUG] [2024-12-09 16:07:27,246 +0000[ - Fallback url - https://registry.npmjs.org/@<redacted-scope-1>/helpers/10.2.1

Hope this helps.

[NitzanOrK]

@jasonpraful The purpose of the Unified Agent is to detect and provide insights about open source libraries in your project. At this stage of the resolution process, we attempt to calculate the SHA-1 hash of a library to retrieve corresponding information from our database about open source components.

In this case, the library @/helpers/10.2.1 is identified as a private dependency, not an open source library. Since private dependencies are not relevant to the information we display about open source components, we do not calculate a SHA-1 hash for them.

If you have any further questions or need clarification, feel free to let us know.