sf project generate manifest returns inconsistent results

[jakubmplatek]

Note
Before you submit your issue, make sure that:

• You're using the latest version of Salesforce CLI.
• You've searched both open and closed issues for related posts.
• You've used the doctor command to diagnose common issues.
• You understand that GitHub Issues don't adhere to any agreement or SLA.
  • If you require immediate assistance, use official channels such as Salesforce Customer Support.

Summary

I'm building automation to run scheduled snapshots of all metadata in my orgs, using sf project generate manifest --from-org command as a base for generating batches of retrieve calls.
I noticed some unexpected changes listed between runs, and found that that initial package manifest generated is inconsistent: it will randomly return a different count of members for a type, or skip a type entirely.

I've tested it by running the generate manifest command in a loop against the same environment and then comparing the generated packages, and found individual runs completely missing some types, including:
ApexClass,MatchingRule,CustomObject,ValidationRule,WebLink,AppMenu,CustomField,CompactLayout,CustomPermission,ListView,CustomMetadata

Tested on different instances, both in prod and sandbox orgs, and in different sf projects, all using the latest cli (2.69.14)

Steps To Reproduce

IMPORTANT
Provide a repository that's configured to reproduce the issue. If you are unable to provide a repo, please explain why not. The more info we have from the start, the faster we can resolve your issue.
We may close your issue if you don't include proper instructions.

• Generate a project with sf project generate or fork dreamhouse-lwc.
• Provide detailed step-by-step instructions on how to reproduce the issue.

authenticate with a salesforce org and run the command a few times to generate multiple manifests (date used to autogenerate a different filename each time)
sf project generate manifest --from-org <alias> --name package_$( date +%s ).xml
compare the contents of the generated package manifests - some of them will have whole metadata types missing, or some members within a metadata type missing

note: the issue is intermittent, so it might take 5-10 runs to see significant differences

Tip

use sf doctor --create-issue to automatically fill the required information

logs generated by doctor:
logs.zip

Expected result

while testing in an org without active users (i.e. no changes to metadata), I expect each package manifest to have strictly equal contents

Actual result

components listed differ between generated manifests

Additional information

System Information

Windows 10, Git for Windows bash

{
    "architecture": "win32-x64",
    "cliVersion": "@salesforce/cli/2.69.14",
    "nodeVersion": "node-v20.11.0",
    "osVersion": "Windows_NT 10.0.19045",
    "rootPath": "C:\\Users\\jplatek\\AppData\\Roaming\\npm\\node_modules\\@salesforce\\cli",
    "shell": "C:\\Program Files\\Git\\usr\\bin\\bash.exe",
    "pluginVersions": [
      "@copado/copado-cli 3.3.0 (user)",
      "@oclif/plugin-autocomplete 3.2.11 (core)",
      "@oclif/plugin-commands 4.1.12 (core)",
      "@oclif/plugin-help 6.2.18 (core)",
      "@oclif/plugin-not-found 3.2.29 (core)",
      "@oclif/plugin-plugins 5.4.19 (core)",
      "@oclif/plugin-search 1.2.16 (core)",
      "@oclif/plugin-update 4.6.14 (core)",
      "@oclif/plugin-version 2.2.16 (core)",
      "@oclif/plugin-warn-if-update-available 3.1.24 (core)",
      "@oclif/plugin-which 3.2.19 (core)",
      "@salesforce/cli 2.69.14 (core)",
      "apex 3.6.6 (core)",
      "api 1.3.2 (core)",
      "auth 3.6.78 (core)",
      "data 3.12.3 (core)",
      "deploy-retrieve 3.15.17 (core)",
      "info 3.4.25 (core)",
      "limits 3.3.42 (core)",
      "marketplace 1.3.7 (core)",
      "org 5.2.7 (core)",
      "packaging 2.9.5 (core)",
      "schema 3.3.43 (core)",
      "settings 2.4.8 (core)",
      "sobject 1.4.46 (core)",
      "telemetry 3.6.26 (core)",
      "templates 56.3.33 (core)",
      "trust 3.7.45 (core)",
      "user 3.6.4 (core)",
      "@salesforce/sfdx-scanner 4.7.0 (user)",
      "sfdmu 4.35.1 (user)",
      "sfdx-git-delta 5.42.1 (user)"
    ]
  }

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[iowillhoit]

Hey @jakubmplatek, this is a complicated command that performs a lot of concurrent requests to attempt to gather all of the org's metadata. We are actively working on a few changes that will hopefully improve the reliability of this command. I will link to the existing work item (for internal reference). We will try to post here once this is completed for you to test against. You can also watch the Release Notes for updates.

@W-17431548@

[jakubmplatek]

Thanks for the update @iowillhoit
Is there a rough ETA on this work item? I'm working on automation that would rely on this, so need to know if I need to work on a different approach before this is ready.

[iowillhoit]

It is currently In Progress by one of our engineers but this is not a simple task. It is possible we will have something before the holiday break, but it could be early January.

sf project generate manifest is really a convenience wrapper for making sf org list metadata command calls to an org for every known metadata type and including the results in a manifest file. There are certain types that cannot be collected with a wildcard, so we need to make a lot of calls to collect them. Request retries are made but it's not perfect. If you run this command with the --dev-debug flag and/or setting the JSFORCE_LOG_LEVEL=DEBUG env var you will see what I mean :)

The idea with this work is to throttle requests to hopefully reduce connection resets. It will also provide the ability to exclude certain types of metadata (maybe with the .forceignore, maybe with a command flag).

[jakubmplatek]

I can work with that, thanks.
I personally like that it's not affected by forceignore, because of the way I use it with managed component access stuff (I want them in the manifest, but not necessarily in the repo), so a flag would be preferable. Right now I'm just filtering it while generating my batches, so it would be nice to limit step 1 to improve performance.
Have a good holiday!

[iowillhoit]

Thanks @jakubmplatek you too! I'll pass your feedback along 👍

[iowillhoit]

FYI: Sounds like we are taking the flag approach