[bug] Moon commands hang when a repo has multiple projects with node_modules dir

[dudicoco]

Describe the bug

In a repo with multiple projects and node_modules dir per project, moon commands become very slow.
It seems that moon checks the contents of each node_modules dir each time.

Steps to reproduce

1. clone https://github.com/dudicoco/moon-reproduction
2. run moon :install
   • the command completes immediately
3. run pnpm install
4. run moon :install
   • the command hangs for a few seconds

Environment

$ moon --version    
moon 1.29.4

[milesj]

It looks like the slowdown is the glob. There's nothing about node_modules in the logs.

[DEBUG 15:16:58.563] moon_project_graph::project_graph_builder  Locating projects with globs  globs=["apps/*/*"]
[DEBUG 15:17:08.247] moon_project_graph::project_graph_builder  Attempting to load apps/backend/app-01/moon.yml (optional)  id="app-01"

If we change it to apps/backend/*, the perf issues go away:

[DEBUG 15:18:49.164] moon_project_graph::project_graph_builder  Locating projects with globs  globs=["apps/backend/*"]
[DEBUG 15:18:58.732] moon_project_graph::project_graph_builder  Attempting to load apps/backend/app-01/moon.yml (optional)  id="app-01"

We use a 3rd party globbing library, so we don't have much control here.

[dudicoco]

@milesj since this basically makes moon unusable with larger repos what could be done here?
Should we open an issue upstream?

[milesj]

Looks like there's a probably relevant thread here: olson-sean-k/wax#51

[dudicoco]

@milesj I am not sure this is a performance issue but rather a configuration/misconfiguration issue in either moon, wax or walkdir.

There are three problems here:

1. The glob is for apps/*/*, node_modules are in apps/*/*/node_modules. So the glob is continuing walking the dirs. Is there a min/max depth option?
2. Since files here are irrelevant, the glob should only search for directories.
3. The glob is not respecting .gitignore, if it did then node_modules wouldn't have an impact.

Here's an illustration:

$ time find apps/*/*
find apps/*/*  0.98s user 23.13s system 45% cpu 52.589 total

$ find apps/*/* -type d
find apps/*/* -type d  0.80s user 21.56s system 47% cpu 46.765 total

$time find apps/*/* -maxdepth 1
find apps/*/* -maxdepth 1  0.00s user 0.01s system 36% cpu 0.043 total

# emulating what respect .gitignore would do
$ time find apps/*/* -prune -o -name 'node_modules'
find apps/*/* -prune -o -name 'node_modules'  0.00s user 0.00s system 60% cpu 0.006 total

[milesj]

So node_modules is already a globally negated glob: https://github.com/moonrepo/starbase/blob/master/crates/utils/src/glob.rs#L17 This should be avoiding traversing into those folders.

I haven't tried it, but you can also do apps/*/*/moon.yml.

[dudicoco]

@milesj apps/*/*/moon.yml doesn't work, it doesn't detect the projects.
But even if it did it wouldn't be a sufficient solution since some projects don't have moon.yml and rely on inherited tasks.