performance

[fdncred]

I've been working on ways to make globbing with wax a little faster with rayon and I keep coming up empty. I'm beginning to wonder if it needs to be built in somewhere so that it globs a directory and then all the directories inside get their own thread to glob their contents, up to some limit. Any thoughts on this?

[olson-sean-k]

make globbing with wax a little faster with rayon

Yeah, I've also thought about this a bit. Wax depends entirely on walkdir for this, so I think any such features would need to be implemented upstream. The major exception to this is filtering, as Wax is basically a glorified filter atop walkdir. #49 is a great example of a serious performance problem concerning this filtering, wherein directory trees were not discarded and would be read from the file system for no reason (despite what the documentation claims). I just landed a fix for that in 76afaa1.

[fdncred]

I'm wondering how hard it would be to switch from walkdir to jwalk which already supports streaming, sorting, threads, etc. and is also built on walkdir and ignore.
https://github.com/Byron/jwalk

[olson-sean-k]

Oh, cool, I wasn't aware of jwalk! Thanks for pointing it out! It's not clear to me how easy it would be to replace walkdir with jwalk, but I'll take a look.

[milesj]

Just chiming in that we're seeing slow performance for globs like apps/*/*. Would this be related to walkdir?

Context: moonrepo/moon#1721