Minimize module loading

[petdance]

I'm loading Mojo::UserAgent in my web app to go fetch pages behind the scenes. Unfortunately, this brings in many unnecessary modules that have to get loaded in my app, such as Pod::Usage and Getopt::Long.

It would be nice if a module like Mojo::UserAgent that uses something like monkey_patch and term_escape wouldn't also bring in all the other unnecessary modules as well.

Maybe a Mojo::Util::CLI for things that are only used by CLI tools, for example?

[jberger]

While I don't generally feel we need to micro-optimize module loading across the stack, I guess I can see that there are a few dependencies in Mojo::Util that could be easily late-loaded when needed and which are rarely needed unless in specific contexts. I would consider supporting a PR (good for first timers?) that late-loads some of those rarely used dependencies specifically in Mojo::Util.

[Edit: changed from implied promise to support to consider supporting]

[kraih]

Late loading seems like it would be bad for prefork memory usage. I'd be very sceptical of any PRs in that direction and would want to see data on memory usage for various deployment scenarios.

[jberger]

In general that would be true, however, at least Pod::Usage would rarely need to be early-loaded for preforking. I guess I could argue GetOpt::Long either way, though in a non-webapp/Mojo::UserAgent context it would certainly seem to be a win.

On the other hand, it might make things a bit harder for say fat-packing deployments. Missing dependencies because they weren't detected at load-time. Though on the other-other hand, core modules like Pod::Usage and GetOpt::Long don't have to be fat-packed anyway.

I've been debating whether say the Socket dependencies in network_contains would be worth late-loading, but yes, that would suffer from a preforking perspective. All of this is very interesting. @petdance do you have numbers on what the memory usage penalty is for those?

[petdance]

I don't have any memory usage penalty numbers, because I wasn't even at that point yet. So far, it's just a "Hmm, why is that here?"

Every so often I do a survey of modules that are loaded in our preloaded web app. Currently, we have 983 from Algorithm::Verhoeff to XML::Xerces. The goal is to make sure that everything gets early loaded (through a startup script if necessary) vs. late loaded. (I also make sure that there are no Test::* modules that get loaded somehow.)

While perusing the list, I was surprised to see the modules I mentioned above: Getopt::Long and the six Pod::*, since nothing in the app would every do anything with either of them. I tracked down what was bringing them in (since it wasn't in our code tree). (Thanks to Devel::TraceLoad!). And then I wrote this issue up.

So, no, I'm not saying "OMG your preloading is ruining my app!!11!1!!!". On the other hand, I could see it becoming a problem in the future pretty easily just by having something added to Mojo::Util. That's why my first thought, as someone who knows not much about Mojo, was to split CLI-only tools into Mojo::Util::CLI.

[petdance]

FYI, I found that it was about 13MB less memory used in my Apache children if I don't use Mojo::UserAgent. Here are the modules that stopped getting loaded if I remove Mojo::UserAgent from my web app's modules:

• Class::Struct
• Cpanel::JSON::XS
• File::stat
• Getopt::Long
• IO::Poll
• Pod::Escapes
• Pod::Simple
• Pod::Simple::BlackBox
• Pod::Simple::LinkSection
• Pod::Text
• Pod::Usage
• and of course Mojo::*