Local Agent : "too many open files"

[TwilCynder]

I'm trying to follow the basic install instructions on the website, compiled alumet-local-agent, and when I try to run it, I get a bunch of

[2024-11-21T12:25:33Z ERROR alumet::pipeline::elements::source] Error in source creation requested by plugin procfs: error in Trigger::new
    
    Caused by:
        Too many open files (os error 24)

Alays ending with

[2024-11-21T12:25:33Z ERROR alumet::pipeline::control] error in message handling: plugin procfs requested to create 461 sources, 130 failed (see logs above)
[2024-11-21T12:25:35Z ERROR alumet::pipeline::elements::source] Fatal error when polling procfs/source/process-watcher (will stop running): Unexpected IO error(/proc/1/stat): Too many open files (os error 24)
[2024-11-21T12:25:35Z WARN  alumet::pipeline::control] One source task finished with error: fatal error when polling procfs/source/process-watcher

The number of created and failed sources vary, but the difference (i.e. number of sources that were successfully created) is always 331.

I tried changing the user file limit in /etc/sysctl.conf to 1024, I also tried ulimit -n 10000, nothing changes.

I'm on Ubuntu 22.04.5

Does anyone have any idea what could cause that ?

[AngeCyp]

Hello,

You could first check:

• System side limit using: cat /proc/sys/fs/file-max
• Shell side limit using ulimit -n

Did you try to log out and log in back or reboot your system ?

Can you share what you changed in the **/etc/sysctl.conf ** file ?
Did you try with an higher value than 1024 ?

[TwilCynder]

My system wide limit is 9223372036854775807, my shell side limit was 1024 and I increased it to 10000 using ulimit.

What I added in sysctl.conf is fs.inotify.max_user_instances = 10000. Tried with 1024 first, each of these changes was followed by a reboot.

[TheElectronWill]

Hello! Thanks for trying Alumet 🔥 We'll figure it out :)

Quick questions:

• how many processes are running? (ps -aux | wc -l)
• are you running the Alumet agent with sudo?

Changing fs.inotify.max_user_instances is a good idea, but unfortunately not enough, because Alumet uses other kinds of file descriptors too. The limits system is quite complicated: ulimit applies to the shell, there is a "soft" and a "hard" limit, and the actual limit applied to the process and its children can be overriden by various system-wide configurations. For the record, /proc/sys/fs/file-max or sysctl fs.file-max is the Kernel limit, and is usually very high (so, not the problem here).

To see the actual limits applied to a process, use cat /proc/<pid>/limits. In the shell you can do:

cat /proc/self/limits

It can be different than what ulimit gives. I suggest that you also try with the Alumet process.

If it returns the same limit as ulimit then I would suggest to increase the limit more, e.g. with ulimit -Sn 32000.
Note that you need to run the agent in the same shell as your ulimit command, otherwise it won't do anything.

The related system configs are:

• /etc/systemd/system.conf (Look for DefaultLimitNOFILE)
• /etc/systemd/user.conf. (Look for DefaultLimitNOFILE)
• /etc/security/limits.conf (Look for a line with nofile, try to add or modify a line with * - nofile 100000)

[TwilCynder]

Hello !
I have ~450 processes running, and yes I'm using sudo.

Checking /proc/<alumetAgentPID>/limits shows "Max open files" is at 1024(soft)/1048576(hard) so that might be the issue, but that's after

• ulimit -n 32000
• Changing DefaultLimitNOFILE to 10000 in both system.conf and user.conf under /etc/systemd
• Adding * - nofile 100000 in /etc/security/limits.conf
  (and rebooting after each /etc/ change)

[TheElectronWill]

and yes I'm using sudo.

Checking /proc/<alumetAgentPID>/limits shows "Max open files" is at 1024(soft)/1048576(hard) so that might be the issue, but that's after

Here is the issue: ulimit only applies to the commands that are directly launched by your shell. With sudo, you create a level of indirection, and you loose the soft limit defined by ulimit. The solution is to replace ulimit by something like prlimit:

sudo prlimit --nofile=32000: path/to/alumet

[TwilCynder]

Oh indeed I didn't think about that indirection. prlimit did the trick, thank you very much !