macincloud systems reporting 100% inode use in Nagios

[sxa]

We've got a lot of these messages in the infrastructure-bot channel related to the macincloud which is concerning and likely needs remediation if it's correct for the macos file system:
HOST: test-macincloud-macos1201-x64-1 SERVICE: Disk Space Root Partition STATE: WARNING MESSAGE: DISK WARNING - free space: / 16525 MiB (13.47% inode=100%)
HOST: test-macincloud-macos1201-x64-2 SERVICE: Disk Space Root Partition STATE: WARNING MESSAGE: DISK WARNING - free space: / 23837 MiB (19.43% inode=100%)

Originally posted by @sxa in #3372 (comment)

Noting that they are currently offline while we're testing orka, but we should look at checking and resolving this, then re-enabling them.

[sxa]

@steelhead31 Can you take a look at the cause of this - it looks like there is an automounter controlling /home which may be throwing Nagios - also bear in mind that the home directories are under /Users instead of /home on macos.

map auto_home            0        0         0   100%       0         0  100%   /System/Volumes/Data/home

[steelhead31]

Having done some digging, the root file system for these machines is split across 2 volumes.. e.g

DISK WARNING - free space:
/ 16523 MiB (13.46% inode=100%);
| /=106156MiB;98143;110411;0;122679

As a result, the nagios check_disk plugin needs an additional parameter to calculate the combined space of both of these file systems, the --combine-thresholds parameter should be specified..

DISK OK - free space:
/ 16523 MiB (13.46% inode=100%);
| /=106156MiB;98143;110411;0;122679
 

A small update to the nagios configuration now has these filesystems alerting properly :)

We should probably look to extend the coverage to other file systems, if these continue to be required.