Engine crashing when loading chobby without launcher

[saurtron]

Description

This problem might be linux only, not sure.

When running chobby directly (as documented here) with the engine instead of through launcher it will quickly corrupt and crash. It seems to start ok, but after starting skirmish it will corrupt and die in less than a minute.

Also if I try to go back to lobby lots of images seem to be missing at the point of the corruption.

update: This is caused by resource exhaustion due to not properly closing luasockets in chobby. See beyond-all-reason/BYAR-Chobby#827

More information

I have tried this with many different engine versions, as far as the fork from #1774, also master, with and without my personal branches.

All the tests I did are from running spring directly from host os (as opposed to running through flatpak), so this could be also because of custom build instead of something to do with chobby, still I think I have pinpointed the problem to specific module in chobby.

Corruption starts to manifest as errors with images, sounds or fonts (maybe vfs access related no idea).

Not totally sure what's going on, but seems to be related to repeated calling client = socket.tcp(). I'm still debugging this to get a more final assesment and have no idea how that could be causing the problem tbh.

The problem definitely comes from api_spring_launcher.lua. Commenting all SocketConnect calls seems to make the issue go away.

This is something that can be worked around in chobby so the engine doesn't crash, but posting here so it can be properly investigated since it could be affecting other situations.

Error logs

This error usually crashes the game although I think it doesn't always show:

(no errors before this line)
[t=00:01:13.081375][f=0000315] Fatal: [ExitSpringProcess] errorMsg="vector::_M_default_append" msgCaption="Spring: caught std::logic_error" mainThread=1
[t=00:01:13.081504][f=0000315] Error: [Watchdog::ClearTimer(id)] Invalid thread 4 (_threadId=(nil))

When commenting SocketConnect from api_spring_launcher only in Update, it seems to take a few seconds more to crash, and also show some symptoms before the crash:

[t=00:00:31.103424][f=0000031] Loaded atmosphere CEGs config for map: Quicksilver Remake 1.24
[t=00:00:33.103939][f=0000091] [Sound] Error: [LoadWAV(sounds/atmos/windy1.wav)] invalid header
[t=00:00:33.103954][f=0000091] [Sound] Warning: [LoadSoundBuffer] failed to load file "sounds/atmos/windy1.wav"
[t=00:00:33.436761][f=0000100] [Sound] Error: [LoadWAV(sounds/buildings/arm-bld-metal.wav)] invalid header
[t=00:00:33.436780][f=0000100] [Sound] Warning: [LoadSoundBuffer] failed to load file "sounds/buildings/arm-bld-metal.wav"
[t=00:00:33.436813][f=0000100] [Sound] Error: [LoadWAV(sounds/buildings/arm-bld-mex.wav)] invalid header
[t=00:00:33.436829][f=0000100] [Sound] Warning: [LoadSoundBuffer] failed to load file "sounds/buildings/arm-bld-mex.wav"
[t=00:00:33.454693][f=0000101] Warning: Couldn't find texture "LuaUI/Images/commandsfx/line.dds"!
[t=00:00:33.536974][f=0000104] [Sound] Error: [LoadWAV(sounds/atmos/beach4.wav)] invalid header
[t=00:00:33.536988][f=0000104] [Sound] Warning: [LoadSoundBuffer] failed to load file "sounds/atmos/beach4.wav"
[t=00:00:35.636978][f=0000167] [Sound] Error: [LoadWAV(sounds/atmos/beach5.wav)] invalid header
[t=00:00:35.637006][f=0000167] [Sound] Warning: [LoadSoundBuffer] failed to load file "sounds/atmos/beach5.wav"
[t=00:00:35.903652][f=0000175] [Sound] Error: [LoadWAV(sounds/atmos/windy4.wav)] invalid header
[t=00:00:35.903666][f=0000175] [Sound] Warning: [LoadSoundBuffer] failed to load file "sounds/atmos/windy4.wav"
[t=00:00:36.570943][f=0000195] [WatchDog::DeregisterThread] deregistering controls for thread [main]
[t=00:00:43.443528][f=0000195] Fatal: [ExitSpringProcess] errorMsg="vector::_M_default_append" msgCaption="Spring: caught std::logic_error" mainThread=1
[t=00:00:43.443674][f=0000195] Error: [Watchdog::ClearTimer(id)] Invalid thread 4 (_threadId=(nil))

[saurtron]

Ok, found the issue and managed to fix chobby so it won't make the engine crash all the time.

Seems like it's because of not closing luasockets, thus making the system run out of descriptors, this was then creating lots of problem by failing to load images, sounds, fonts... and finally crashing because the engine doesn't have protections against this everywhere.

Not sure what can be done about this, the socket api is definitely unsafe atm (at least on linux), I think a high level wrapper could probably make it safer to use. The engine could check descriptors too I guess and maybe leave some margin XD, or have more error checking when loading resources to avoid crashing at certain places.

At the moment the socket api seems to leave sockets open when doing socket.tcp(), then client:connect(), detect a failure and just nil the thing instead of doing client:close().