Feature: Resume Training

[Poly2it]

Unless I am missing something, it is currently not possible to pause a training job and resume it later. This would be quite beneficial for splitting large jobs on limited resources, and it would also provide a safeguard against crashes.

[ArthurBrussee]

Yes great point. It is somehwat-sorta-kinda possible if you export a ply, and put it in your training data as an "init.ply".

That of course needs some improvements like:

• Automatically save a splat every N steps
• Save settings (Include a settings.json in the input #25) and other training data
• Save optimizer state (though irrelevant before Adam surgery to keep momenta #30)
• Do this in a way so that it is just included in the training data, and you can easily restart from it. That is... very annoying before Ability to read from directories instead of .zip files when possible. #1 is fixed.

Burn has a checkpointing mechanism so could integrate with that, but I do like the idea that exports are just viewable ply files (mostly).