Save model weights for each epoch 1720

[mmcs-work]

Overview

Save model weights for each epoch.

Notes

From #1720, proposition 1 is followed. Last checkpoint is still kept as last-model.pth. Instead of overwriting, it copies the previous checkpoint to model-ckpt-epoch-{N}.pth where N is the epoch number.

Testing Instructions

• From either running the notebooks or the unittests where Learner is extended. The option save_all_checkpoints can be added as in

learner = SemanticSegmentationLearner(
    cfg=learner_cfg,
    output_dir='./train-demo/',
    model=model,
    train_ds=train_ds,
    valid_ds=val_ds,
    save_all_checkpoints=True
)

Closes #1720

[AdeelH]

This is a simple and elegant solution to this problem. Really appreciate the high-quality PR!

Some minor changes I would suggest are:

• Make save_all_checkpoints a field in LearnerConfig instead of an argument to Learner.
• Store the checkpoints in a checkpoints subdirectory under the Learner's output directory to avoid cluttering up the latter. You can use the make_dir() which is already available in learner.py.

Additionally, if we want to make this field usable in the RV-as-a-framework context, we would also want to add it to PyTorchLearnerBackendConfig and then modify the get_learner_config() methods of all 3 of its subclasses to pass it to the respective LearnerConfig subclasses (identical to how log_tensorboard and run_tensorboard are handled). If you are only using RV as a library then this is probably not useful/relevant to you, so it is totally okay if you want to skip this in this PR!

[AdeelH]

Looks like there are linter errors. You can run scripts/style_tests locally to find those. And also scripts/format_code to format your code correctly.

If you are using VS Code, you can install the flake8 extension and set yapf as the formatter to ensure that your code is always compliant. Be sure to install the flake8 and yapf versions from requirements-dev.txt.

[codecov]

Codecov Report

Merging #1921 (e04168d) into master (30fd26f) will decrease coverage by 0.02%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master    #1921      +/-   ##
==========================================
- Coverage   82.49%   82.47%   -0.02%     
==========================================
  Files         190      190              
  Lines        9316     9325       +9     
==========================================
+ Hits         7685     7691       +6     
- Misses       1631     1634       +3     

Files Changed	Coverage Δ
...orch_backend/pytorch_chip_classification_config.py	62.96% <ø> (ø)
...pytorch_backend/pytorch_object_detection_config.py	61.53% <ø> (ø)
...ch_backend/pytorch_semantic_segmentation_config.py	62.96% <ø> (ø)
...ch_learner/rastervision/pytorch_learner/learner.py	75.66% <57.14%> (-0.24%)	⬇️
.../pytorch_backend/pytorch_learner_backend_config.py	68.75% <100.00%> (+0.66%)	⬆️
...ner/rastervision/pytorch_learner/learner_config.py	83.12% <100.00%> (+0.03%)	⬆️

[AdeelH]

This looks good to me. Thank you!

[mmcs-work]

This looks good to me. Thank you!

Thanks for your help.