Final week task list

[sofstef]

Here's the list of tasks left to do by Friday (a few could be left for later). I suggest everyone writes which ones they're taking on and then I'll add a name next to the task and you can tick it when done. Use the comments to extend the list as needed and I can keep editing this issue.

Model training

• 1. Make sure we understand if there are any unwanted consequences from setting num_workers=0 (@Hamish-Cam cool to take this one?)
• 2. Do we even need the option with a lot of samples being completely without fire? That background class should be represented with the balances sample anyways, so my intuition is that we should keep burn_prop=1. Maybe someone could ask Martin about this on Monday. (@Hamish-Cam cool to take this one?)
• 3. Think about what the samplers are doing in validation and testing–what are the options we could/should try out? (@Hamish-Cam cool to take this one?)
• 4. Try and make wandb auto learning rate finding work
• 5. Length passed to sampler: should we fix it to a set value using the code snippet from the torchgeo issue (Improvements to RandomBatchGeoSampler microsoft/torchgeo#380)
• 6. Tune batch size (see if this is something wandb can do as well)
• 7. Run with sentinel on Jasmin GPU @sofstef
  • 1. Run small sample first
  • 2. Run all five years with monthly means
• 8. Add ERA5 downscaled to landcover resolution
• 9. Tuning steps apply when sentinel and ERA5 are added as well.

Model evaluation

• 1. Save outputs of the model in a sensible way when running on JASMIN @sofstef
• 2. Inspecting predictions
  • Run training script with binary segmentation task in colab and get binary mask predictions @Hamish-Cam
  • Log predictions in training with wandb (check out log predictions branch and try and adapt the plotting and logging functionality in the BinarySegmentationTask for wandb) @Hamish-Cam
• 3. (Not needed if the above works) Improve function that logs predictions during training (found in src/callbacks) @sofstef
• 4. Look into metrics other than jaccard index which would be appropriate to use in eval @graceebc9
• 5. Visualisations
  • 1. General reqs:
    • 1. sensible colormaps (especially colour-deficiency friendly)
  • 2. Plots showing all predictors, ground truth and prediction

FAIR tasks:

• 1. Make sure all data is accessible - @graceebc9 to take
  • - MODIS– describing where the data is on JASMIN
  • - Sentinel–making the pulled and processed data available somewhere, alongside the scripts to do the preprocessing
  • - Landcover data – either make available through Jasmin or upload on some drive and share link in the repo (ideally there would then be a download option in the landcover class itself, but this could be an extra post project)
• 2. Describe things in the Readme properly - @ThomasDodd97
  • - Project goals, description of datasets, how to run things and other things we made notes for on GHub.
  • - Some of this can be used in both report and Readme
• 3. Set up environment file
  • - All libraries used should be in the environment.yml file
  • created env file for loading datasets, can try merge this with the one for the models if we think is necessary
• Think about
• 4. Extras:
  • - Set up config with wandb https://docs.wandb.ai/guides/track/config @sofstef
  • - Consider creating a binder notebook for presentation demo (this could basically be similar to our colab notebook where we describe how to train the model and look at the predictions)

Repository organising

• Rename data_loading into datasets and apply changes everywhere
• Move all helper functions in data_loading into utils.py in the datasets folder

Report writing

• 1. Get everyone access to the Overleaf doc @sofstef
• 2. Make report outline with all relevant sections; some suggestions: - @graceebc9 to take
  • - Intro w lit review - @ThomasDodd97
  • - Datasets -@graceebc9
  • - Methodology -HC / SS
  • - Results w discussion - HC / SS
  • - Conclusion & any further steps - @graceebc9
• 3. Decide who writes what

[graceebc9]

I'll note down some next steps for the reports here as I go

Figures for the appendix/ data sources:
@ThomasDodd97 please could you add a legend to the landcover plot you created with the mosaic and save that image to the github under report / figures?

UPDATE - ive pulled a figure from the classification report from the landcover with the classes so no need to create this :)

[graceebc9]

@Hamish-Cam - please could you generate one of the modis fire map plots without the blue dotted box? I've deleted the folder with modis data so cant re run your script.

[Hamish-Cam]

Model Training

1. num_workers effects how quickly the CPU loads the data into the GPU => its value only effects the speed/efficiency of the GPU: https://chtalhaanwar.medium.com/pytorch-num-workers-a-tip-for-speedy-training-ed127d825db7. Given that the threading error seen is in the rasterio library and so is by no means (at least easily) fixable, I suggest we continue to take a speed hit, given that it means we can run bug free.
2. Martin agrees that for now burn_prop=1 for all cases. Since this isn't necessarily a long term solution, I won't alter the code so that we have the flexibility to change it if we wish.
3. Similarly, we are interested in the ability of our model to predict fires, not lack of fires. Therefore, I believe we should be using the same balanced sampler for val/test as for training (Martin agrees). As such I have pushed changes to remove the grid_sampler option (which was causing issues anyway) and instead use the constrained sampler for val/test when balance_samples = True.

[Hamish-Cam]

Martin has also suggested trying to overfit our model to one/two samples (by repeating training on these) to see if it can train to predict these fires.

He has also suggested we try a 'class balance cross entropy loss function' which would penalise the non-prediction of a fire more than the prediction of one when there is no fire. This may help with our model just predicting no fires.

[sofstef]

@Hamish-Cam re choice of loss: we are currently using jaccard loss which is suited for tackling our type of problem with very few pixels having fire. I have also added an option to use focal tversky loss, which is essentially a generalisation of Jaccard and allows for choosing weight parameters to penalise false negatives more + has an additional parameter which can be used to force the network to focus on pixels where it's struggling to make predictions. Will try the tversky loss out now and push the code so you can use it in colab!

[Hamish-Cam]

@Hamish-Cam re choice of loss: we are currently using jaccard loss which is suited for tackling our type of problem with very few pixels having fire. I have also added an option to use focal tversky loss, which is essentially a generalisation of Jaccard and allows for choosing weight parameters to penalise false negatives more + has an additional parameter which can be used to force the network to focus on pixels where it's struggling to make predictions. Will try the tversky loss out now and push the code so you can use it in colab!

Brill, sounds like this is pretty well covered then. Trying to overfit might still be a good test to run. Thanks!

[graceebc9]

Proposed for who writes what:

Intro / lit review: Thomas
Datasets - grace
Methodology - Hamish or Sofija
Results - Hamish or Sofija
Conclusion & further - Grace

[graceebc9]

had an issue with training on sentinel - note that 'his seems related to the following bug reports. Basically, the UNet that comes with SMP requires images with patch_size divisible by 32. Can you try switching from 250 to 256 and see if that solves your issue? -- switching to 256 solved the issue.