smooth_output=True causing long run time?

[NoaMillsUSDA-ARS]

I ran two variations of a model, one with discrete prediction outputs (smooth_output=False in SemanticSegmentationLabelStoreConfig) and one with continuous prediction outputs (smooth_output=True). For some reason, the continuous prediction version takes a really long time to run, specifically in the prediction phase. I'm working on a compute cluster, and the discrete version runs in under 4 hours. I kept increasing the time limit for the continuous version up to 33 hours, but even that was not enough time to complete the prediction stage. When running the continuous version with a 4 hour time limit, it kills the process during the prediction stage, so this is the stage that is causing delays. Why does this happen, and is there anything I can do to make it run faster?

[AdeelH]

Which RV version? I'm guessing those are numbers for the time it takes to run on the entire dataset. Can you provide numbers for a single scene and also how big the scene is?

[NoaMillsUSDA-ARS]

Update. I ran the model with a smaller validation raster (1312 pixels by 1320 pixels, 4 bands), and it took 1 day and 15 hours to run with 1 training epoch. It did complete the prediction stage, however it errored out when trying to store the continuous prediction values. My traceback is below. Note that I am using the pytorch-fpn model.

First question: why is it taking so long to execute the prediciton stage?

Second question: It looks like there is a mismatch between int16 and int32 values. Should I cast the int16 values into int32? Where would I do that?

`................................................................................................................................................................................................2022-11-19 02:20:13:rastervision.core.data.label_store.semantic_segmentation_label_store: INFO - Writing labels to disk.

2022-11-19 02:21:48:rastervision.core.data.label_store.semantic_segmentation_label_store: INFO - Writing smooth labels to disk.

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/opt/src/rastervision_pipeline/rastervision/pipeline/cli.py", line 248, in
main()
File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/opt/src/rastervision_pipeline/rastervision/pipeline/cli.py", line 240, in run_command
runner=runner)
File "/opt/src/rastervision_pipeline/rastervision/pipeline/cli.py", line 217, in _run_command
command_fn()
File "/opt/src/rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py", line 184, in predict
for s in dataset.validation_scenes
File "/opt/src/rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py", line 180, in _predict
label_store.save(labels)
File "/opt/src/rastervision_core/rastervision/core/data/label_store/semantic_segmentation_label_store.py", line 201, in save
chip_sz=self.rasterio_block_size)
File "/opt/src/rastervision_core/rastervision/core/data/label_store/semantic_segmentation_label_store.py", line 233, in write_smooth_raster_output
self._write_array(dataset, window, score_arr)
File "/opt/src/rastervision_core/rastervision/core/data/label_store/semantic_segmentation_label_store.py", line 349, in _write_array
dataset.write_band(i, band, window=window)
File "rasterio/_io.pyx", line 1363, in rasterio._io.DatasetWriterBase.write_band
File "rasterio/_io.pyx", line 1322, in rasterio._io.DatasetWriterBase.write
ValueError: the array's dtype 'float16' does not match the file's dtype 'float32'
/opt/data/mini_test/Makefile:9: recipe for target '1' failed
make: *** [1] Error 1`

[AdeelH]

• Did it take 1 day and 15 hours for the training or just for the predictions? That long for predicting (or even training) for a 1300x1300 scene does not make sense. It shouldn't even take that long if running on a CPU.
• This error is a bug that was later fixed on master.
  • You can work around this in 0.13 by passing smooth_as_uint8=True to SemanticSegmentationLabelStoreConfig (the same place where you pass smooth_output=True). This will save convert the float predictions to uint8 when saving, which takes up less space. It will map 0-1 to 0-255. If you ever want the original probabilities back, you can divide the values by 255.
  • Once you make the above change, you do not need to rerun the training. You can directly run the predict stage.

[NoaMillsUSDA-ARS]

It took 1 day and 15 hours including the training procedure, although the vast majority of that time was spent doing predictions. In my initial trials, when I set the time limit to 3 hours, it would time out in the prediction phase so I know it finished the training stage comparatively quickly. I was able to fix the previous error with the smooth_as_uint8=True but it's still taking a really long time to run the prediction stage.

[AdeelH]

You might want to try parallelizing the predictions via the --splits option: https://docs.rastervision.io/en/0.13/cli.html?highlight=splits#splits .

Upgrading to master will also help speed this up a bit because it RV now uses multiprocessing for data-loading unlike 0.13.

[NoaMillsUSDA-ARS]

Update: It works as expected in version 0.20! My runtime for continuous prediction was slightly lower than my runtime for discrete predictions, which makes sense.