main thread not in main loop error when RV pipeline plots sample train/validation images

[aerotractjack]

Hi, I am using the pipeline configuration of rastervision for my object detection problem. Things are looking good, but we keep getting an error main thread not in main loop when RV tries to plot sample training or validation plots. I have seen this error in the past on different projects, and always fixed it by replacing matplotlib's backend with something non-giu (e.g. agg) by doing

import matplotlib
matplotlib.use("agg")

But with RV, this does not seem to prevent the error. If it makes a difference, I am running my code through a shell script, which is really just a shortcut for the RV command

#!/bin/bash
train() {
    rastervision run local train.py -a config $1
}
predict() {
    python3 predict.py $1
}
$@

# ex. ./rv.sh train /path/to/train_config.yaml

I seem to have prevented this error from occurring by adding a keyword preview_batch_limit=0 to my GeoDataConfig object. This is not the best solution for me, as this prevents any sample plots from being generated, which I want to see.

    data = ObjectDetectionGeoDataConfig(
            ...,
            preview_batch_limit=0)

What is my best option here? Thank you

[lewfish]

Which version of RV are you using and what environment are you running this in? Where did you make the change to use agg? Could you share the config? This will help me try to replicate the problem.

[aerotractjack]

RV versions:

rastervision==0.20.2
rastervision-aws-batch==0.20.2
rastervision-aws-s3==0.20.2
rastervision-core==0.20.2
rastervision-pipeline==0.20.2
rastervision-pytorch-backend==0.20.2
rastervision-pytorch-learner==0.20.2

Python: version 3.9.16 via minconda

My config:
Full code: config repo

I import matplotlib before anything else:

import matplotlib as plt
plt.use("agg")
from rastervision.core.data import ClassConfig, DatasetConfig, SceneConfig
from rastervision.core.data.raster_source.rasterio_source_config import RasterioSourceConfig
...

And here is the relevant dataset config

    data = ObjectDetectionGeoDataConfig(
            scene_dataset=scene_dataset,
            window_opts=window_opts,
            img_sz=input_config["window_config"]["img_sz"],
            num_workers=input_config["num_workers"],
            augmentors=['RGBShift', 'RandomRotate90', 'HorizontalFlip', 'VerticalFlip'],
            preview_batch_limit=0)

[lewfish]

Putting

import matplotlib as plt
plt.use("agg")

at the start of your config file isn't going to work as expected if you are using the local runner (by passing local to rastervision run). This is because the local runner starts a separate Python processes for each command. To get the effect you are looking for you will need to use the inprocess runner by passing inprocess. This will run everything within the same Python process that is interpreting the config file and running plt.use("agg").

That might be a quick fix, but I still wouldn't expect this to occur at all.

[aerotractjack]

Ah, thank you!