low volatile gpu-util in data loading

[XLIU-hub]

Congratulation on your work. We would like to adopt this method to a multi-outputs network to detect diseases in liver. We face a problem when loading dataset, maybe you had the same problem and could you share your experience.

I understand that openslide and DataLoader are used to load tiles. With multiple workers and data stored in local disk, the GPU usage is very high, while the volatile GPU-Util stays at 0 most of the time, jumps to 100 for a few seconds. This results in the training stage to be quite long. Any suggestion to solve this bottleneck? Thanks in advance.