About the issue of multi-GPU training.

[do1nothing]

My server has four NVIDIA 4090 GPUs. Single-card training doesn't throw any errors, but when the batch size is changed to 2 for single-card training, it throws an error after completing just one epoch. No other parameters have been changed. I wanted to try multi-GPU training, but it keeps throwing errors. I searched online for solutions, but none of them seem to resolve the issue. The error message is as follows:
Traceback (most recent call last):
File "./train.py", line 186, in
trainer.train()
File "../../tasks/semantic/modules/trainer.py", line 280, in train
show_scans=self.ARCH["train"]["show_scans"])
File "../../tasks/semantic/modules/trainer.py", line 391, in train_epoch
output = model(in_vol)
File "/root/anaconda3/envs/salsanext/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/root/anaconda3/envs/salsanext/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 156, in forward
"them on device: {}".format(self.src_device_obj, t.device))
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1