RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)

[gwen233666]

Traceback (most recent call last):
File "train.py", line 527, in
training(lp.extract(args), op.extract(args), pp.extract(args), dataset, args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, wandb, logger)
File "train.py", line 87, in training
scene = Scene(dataset, gaussians, ply_path=ply_path, shuffle=False)
File "/home/huigang/Scaffold-GS/scene/init.py", line 82, in init
self.train_cameras[resolution_scale] = cameraList_from_camInfos(scene_info.train_cameras, resolution_scale, args)
File "/home/huigang/Scaffold-GS/utils/camera_utils.py", line 59, in cameraList_from_camInfos
camera_list.append(loadCam(args, id, c, resolution_scale))
File "/home/huigang/Scaffold-GS/utils/camera_utils.py", line 53, in loadCam
image_name=cam_info.image_name, uid=id, data_device=args.data_device)
File "/home/huigang/Scaffold-GS/scene/cameras.py", line 56, in init
self.full_proj_transform = (self.world_view_transform.unsqueeze(0).bmm(self.projection_matrix.unsqueeze(0))).squeeze(0)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)

Hello, thank you for your excellent work, but I have encountered the following problems during training. Could you please give me some advice? Thank you in advance.

[inspirelt]

Hi, I would suggest you to try to change the pytorch and cuda version.

[Xun2001]

Hello, I also encountered the same problem. My CUDA version is 11.6, and Torch is also the corresponding version of 11.6. Have you solved this problem?

[Xun2001]

Here is my CUDA and Torch version

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver  
Copyright (c) 2005-2021 NVIDIA Corporation  
Built on Fri_Dec_17_18:16:03_PST_2021  
Cuda compilation tools, release 11.6, V11.6.55  
Build cuda_11.6.r11.6/compiler.30794723_0

>>>  import torch  
>>> print(torch.version.cuda)  
11.6
>>> print(torch.__version__)
1.12.1

[Xun2001]

I have solved this problem by unset LD_LIBRARY_PATH