Error during Training RuntimeError: mat1 and mat2 shapes cannot be multiplied (511x2 and 3x4096)

[kasakh]

I followed the installation steps and used the example-data-maya-wiki.txt dataset as a finetuning examples. I keep getting this error towards the end of training. I tried with other datasets as well still the same error persists

output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (511x2 and 3x4096)

Can you please help me resolve it? Thanks in advance.

[kasakh]

It seems that this only happens when I try using Multi-gpu set up. On Single Gpu, it works fine.

[lxe]

Yeah, I basically hardcoded it for single GPU. The tloen/alpaca-lora code has multi-gpu support.