REQUEST: gfx803 support

[chboishabba]

Hiya!

I got pretty excited seeing this, as gfx803 is generally not supported in recent ROCm and I presumed it would be included in this project.

I'm running on RX580 which was highly sought after during GPU shortages of COVID... I believe latest supported ROCm for gfx803 and gfx900 is 5.4.2

https://github.com/jrcichra/rocm-pytorch-gfx803
https://github.com/xuhuisheng/rocm-build/tree/develop/gfx803
https://wiki.archlinux.org/title/AMD_Radeon_Instinct_MI25#ROCm

[lamikr]

I have newer had that generation of AMD cards by my self, so I have not never tried to add the support for that. If you have time to try, I could first try to add some very basic component support for it to some build branch.

Build would propably fail at some point but at least we could try to test whether rocminfo and amd-smi will detect the card and and also test whether hipcc and opencl apps works. And maybe also llama.cpp, vllm. etc which does not require whole stack like pytorch with gpu acceleration to work.

[chboishabba]

Happy to run it for you. I've successfully run rocm stuff using I think the xui persons docker but rocm version too low and my main usage is whisperx which uses ctranslate. It would definitely be great to be able to run llama locally, especially on GPU so CPU doesn't get bogged down, eg as code assistant. As far as I understood, the process was to to basically just have ROCm reasonably updated so one can build rocm pytorch from source (or ideally just use the appropriate existing rocm-pytorch version). There is also https://github.com/arlo-phoenix/CTranslate2-rocm This would be the ultimate ideal, as faster-whisper and whisperx (vs vanilla whisper) use ctranslate, and simply by doing so, become competitive on CPU (whisperx ctranslate2 cpu) what was already fast on gpu (whisper) Running ctranslate on the GPU would suggest a significant uplift in performance.

…

On Thu, Nov 7, 2024, 7:23 PM Mika Laitio ***@***.***> wrote: I have newer had that generation of AMD cards by my self, so I have not never tried to add the support for that. If you have time to try, I could first try to add some very basic component support for it to some build branch. Build would propably fail at some point but at least we could try to test whether rocminfo and amd-smi will detect the card and and also test whether hipcc and opencl apps works. And maybe also llama.cpp, vllm. etc which does not require whole stack like pytorch with gpu acceleration to work. — Reply to this email directly, view it on GitHub <#173 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGM4B3QIGRYH44HH4HD67IDZ7MWRVAVCNFSM6AAAAABRCHV7OGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRRG4ZDAMZUG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>