Community to discuss about LLaMA, the large language model created by Meta AI.
This is intended to be a replacement for r/LocalLLaMA on Reddit.
To do general purpose GPU calculations on AMD hardware you need a GPU that is supported by ROCm (AMD's equivalent to CUDA). Most of the gaming GPUs are not.
There is a list here but be aware that that is for the latest rocm version, some tools might still use older versions with different supported devices.
Has that changed recently? I've ran ROCm successfully on an RX6800. I seem to recall that was supported, the host OS (Arch) was not.
No, GFX1030 is still supported.
When I tried it maybe a year or so ago there were four supported chipset in that version (5.4.2 I think) of rocm but I don't remember which card models those were since they were only specified in that internal chip name. Mine wasn't supported at the time (5700XT)
This link is misleading. For example, the Radeon RX6800 IS supported because it is the same chip as one of the Radeon Pros. GFX1030. Many others are too…though support does not go very far back.
Llama.cpp supports OpenCL as well and performs better than rocm in my limited experience. That should work on basically any GPU.
Latest ollama has support for AMD GPUs. I had to compile from source to make it pick up the GPU on my system.
Look into llamafile. It makes things so easy.
Basically, RoCM and CUDA allows one to do math on the GPU. Most Linear Algebra operations (i.e. LLM or NNs and ML generally) can be parallelized over a GPU which is much more performant than CPU.
To perform calculations on GPU, one needs some sort of interface to to their programming language of choice, NVIDIA has CUDA which is in CPP with bindings to python: (pytorch, Tensorflow etc. ), Julia: Flux etc.
RoCM is AMDs solution, there bindings are young and not widely implemented.
My advice, play around with Flux RoCM and PyTorch RoCM just to get an idea. Suffice it to say, when I started doing RL and LLMs more seriously I gave up my colab and sold my AMDs to fund a 3060.
llama.cpp (and ollama) has AMD support through ROCm and also now Vulkan.