this post was submitted on 10 Jan 2025
12 points (92.9% liked)

LocalLLaMA

2552 readers
10 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago
MODERATORS
SkySyrup
pax
noneabove1182
12
How much gpu do i need to run a 90b model (lemm.ee)
submitted 1 month ago by [email protected] to c/localllama
16 comments fedilink hide all child comments
 

Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 2 points 1 month ago

The biggest issue will be your VRAM. If you don't have enough of it (which is very likely, even the 8B models I use need ~10gb), you'll have to use a GGUF model which will need to use your system RAM and CPU for the parts that don't fit in the VRAM, which will heavily slow it down.