this post was submitted on 20 Dec 2023
17 points (100.0% liked)

LocalLLaMA

2268 readers
1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago
MODERATORS
SkySyrup
pax
noneabove1182
17
GitHub - SJTU-IPADS/PowerInfer: High-speed Large Language Model Serving on PCs with Consumer-grade GPUs (github.com)
submitted 11 months ago by [email protected] to c/localllama
1 comments fedilink hide all child comments
 

Discussion with one of the paper authors in llama.cpp: https://github.com/ggerganov/llama.cpp/discussions/4534

Thread by (apparently) a paper author on Reddit: https://www.reddit.com/r/LocalLLaMA/comments/18luk10/wait_llama_and_falcon_are_also_moe/

you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 3 points 11 months ago

Amazing speedup, too bad there is no amd GPU support yet.