this post was submitted on 27 Jul 2023
9 points (90.9% liked)

LocalLLaMA

2292 readers
1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago
MODERATORS
SkySyrup
pax
noneabove1182
9
Huggingface Text Generation Inference adds exllama support (github.com)
submitted 1 year ago by noneabove1182 to c/localllama
0 comments fedilink hide all child comments
 

This is actually a pretty big deal, exllama is by far the most performant inference engine out there for CUDA, but the strangest thing is that the PR claims it works for starcoder which is a non-llama model:

https://github.com/huggingface/text-generation-inference/pull/553

So I'm extremely curious to see what this brings...

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here