LocalLLaMA

2593 readers

4 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago

MODERATORS

SkySyrup

pax

noneabove1182

More docker images! Now for oobabooga's text-generation-webui, CPU and GPU versions (hub.docker.com)

submitted 2 years ago by noneabove1182 to c/localllama

1 comments fedilink hide all child comments

Main link is to GPU image, CPU image can be found here:

https://hub.docker.com/r/noneabove1182/text-gen-ui-cpu

The CPU one is built for exclusively running with a CPU. The GPU one is compiled with CUDA support and gets blazing fast ingestion and generation.

Included in each readme is a disclaimer that I am once again not affiliated, and I include an example working docker-compose.yml, make sure you change the args to fit your own personal setup! :)

Feel free to ask any questions or let me know if anything doesn't work! Hacked it together by the skin of my teeth, and put a LOT of effort into reducing image size for the GPU one (16GB down to 9GB, still massive..) so please do post if you have any issues!

top 1 comments

sorted by: hot top controversial new old

[–] noneabove1182 1 points 2 years ago

I'm also working on getting koboldcpp working with GPU support, currently it works for ingestion but for some reason the generation itself is still pretty slow.. will post an update when it's working well!