LocalLLaMA

2293 readers

1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago

MODERATORS

SkySyrup

pax

noneabove1182

Update to my text-generation-webui docker image - now with ExLlama support! (hub.docker.com)

submitted 2 years ago by noneabove1182 to c/localllama

4 comments fedilink hide all child comments

Took me some time to figure this one out, and unfortunately requires a significantly larger image (need so much more of nvidia's toolkit D: couldn't figure out a way to get around it..)

If people prefer a smaller image, I can start maintaining one for exllama and one without, but for now 1.0 is identical minus exllama support (and I guess also from an older commit) so you can use that one until there's actual new functionality :)

top 4 comments

sorted by: hot top controversial new old

[–] noneabove1182 1 points 2 years ago

Also, do note that the model needs to be made with gptq-for-llama, not autogtpq

[–] [email protected] 1 points 1 year ago (1 children)

Hi there. I'd love to try your docker images. I just tried your latest image on my M2 MacbookPro but I don't know what to do next. I created a folder and typed the following in the terminal: docker pull noneabove1182/text-gen-ui-gpu:latest

But I don't know what to do next to launch it. Sorry for the basic question, but how do I run it? Thanks!

[–] noneabove1182 1 points 1 year ago

Yeah no problem! First issue however is that apple silicon plays kinda funny with this kind of setup, so I may need to make you a custom image to use. Otherwise you should have no problem running the -cpu build.

As for running the image itself, you can either run it form the command line with docker run, or you can make yourself a docker compose file

I personally tend to go the latter, and for that you can copy my docker-compose.yml file from here: https://hub.docker.com/r/noneabove1182/text-gen-ui-cpu

I'll work on making a mac-specific image and you can test it for me ;)

[–] [email protected] 1 points 1 year ago

Thank you so much! I’d be happy to test it out.