this post was submitted on 26 Oct 2023
12 points (100.0% liked)

Oobabooga Text Generation

6 readers
1 users here now

Community for Oobabooga / Pygmalion / TavernAI / AI text generation

Let's rebuild our knowledge base here!

Ooba community is still dark on reddit, so we're starting from scratch. https://www.reddit.com/r/Oobabooga/

Subscribe, engage, post, comment!

Helpful links:

https://github.com/oobabooga/text-generation-webui

https://zoltanai.github.io/character-editor/

https://www.chub.ai/characters

Remember to mark NSFW posts.

Other AI communities to check out:

[email protected]

founded 1 year ago
MODERATORS
[email protected]
12
LLM for GTX1080 (8GB) for local use. (lemmy.dbzer0.com)
submitted 10 months ago by [email protected] to c/[email protected]
7 comments fedilink hide all child comments
 

Could someone recommend a LLM for the Nvidia GTX1080? I've used the gptq_model-4bit-128g of Luna AI from the Bloke and i get a response every 30s-60s and only 4-5 prompts before it starts to repeat or hallucinate.

you are viewing a single comment's thread
view the rest of the comments
[–] SkySyrup 4 points 10 months ago (1 children)

try openorca-mistral-7b, it should fit in your GPU. Try using exllama2 to speed up interference.

[–] [email protected] 2 points 10 months ago (1 children)
[–] SkySyrup 3 points 10 months ago (1 children)

yeah that should work!

[–] [email protected] 2 points 10 months ago (1 children)

Yes it does and fits the GPU just fine. Didn't hallucinate but it was slow like 60s+ in the first run but did it's job. Thanks.

[–] SkySyrup 2 points 10 months ago (1 children)

good to hear it worked, it’s weird it’s so slow. I’m lucky to have access to a 3060, which isn’t that far out from a 1080, and get at least 40t/s on it. Are you running on CPU or are you using exllama?

[–] [email protected] 1 points 10 months ago (1 children)

It's running on gpu, the task-manager shows 92% GPU utilization and i chose exllamaV2.

[–] SkySyrup 2 points 10 months ago

that’s really weird, I’m not sure how to help you there unfortunately :(