try openorca-mistral-7b, it should fit in your GPU. Try using exllama2 to speed up interference.
thx, this one? https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GPTQ
yeah that should work!
Yes it does and fits the GPU just fine. Didn't hallucinate but it was slow like 60s+ in the first run but did it's job. Thanks.
good to hear it worked, it’s weird it’s so slow. I’m lucky to have access to a 3060, which isn’t that far out from a 1080, and get at least 40t/s on it. Are you running on CPU or are you using exllama?
It's running on gpu, the task-manager shows 92% GPU utilization and i chose exllamaV2.
that’s really weird, I’m not sure how to help you there unfortunately :(
Community for Oobabooga / Pygmalion / TavernAI / AI text generation
Let's rebuild our knowledge base here!
Ooba community is still dark on reddit, so we're starting from scratch. https://www.reddit.com/r/Oobabooga/
Subscribe, engage, post, comment!
Helpful links:
https://github.com/oobabooga/text-generation-webui
https://zoltanai.github.io/character-editor/
https://www.chub.ai/characters
Remember to mark NSFW posts.
Other AI communities to check out: