LocalLLaMA

2950 readers

8 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

founded 2 years ago

MODERATORS

is the 4k context length of llama2 for real? (self.localllama)

submitted 2 years ago by actuallyacat to c/localllama

13 comments fedilink hide all child comments

I've been using airoboros-l2-70b for writing fiction, and while overall I'd describe the results as excellent and better than any llama1 model I've used, it doesn't seem to be living up to the promise of 4k token sequence length.

Around 2500 tokens output quality degrades rapidly, and either starts repeating previous text verbatim, or becomes incoherent (grammar, punctuation and capitalization disappear, becomes salad of vaguely related words)

Any other experiences with llama2 and long context? Does the base model work better? Are other fine tunes behaving similarly? I'll try myself eventually, but the 70b models are chunky downloads, and experimentation takes a while at 1 t/s.

(I'm using GGML Q4_K_M on kobold.cpp, with rope scaling off like you're supposed to do with llama2)

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 2 points 2 years ago* (last edited 2 years ago)

I read some other people complain, too. Maybe try the base model. I'm not sure if it's the fine-tune or llama2's fault.

There are ways to measure that. To measure perplexity across the context. And whatever people did to measure if the things went into the right direction that increased the first llama context size past 2048. But I didn't find measurements for Llama2 at least with a quick google .

Edit: And people mentioned Llama2 has a different attention mechanism at the 70B version. This also might be specific to the 70B version. Make sure to use the most recent version of KoboldCPP or whatever you use and to configure the scaling correctly. At 4096 it shouldn't need any context scaling as far as i understand.