LocalLLaMA

Thanks. Whoo boy are there parts I didn't understand. Sorry for asking if you did the summary yourself. I really couldn't tell if it's my intellectual abilities to follow what you said.... or if that paragraph just didn't make any sense because it was written by ChatGPT. So I just asked. (I still think it's not quite right. Both methods quantize the model in the same way (contrary to what you said). The thing is QA-LoRA is able to keep it quantized during the whole process and Q-LoRA not.)

I think my knowledge level is about undergraduate computer science, without having attended any course on machine learning. But I like to read those papers and play with the stuff.

I think I got it now. The important explanation is in 3.2 and 3.3:

• "In brief, the only benefit brought by QLoRA is the reduced memory cost for fine-tuning."
• "achieving the second goal lies in that W˜ (i.e., the quantized W) and s·AB can be merged without using high-precision numbers (e.g., FP16)."
• "this is impossible in the original setting, i.e., W is quantized into W˜ in a column-wise manner while both A and B are unconstrained."

And then they invent the maths and group the operations and numbers in a way to make it possible. In result reducing the complexity of an important part of the calculation and keeping it within the same datatype, and successfully avoiding another lossy quantization step.

I hope I'm right. Because I had to work hard and read it several times to get to this point.

I have one serious question left:

I don't have a solid understanding of the consequences of using quantized numbers in (fine-)tuning. I always thought we needed a tiny learning rate for the gradient decent in machine learning. With more "coarse" quantized numbers, I thought that limits your ability to do small adjustments. Isn't that how discrete number work? And like 4 bits isn't much. It's just 16 different values. I don't get why the loss function converges under these circumstances. I'd expect the error to be significant at some point and making the process not stable. But obviously it is possible.

And I have so many questions regarding the wording of this specific paper. The paper is a bit weird to me. Every few sencences, I get lost. You don't need to read the rest of this comment. It's more or less just my incomprehension of why I cannot understand the authors of that specific paper.

I completely don't get why they start their abstract with the words: "Recently years have witnessed a rapid development of large language models (LLMs)."

No shit Sherlock? Who are they expecting to read this? Did they need to come up with a few more words for their abstract for some reason? And why a grammar/spelling mistake in the very first word of a paper?

While having a look at their bibliography, I noticed it's fairly common to open your paper with a stupid sentence like "Recent advances in ... have ...". But usually it's a bit more focused than referencing what broadly happened in the world in the last 5 years.

And then in the introduction they just drop random papers. I don't understand the connection to "Language Models are Few-Shot Learners" and "BLOOM: A 176B-Parameter Open-Access Multilingual Language Model". But I guess I'm not a scientist and never wrote a paper myself.

Then they introduce the concept of parameter efficient fine tuning (in this case LORA) and quantization. I just skimmed through the QLoRA paper again, and I can understand those explanations (section "2 Background") way better.

And then they drop what I'd expect are the two most influential papers for this subject: "LLM-QAT: Data-Free Quantization Aware Training for Large Language Models" (arXiv:2305.17888) and "QLoRA: Efficient Finetuning of Quantized LLMs" (arXiv:2305.14314).

And only then they say: "To the best of our knowledge, the most related work is QLoRA (Dettmers et al., 2023a)". So they completely left me confused until the very end of section 2. I'm not an expert in this field. And my memory is bad. So upon first reading the paper, I always asked myself if I remember correctly that I've read something like this already. Or if it was some incohesive ramblings on Reddit, back in the time.

And why are they constantly comparing it to LORA as if QLORA wasn't the leading paper to compare oneself against? They don't even mention it in "Related Work" until the very end in the only paragraph that has no bold text in it and I probably didn't read it while skimming through the paper for the first three times. Isn't the way science works to say what's current state of scientific research and then say how you were able to improve on that? Why reference some arbitrary state in history of science instead of the current state?

I'm not able to follow the maths. But the diagram "Figure 2" looks like QLoRA and QA-LoRA are almost the same, except for the magic with the D_in and L that makes it take less multiplications and the direct output of INT4 weights. This was the figure that lead me to the conclusion that I needed to dig further.

I'm sorry for rambling on and on. Most of the time I can get something out of these papers, despite not being educated properly in machine learning. This was one of the few papers that left me completely without a clue and dumbfounded. Guess my brain is wired differently than the authors' brains.

I still don't get why they explain things like they do. For example the abstract is a few sentences of unnecessary stuff every 8 year old knows. And then they immediately leap into super specific math details of one aspect, with "the imbalanced degrees of freedom of quantization", talk about their "group-wise operators which increase the degree of freedom of quantization meanwhile decreasing that of adaptation". Which I find ridiculous. It is why their proposed approach mathematically works. But they haven't yet explained what their approach even is or what they're trying to do in the first place and why this is important. Sure: "edge devices". But there's quite a bit missing. If you can infer how low rank adaptation is connected to edge devices, you don't really need them to teach you the importance of LLMs. Then they say how many lines of code they need, which is a bit out of place. And then state their acheivements upon LoRA, which I guess is fine. So is mentioning they applied it and are releasing the code.