How much time have you yourself spent trying to construct a system which applies LLMs to AGI problems? For me, it's a few full work days' worth of tinkering -- not much, but enough that I feel like I have some real-world perspective on what's involved. Please be careful about talking down to me about my need to broaden my understanding.

So I did watch part of Bubeck's talk quite a while ago when it first came out, because this is a deeply important topic to me. I don't fully agree with it; so let me take an excerpt from his paper to explain what I disagree with:

A question that might be lingering on many readers’ mind is whether GPT-4 truly understands all these concepts, or whether it just became much better than previous models at improvising on the fly, without any real or deep understanding. We hope that after reading this paper the question should almost flip, and that one might be left wondering how much more there is to true understanding than on-the-fly improvisation. Can one reasonably say that a system that passes exams for software engineering candidates (Figure 1.5) is not really intelligent? Perhaps the only real test of understanding is whether one can produce new knowledge, such as proving new mathematical theorems, a feat that currently remains out of reach for LLMs.

To me, this is a pretty clear statement of the core of what Bubeck is saying, both in the paper and the talk: He goes through a very accurate list of the unbelievably impressive things that GPT-4 can do. Then, he says more or less that because it can do those things, it must be intelligent (or at least have the first sparks of real intelligence). To me, I simply don't agree with that. Computerized systems could already do extremely impressive computational things; extending that into the domain of language is a huge leap forward, maybe towards AGI. But, if you're going to say that because it can mimic the language of reasoning, it must be able to reason, because there's in practice no difference between those things, then I don't agree with that.

Could we use LLMs as building blocks for a real AGI system? Yes, absolutely; like I say, I've spent a slight but nonzero amount of time actually experimenting with that myself. Are LLMs impressive? Fuck yes. Are they necessarily intelligent because they can do these impressive language-related tasks? To me, no. That seems like a non sequitur. To me, it's still clear interacting even with GPT-4 that it doesn't have real understanding of the underlying concepts, and these models are just getting better and better at moving symbols around. Again, I'd actually contrast that specifically against things like AlphaZero, where it does have a deep understanding of the underlying concepts, to the point that it can easily arrive at novel ideas on its own, beyond and superior to what it was programmed with.