I think it's pretty natural for people to confuse the way mechanisms of communication are used with inherent characteristics of the entity you're communicating with: "If it talks like a medical docture then surelly it's a medical doctor".

Only that's not how it works, as countless politicians, salesmen and conmen have demonstrated - no matter how much we dig down intonsubtle details, comms isn't really guaranteed to tell us all that much about the characteristics of what's on the other side - they might be just lying or simulating and there are even entire societies and social strata educated since childhood to "always present a certain kind of image" (just go read about old wealth in England) or in other words to project a fake impression of their character in the way they communicate.

All this to say that it doesn't require ill intent for somebody to go around insisting that LLMs are intelligent: many if not most people are trying to read the character of a subject from the language the subject uses (which they shouldn't but that's how humans evolved to think in social settings) so they trully belive that what produces language like an intelligent creature must be an intelligent creature.

They're probably not the right people to be opinating on cognition and inteligence, but lets not assign malice to it - at worst it's pigheaded ignorance.

I think you're misreading the point I'm trying to make. I'm not arguing that LLM is AGI or that it can understand anything.

I'm just questioning what the true use case of AGI would be that can't be achieved by existing expert systems, real humans, or a combination of both.

Sure Deepseek or Copilot won't answer your legal questions. But neither will a real programmer. Nor will a lawyer be any good at writing code.

However when the appropriate LLMs with the appropriate augmentations can be used to write code or legal contracts under human supervision, isn't that good enough? Do we really need to develop a true human level intelligence when we already have 8 billion of those looking for something to do?

AGI is a fun theoretical concept, but I really don't see the practical need for a "next step" past the point of expanding and refining our current deep learning models, or how it would improve our world.