this post was submitted on 25 Oct 2024
73 points (98.7% liked)

TechTakes

1437 readers
109 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 1 year ago
MODERATORS
 

Hermansson logged in to Google and began looking up results for the IQs of different nations. When he typed in “Pakistan IQ,” rather than getting a typical list of links, Hermansson was presented with Google’s AI-powered Overviews tool, which, confusingly to him, was on by default. It gave him a definitive answer of 80.

When he typed in “Sierra Leone IQ,” Google’s AI tool was even more specific: 45.07. The result for “Kenya IQ” was equally exact: 75.2.

Hmm, these numbers seem very low. I wonder how these scores were determined.

top 13 comments
sorted by: hot top controversial new old
[–] [email protected] 14 points 1 month ago (1 children)

Hmm, these numbers seem very low. I wonder how these scores were determined.

They weren't, because LLMs don't have reasoning ability, at least not in the way you as a human do. They are generative models, so the short answer is the model most likely made the numbers up, though there's a chance they pulled them directly from some training data that's likely completely unrelated to the user's prompt.

What they generate is supposed to have similar multidimensional correlation as the input data, so there are complex relationships between what the question asked and the output it gave, but these processes don't look anything like the steps you would go through to answer the same question.

[–] [email protected] 20 points 1 month ago (1 children)

so the short answer is the model most likely made the numbers up

Right crime, wrong perp. The Google overview "correctly" sourced the IQ scores from one of Arthur Jensen's "studies" where he reports the IQ of every country in the world and fully makes up numbers for over a third of them

[–] [email protected] 7 points 1 month ago

I expected the same, except the names being Richard Herrnstein and Charles Murray and/or Richard Lynn and Tatu Vanhanen.

[–] [email protected] 12 points 1 month ago* (last edited 1 month ago)

"garbage in, garbage out" my beloathed

Not the first time this has happened Google's own AI overviews have misinterpreted u/fucksmith, eaten rocky onions and hallucinated cats on the moon before) but this is probably the worst such incident

Anyways, sidenote time:

Right now, there's no legal precedent determining whether or not "AI overviews" like Google's are protected under Section 230, but between shit like this and the recent lawsuit against character.ai, I suspect there's gonna be plenty of effort to deny them Section 230 protection.

If that happens, I expect it will put an immediate end to public-facing autoplag like this, as such products immediately become legal timebombs waiting to go off. I suspect it will also kill any future attempts at AI for the foreseeable future, for similar reasons.

As for AI as a concept, which I've discussed previously, I expect this incident will help further a public notion of "artificial intelligence" being an oxymoronic concept, and of intelligence being something that either cannot be replicated by artificial means, or something which should not be replicated by artificial means.

[–] [email protected] 5 points 1 month ago (4 children)

I don't understand the title. LLM hallucinations have nothing to do with JAQing off.

[–] [email protected] 22 points 1 month ago* (last edited 1 month ago) (1 children)

Problem it wasn't a hallucination - it was referencing a paper that has been debunked. These aren't made up numbers, they're VERY specific numbers that come from a VERY specific paper.

This one: https://www.sciencedirect.com/science/article/abs/pii/S0160289610000450 -- If I'm not mistaken. Created by a Nazi Sympathizer Richard Lynn and the Pioneer Fund

The problem is that this also managed to get cited more than 22,000x creating a feedback effect that reinforced the AI's learning.

[–] [email protected] 1 points 1 month ago (1 children)

Okay, but it's still got nothing to do with the dishonest rhetorical technique called "JAQing off" (a.k.a. "Just Asking Questions," a.k.a. "sealioning").

[–] [email protected] 2 points 1 month ago

It's kind of a ... symptom ... of the community we're in. I wouldn't read into it too deeply.

[–] [email protected] 5 points 1 month ago (1 children)

I think the usual output from the AI Overview (or at least the goal) is to give a long and ostensibly Fair and Balanced summary. So in this case it would be expected to throw out "some say that people from Australia are extra dumb because of these studies, but others contend that those studies were badly performed" or whatever. Asking the question on more words to represent both sides so that it can pretend not to be partisan.

[–] [email protected] 1 points 1 month ago (1 children)

Let me be more clear about this: an LLM trying to answer a question (successfully or otherwise) is doing basically the opposite of a human asking questions (disingenuously, as in "JAQing off," or otherwise).

I wasn't trying to solicit comments trying to explain what the LLM was doing; my point was simply that OP is confused and used a term incorrectly in the title.

[–] [email protected] 8 points 1 month ago

i like turtles

[–] [email protected] 2 points 1 month ago* (last edited 1 month ago)

It's a reference to the fact that the kind of person who would try and justify this sort of race science is also the kind of person who is "just asking questions." Combined with the tech industry's tepid "it's just a tool, it's not inherently evil" bullshit, I think OPs point is obvious to anyone who isn't a pedant, deliberately acting in bad faith.

[–] [email protected] 2 points 1 month ago

you may wish to read the sidebar