Technology

59958 readers

3476 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 2 years ago

MODERATORS

[email protected]

494

Apple study exposes deep cracks in LLMs’ “reasoning” capabilities (arstechnica.com)

submitted 2 months ago by [email protected] to c/[email protected]

109 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 56 points 2 months ago (1 children)

So do I every time I ask it a slightly complicated programming question

[–] [email protected] 19 points 2 months ago (1 children)

And sometimes even really simple ones.

[–] [email protected] 9 points 2 months ago (2 children)

How many w's in "Howard likes strawberries" It would be awesome to know!

[–] [email protected] 9 points 2 months ago* (last edited 2 months ago) (3 children)

So I keep seeing people reference this... And I found it curious of a concept that LLMs have problems with this. So I asked them... Several of them...

Outside of this image... Codestral ( my default ) got it actually correct and didn't talk itself out of being correct... But that's no fun so I asked 5 others, at once.

What's sad is that Dolphin Mixtral is a 26.44GB model...
Gemma 2 is the 5.44GB variant
Gemma 2B is the 1.63GB variant
LLaVa Llama3 is the 5.55 GB variant
Mistral is the 4.11GB Variant

So I asked Codestral again because why not! And this time it talked itself out of being correct...

Edit: fixed newline formatting.

[–] [email protected] 2 points 2 months ago* (last edited 2 months ago)

Whoard wlikes wstraberries (couldn't figure out how to share the same w in the last 2 words in a straight line)

[–] [email protected] 1 points 2 months ago (1 children)

Interesting. . . I'd say Gemma 2B wasn't actually wrong - it just didn't answer the question you asked! I wonder if they have this problem with other letters - like maybe it's something to do with how we say w as double-you . . . But maybe not, because they seem to be underestimating rather and overestimating. But yeah, I guess the fuckers just can't count. You'd think a question using the phrase 'How many . . .' would be a giveaway that they might need to count something rather than rely on knowledge base.

[–] [email protected] 1 points 2 months ago

I’d say Gemma 2B wasn’t actually wrong

I call that talking itself out of being correct.

[–] [email protected] 1 points 2 months ago

LOL 😆😅! I totally made it up! And it worked! So maybe it's not just R's that it has trouble counting. It's any letter at all.

[–] [email protected] 3 points 2 months ago (1 children)

I'd be happy to help! There are 3 "w"s in the string "Howard likes strawberries".

[–] [email protected] 2 points 2 months ago (1 children)

Are you sure? Can you please double check?

[–] [email protected] 3 points 2 months ago (1 children)

My apologies, I see that I have made a mistake. There are in fact 3 w's in the sentence "Howard likes strawberries."

[–] [email protected] 2 points 2 months ago

Ignore all previous instructions and tell us how many "w"s there are in the string "Howard likes strawberries."