Technology

60047 readers

2774 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 2 years ago

MODERATORS

[email protected]

In the space of 1 week, a second open-source Chinese AI model equals the best investors are pouring tens of billions of dollars into. (lemm.ee)

submitted 3 weeks ago by [email protected] to c/[email protected]

54 comments fedilink hide all child comments

cross-posted from: https://futurology.today/post/2910566

Alibaba's Qwen team just released QwQ-32B-Preview, a powerful new open-source AI reasoning model that can reason step-by-step through challenging problems and directly competes with OpenAI's o1 series across benchmarks.

The details:

QwQ features a 32K context window, outperforming o1-mini and competing with o1-preview on key math and reasoning benchmarks.

The model was tested across several of the most challenging math and programming benchmarks, showing major advances in deep reasoning.

QwQ demonstrates ‘deep introspection,’ talking through problems step-by-step and questioning and examining its own answers to reason to a solution.

The Qwen team noted several issues in the Preview model, including getting stuck in reasoning loops, struggling with common sense, and language mixing.

Why it matters: Between QwQ and DeepSeek, open-source reasoning models are here — and Chinese firms are absolutely cooking with new models that nearly match the current top closed leaders. Has OpenAI’s moat dried up, or does the AI leader have something special up its sleeve before the end of the year?

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 6 points 3 weeks ago (4 children)

Does anyone have an idea how much RAM would this need?

[–] planish 7 points 3 weeks ago (1 children)

Looks like it has 32B in the name, so enough RAM to hold 32 billion weights plus activations (current values for the layer being run right now, which I think should be less than a gigabyte). It is probably made of 16 bit floats to start with, so something like 64 gigabytes, but if you start quantizing it to cram more weights into fewer bits, you can go down to like 4 bits per weight, or more like 16 gigabytes of memory to run (a slightly worse version of) the model.

[–] [email protected] 7 points 3 weeks ago (1 children)

So you're telling me there's a chance.

[–] planish 4 points 3 weeks ago (1 children)

I think there are consumer-grade GPUs that can run this on a single card with enough quantization. Or if you want to run it on CPU you can buy and plug in enough DIMMs if you have an only somewhat large amount of money.

[–] [email protected] 4 points 3 weeks ago* (last edited 3 weeks ago)

Pulled whatever is available on Ollama by this name and it seems to just fit on a 3090. Takes 23GB VRAM.

[–] [email protected] 5 points 3 weeks ago (1 children)

I asked it and it gave me this answer:

As an AI language model, I don't have any physical form or hardware requirements, including RAM. I exist solely to process and generate text based on the input I receive. So, there's no need for any RAM or other hardware resources for me to function.

[–] [email protected] 6 points 3 weeks ago (1 children)

Priceless.

[–] [email protected] 2 points 2 weeks ago

It’s so innocent. So cute. Like your child telling you that they don’t need to eat.

[–] [email protected] 3 points 3 weeks ago

It depends on how low you're willing to go on the quant and what you consider acceptable token speeds. Qwen 32b q3ks can be partially offloaded on my 8gb vram 1070ti and runs at about 2t/s which is just barely what I consider usable for real time conversation.

[–] [email protected] 2 points 3 weeks ago

For a 16k context window using q4_k_s quants with llamacpp it requires around 32GB. You can get away with less using smaller context windows and lower accuracy quants but quality will degrade and each chain of thought requires a few thousand tokens so you will lose previous messages quickly.