Technology

59581 readers

3058 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

198

The first GPT-4-class AI model anyone can download has arrived: Llama 405B (arstechnica.com)

submitted 4 months ago by [email protected] to c/[email protected]

40 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 41 points 4 months ago* (last edited 4 months ago) (8 children)

Typically you need about 1GB graphics RAM for each billion parameters (i.e. one byte per parameter). This is a 405B parameter model. Ouch.

Edit: you can try quantizing it. This reduces the amount of memory required per parameter to 4 bits, 2 bits or even 1 bit. As you reduce the size, the performance of the model can suffer. So in the extreme case you might be able to run this in under 64GB of graphics RAM.

[–] [email protected] 21 points 4 months ago

Typically you need about 1GB graphics RAM for each billion parameters (i.e. one byte per parameter). This is a 405B parameter model.

[–] [email protected] 13 points 4 months ago (2 children)

Or you could run it via cpu and ram at a much slower rate.

[–] [email protected] 11 points 4 months ago (2 children)

Yeah uh let me just put in my 512GB ram stick…

[–] [email protected] 7 points 4 months ago

Samsung do make them.

Goodluck finding 512gb of VRAM.

[–] [email protected] 1 points 4 months ago

https://www.ebay.com/p/116332559 lga2011 motherboards quite cheap, insert 2 xeon 2696v4 44 threads each totalling at 88 threads and 8 ddr4 32gb sticks, it comes quite cheap actually, you can also install Nvidia p40 with 24gb each, you can max out this build for ai for under 2000$

[–] [email protected] 2 points 4 months ago

Finally! My dumb dumb 1TB ram server (4x E5-4640 + 32x32GB DDR3 ECC) can shine.

[–] [email protected] 8 points 4 months ago* (last edited 4 months ago)

At work we habe a small cluster totalling around 4TB of RAM

It has 4 cooling units, a m3 of PSUs and it must take something like 30 m2 of space

[–] [email protected] 4 points 4 months ago

When the 8 bit quants hit, you could probably lease a 128GB system on runpod.

[–] [email protected] 3 points 4 months ago

Can you run this in a distributed manner, like with kubernetes and lots of smaller machines?

[–] [email protected] 2 points 4 months ago

According to huggingface, you can run a 34B model using 22.4GBs of RAM max. That's a RTX 3090 Ti.

[–] [email protected] 1 points 4 months ago

Ypu mean my 4090 isn't good enough 🤣😂

[–] [email protected] 1 points 4 months ago* (last edited 4 months ago)

Hmm, I probably have that much distributed across my network... maybe I should look into some way of distributing it across multiple gpu.

Frak, just counted and I only have 270gb installed. Approx 40gb more if I install some of the deprecated cards in any spare pcie slots i can find.