AI

4006 readers

12 users here now

Artificial intelligence (AI) is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans and animals, which involves consciousness and emotionality. The distinction between the former and the latter categories is often revealed by the acronym chosen.

founded 3 years ago

How Gradient created an open LLM with a million-token context window (venturebeat.com)

submitted 2 months ago by [email protected] to c/[email protected]

5 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 1 points 2 months ago (2 children)

That's cool. Am I reading right that this wouldn't run on consumer grade hardware though?

[–] [email protected] 4 points 1 month ago

I believe you'd need roughly 500GB of RAM to run it minimum at full context length. There is chatter that 125k context took and used 40GB

I know I can load the 70B models into my laptop at lower bits but it consumes about 140GB of RAM.

[–] [email protected] 3 points 1 month ago

It is llama3-8B so it is not out of question but I am not sure how much memory you would need to really go to 1M context window. They use ring attention to achieve high context window, which I am unfamiliar with but that seems to lower greatly the memory requirements.