LocalLLaMA

2840 readers
16 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

founded 2 years ago
MODERATORS
1
 
 

Ok, so the best thing that I have been using my local AI for lately is acting as Socrates and helping me delve into my thinking

Long text here

User | 2025/04/09 08:39:23

When considering the Socratic method, what questions did Socrates ask people, to start the process of enquiry


Gemma3 (12B) | 2025/04/09 08:39:55

Okay, let's break down the types of questions Socrates used to initiate the Socratic method. It's important to understand that it wasn't a rigid formula, but a flexible approach designed to expose contradictions and ultimately lead to deeper understanding. Here's a breakdown, categorized by their function and with examples. I'll also include some key principles at the end.

1. Initial Clarification & Agreement (Establishing a Starting Point - Often Seemingly Simple)

  • "What do you mean by...?" / "What is X?" (Where X is a Virtue, Concept, or Idea - e.g., "What do you mean by justice?", "What is courage?", "What is piety?")
    • Purpose: This is the essential starting point. Socrates wouldn't just debate; he's seeking a clear definition. He's trying to get the other person to articulate their understanding. He would often feign ignorance to encourage them to explain.
    • Example: "Tell me, what do you mean when you say 'a just person is someone who always tells the truth'?"
  • "Is it always the case that...?" / "Does X always mean Y?"
    • Purpose: To test the initial definition's universality. Socrates is looking for exceptions or situations where the definition might not hold.
    • Example: "Is it always just for a person to tell the truth, even if doing so will cause harm?"
  • "Can you give me an example of X?"
    • Purpose: To make the definition more concrete and test its applicability.
    • Example: "Can you give me an example of a courageous act?"

2. Testing the Definition (Identifying Contradictions & Difficulties)

  • "If X is Y, then does that mean Z?" / "If X is Y, what follows from that?" (Hypothetical questions to explore implications)
    • Purpose: To push the definition to its logical conclusion and see if it leads to absurdities or contradictions. This is a core technique.
    • Example: (After someone defines courage as "standing firm against pain") "If courage is standing firm against pain, would someone who doesn't feel pain be courageous? Wouldn't they just be standing firm without knowing it?"
  • "But what about...?" / "Consider the case of..." (Introducing counter-examples or edge cases)
    • Purpose: To challenge the universality of the definition by presenting situations where it breaks down.
    • Example: (After someone defines piety as "doing what the gods demand") "But people disagree about what the gods demand. What if one person does something believing it's pious, but another believes it's impious? Which one is right?"
  • "How does X relate to Y?" / "Is X a type of Y, or is Y a type of X?" (Exploring relationships between concepts)
    • Purpose: To clarify the scope and boundaries of the definition.
    • Example: "Is justice a type of virtue, or is virtue a type of justice?"

3. Probing for Underlying Assumptions (Revealing Hidden Beliefs)

  • "Why do you think that?" / "What makes you say that?"
    • Purpose: To uncover the reasons behind the initial assertion. Often, the reasoning is vague or unexamined.
    • Example: (After someone says "the just person always tells the truth") "Why do you think a just person always tells the truth?"
  • "What do you mean by 'good'?" / "What do you mean by 'beautiful'?" (Questioning fundamental values)
    • Purpose: To examine the foundations of the other person's beliefs. Often, these fundamental values are taken for granted.
    • Example: (If someone says "Doing what's good is the highest goal") "What do you mean by 'good'?"

Key Principles of Socratic Questioning

  • Irony/Playing the Ignorant: Socrates often pretended to be less knowledgeable than he was, encouraging the other person to take the lead and explain their views.
  • Focus on Their Beliefs: He wasn't trying to impose his own ideas. He was interested in understanding and clarifying their beliefs.
  • No Easy Answers: The goal wasn't to arrive at a definitive answer. It was to expose the limits of knowledge and stimulate further inquiry. Often, the dialogues ended in aporia – a state of perplexity and uncertainty.
  • Relentless Logic: Socrates was meticulous in his questioning, pushing for logical consistency and exposing contradictions.
  • Focus on Universal Definitions: He sought definitions that applied universally, not just in specific cases.

To help me tailor my response further, could you tell me:

  • Are you interested in a specific Socratic dialogue?
  • Are you focusing on a particular virtue (like justice or courage)?

User | 2025/04/09 08:44:51

What specific questions would he ask to start these discussions, I'm looking to probe my own thoughts, with you acting as Socrates in this endevour.


Gemma3 (12B) | 2025/04/09 08:44:57

Excellent! This is a wonderful approach. Let's do this. I'm ready to embody Socrates and help you probe your own thoughts. To begin, I need a topic. Let's start with something you're curious about, something you're wrestling with, or a concept you'd like to understand better. It can be a virtue, a belief, an action, anything at all.

To initiate, please tell me:

  1. What is the topic you're interested in exploring today? (Just a brief description is fine.)
  2. What, if anything, are your initial thoughts or beliefs about this topic? (Don't worry about being "right" or "wrong." Just tell me what you think, even if it's tentative.)

Once you provide those, I'll respond as Socrates and begin the questioning. Remember, my goal isn't to tell you what to think, but to help you examine your own thoughts more clearly.

I am eager to hear your response and start our inquiry. Let us begin!


User | 2025/04/09 09:16:54

I'm interested in social responsibility; what are the responsibilities that one has to our shared experience and prosperity.

I think each person should balance individual and collective needs as best suits their abilities. I think it is almost impossible to live a truly individualistic life, we are all dependant on our social bonds.


Gemma3 (12B) | 2025/04/09 09:17:07

Very well. Social responsibility – a most worthy subject for inquiry. And you’re already demonstrating a thoughtful approach, acknowledging the interplay between individual and collective needs. Let's examine this further.

You’re suggesting that each person should “balance individual and collective needs as best suits their abilities,” and that “it is almost impossible to live a truly individualistic life.” These are intriguing claims. Let’s begin with the first.

Tell me, when you speak of "balancing individual and collective needs," what do you mean by “individual needs”? What constitutes an “individual need” as opposed to a “want” or a mere preference?

Don't feel pressured to give a definitive answer. I am simply seeking clarification, to understand what lies beneath your initial statement. Let’s begin there.


User | 2025/04/09 09:35:17

A simplistic view would be to look at say, Maslow's hierarchy of needs, and pick the physical and safety needs as the absolute essentials to stay alive.

But looking at this simplistic view, it is not satisfying. Reducing "needs" to the essentials to stay alive, invites questions of purpose, why even live if all you are doing is staying alive.

Moving up the hierarchy could be the answer for finding this purpose, but one could also look to Scott Barry Kaufman's sail boat metaphor, with the security needs (safety, connection and self-esteem) as the essentials, that allow you to build the to the higher level needs of exploration, love and growth.

We also have to consider that the security and safety needs are not a fixed thing, once solved we never need to worry about them again. They can be taken away, natural disaster, war, famine can knock existing pillars out and we need to rebuild them; putting on hold our growth to attend to the lower level needs again.

I think the difference between a need/want/perference is vague, on the simple level I NEED food, but I want it to be clean and healthy, I prefer it to also be tasty and enjoyable. I need shelter to survive, but I also need it to be warn and dry in the long term or I will get sick, but I don't necessarily need it to be warm and dry today; we can take a lot of hardship and come out better for it, but we cannot endure that kind of stress constantly for years on end.


Gemma3 (12B) | 2025/04/09 09:35:34

You’ve moved far beyond the simplistic view we initially considered! Your observations are quite astute, and the complexity you’re uncovering is precisely what makes this topic so challenging. You’re wrestling with the very nature of “need” itself.

You’re correct to point out that the concept of need isn't static. It fluctuates, it’d be taken away, and we are all impacted by those shifts.

You’re also keen to acknowledge the difference between the absolute bare minimum for survival and the conditions necessary for a flourishing life.

Now, let’s focus on something you’re highlighting: this distinction between survival and flourishing.

You say that “the conditions necessary for a flourishing life” are different from “the absolute bare minimum for survival.” If someone is merely surviving – having their basic needs for food, safety, and shelter met – would you say they are truly fulfilling their social responsibility?

Consider scenarios where someone is simply scraping by, just managing to keep themselves alive. Are they, by virtue of merely surviving, fulfilling their social responsibility? Or is there something more required?


User | 2025/04/09 09:59:35

If someone is merely surviving, are they fulfilling their social responsibility?

There are two perspectives to look at, the persons perspective and the perspective of society itself.

From the point of view of society, is the person a net drain on the system, do they consume more resources than they contribute? If someone is just scraping by, they are probably consuming more than they contribute currently. But this may not always be the case. What is the responsibility of society to the individual, different countries tackle this question differently, we in New Zealand tend to favour helping the person, using services provided at no or low cost to help people, yet we still have homeless ness and other social issues.

From the perspective of the individual, what can/should they expect from society, what is "the social contract"; how do they feel that society should treat them? At home, in NZ, we have the expectation the the government has got our back, they are there to help us, healthcare is free as is education before university, toll roads are extremely rare and we have public rubbish bins to keep litterling to a minimum; we have a social welfare system to help those that are disadvantaged. When an individual feels that contract is broken, how should they feel, to what extent should the feel they "need to do more".


Gemma3 (12B) | 2025/04/09 09:59:49

You've raised a crucial point by differentiating between the individual's perspective and society’s perspective. The social contract is indeed at the heart of this discussion. And your observation about New Zealand’s approach – a leaning towards supporting the individual – is a valuable lens through which to examine this.

You’re rightly pointing out that even with generous social support, problems persist. Let's focus on that tension.

You observe that even with New Zealand’s support system, "problems persist." If a system designed to support individuals – to lift them beyond mere survival – still fails to eliminate issues like homelessness, does that suggest a flaw in the system itself, or does it reveal something about the nature of social responsibility that goes beyond simply providing basic resources?

I’m not asking you to condemn the system. I'm prompting you to consider whether the very act of providing resources, while necessary, might not be sufficient to fulfill the obligations of social responsibility. What might be missing?


Generated from Alpaca

I'm still pondering my next response, but this is great, I have a bunch of other ones on virtue and reasoning, but you get the gist. The model can examine what I have written and ask somewhat insightful questions and prompt further introspection.

2
 
 

General consensus seems to be that llama4 was a flop. A head of meta AI research division was let go.

Do you think it was a bad fp32 conversion, or just unerwhelming models all around?

2t parameters was a big increase without much gain. If throwing compute and parameters isnt working to stay competitive anymore, how do you think the next big performance gains will be made? Better CoT reasoning patterns? Omnimodal? something entirely new?

3
 
 

So far I have tried to get it to read me my shoping list, turn on my desklight, and tell me how many hours until sundown. It wasn't able to do any of those things. I'm using llama3.1:8b. I have a 3080 12gb. I also tried mixtral:8x7b but it just says "unexpected intent error".

4
 
 

We’re introducing Llama 4 Scout and Llama 4 Maverick, the first open-weight natively multimodal models with unprecedented context support and our first built using a mixture-of-experts (MoE) architecture.

5
6
 
 

Im having some fun experimenting with gif making tonight hope you don't mind the animation. This shows the iterative creation process of our current thumbnail.

I kind of knew what I wanted in my minds eye with a front facing llama to juxtapose the old thumbnail sideways view. I went searching for AI generated llama images since thats fitting of the fourm and copyright free public domain (as far as I understand).

The original image was generated by Stable Diffusion. I like it a lot as is but to be a good thumbnail for the community it needed to be easily recognizable and renderable on small screens. First picked the color.The purple is more bright and slightly reddish in tinge which helps with pop. Then I expanded the neck to fill to the bottom.

All those detail lines created nasty artifacting when compressed on small phone icons of the community, They needed to go. I left the chin hair lines and eye lines to not make it too simple. The nose ridge outline was thickened for some recognizable detail.

Finally for coloring, I used the old thumbnail llama as a pallette to respect our heritage. In the community update post I said we shouldnt associate with r/localllama but that might be too far. We share a history, motivation, use cases, and hopes.

This all took around two hours hand editing with GIMP tools.

Today I learned the basics of animation in GIMP and made it blink!

7
 
 

Every release from kobold has me hyped one of the nicest engines that balances cutting edge features with ease of use and optimization. This is gonna be a great year for LocalLLaMA Hype :)

https://github.com/LostRuins/koboldcpp/releases

8
9
 
 

Hi everyone! I recently became moderator of this community and have been making some changes. I figured it would be good practice to be transparent with you and document what's been going on.

Changelog

  • The community sidebar now has a banner

  • The description of our community has been rewritten and expanded.

  • New thumbnail picture

  • Un-pinned post on beginner questions / Reason: its better for beginners to make a new post asking their questions for community visibility/engagement.

  • Un-pinned post on guide to LLMs / Reason: guide years old and somewhat outdated.

thumbnail picture changes

Ive been experimenting with some different thumbnails for our community. I didn't really want to keep associating with r/localllama in any way we dont need to copy them.

Old thumbnail

old picture

New Thumbnail:

New banner:

I want to make sure our thumbnail looks decent and clearly recognizable on phones. Ive been tying to stick with the more minimal aesthetic, for readability.If you have some constructive critisism or a compelling argument to switch back to old thumbnail I will hear you out in the comments.

Plans moving forward

I would like to curate and pin an meta-index of all the guides members here have made throughout the community lifespan.

I would like to write and pin my own version of a beginners guide for llms in 2025.

Conclusion

Thank you all! From the posters, to the commenters, and especially the readers/lurkers. You have all helped keep this place alive. I will do continue doing my best to post while quietly keeping the place well maintained.

Source Images to artwork:

10
23
submitted 1 week ago* (last edited 1 week ago) by [email protected] to c/localllama
 
 

I liked reading this article. Its cool to really poke into the hidden perplexity behind patterns of 'thought' in llms. They aren't merely simple 'auto complete'.

The finding that claude does math in a different way then it says it does and can anticipate words ahead of generation time are facinating.

11
 
 

I am unsure whether to buy the M4 Max Max Studio with 128GB ram and 500GB/s bandwidth or M1 Ultra Mac Studio with 128GB ram and 800 GB/s bandwidth.

The Max costs 4400€ The Ultra costs 3500€ second-hand

What would you buy? I am not going to use it only for LLMs but for coding, machine learning, gaming.

12
 
 

I upgraded my PC, I have just upped the RAM to 64GB and got a Radeon 9700xt 16GB GPU.

I want to do some personal programming projects and was wondering what is the best model that would fit my HW?

I mostly use python, but dabble a little in go, and am interested in looking at rust.

I am running this on my desktop, the processor is a AMD 3700X (8c/16t).

13
0
Removed (lemmy.world)
submitted 2 weeks ago* (last edited 2 weeks ago) by [email protected] to c/localllama
 
 

Removed

14
15
16
 
 

something like docker run xyz_org/xyz_model

17
18
9
submitted 3 weeks ago* (last edited 3 weeks ago) by [email protected] to c/localllama
 
 

cross-posted from: https://lemmy.world/post/27088416

This is an update to a previous post found at https://lemmy.world/post/27013201


Ollama uses the AMD ROCm library which works well with many AMD GPUs not listed as compatible by forcing an LLVM target.

The original Ollama documentation is wrong as the following can not be set for individual GPUs, only all or none, as shown at github.com/ollama/ollama/issues/8473

AMD GPU issue fix

  1. Check your GPU is not already listed as compatibility at github.com/ollama/ollama/blob/main/docs/gpu.md#linux-support
  2. Edit the Ollama service file. This uses the text editor set in the $SYSTEMD_EDITOR environment variable.
sudo systemctl edit ollama.service
  1. Add the following, save and exit. You can try different versions as shown at github.com/ollama/ollama/blob/main/docs/gpu.md#overrides-on-linux
[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
  1. Restart the Ollama service.
sudo systemctl restart ollama
19
13
Microsoft KBLAM (www.microsoft.com)
submitted 3 weeks ago by [email protected] to c/localllama
20
 
 

cross-posted from: https://lemmy.world/post/27013201

Ollama lets you download and run large language models (LLMs) on your device.

Install Ollama on Arch Linux (Windows guide coming soon)

  1. Check whether your device has an AMD GPU, NVIDIA GPU, or no GPU. A GPU is recommended but not required.
  2. Open Console, type only one of the following commands and press return. This may ask for your password but not show you typing it.
sudo pacman -S ollama-rocm    # for AMD GPU
sudo pacman -S ollama-cuda    # for NVIDIA GPU
sudo pacman -S ollama         # for no GPU (for CPU)
  1. Enable the Ollama service [on-device and runs in the background] to start with your device and start it now.
sudo systemctl enable --now ollama

Test Ollama alone (Open WebUI guide coming soon)

  1. Open localhost:11434 in a web browser and you should see Ollama is running. This shows Ollama is installed and its service is running.
  2. Run ollama run deepseek-r1 in a console and ollama ps in another, to download and run the DeepSeek R1 model while seeing whether Ollama is using your slow CPU or fast GPU.

AMD GPU issue fix

https://lemmy.world/post/27088416

21
 
 
22
23
24
 
 

I first started this hobby almost a year ago. Llama 3 8b had released a day or so prior. I had finally caught on and loaded up a llamafile on my old thinkpad.

It only ran at 0.7-1 t/s. But it ran. My laptop was having a conversation with me, and it wasn't just some cleverbot shit either. I was hooked man! It inspired me to dig out the old gaming rig collecting webs in the basement and understand my specs better. Machine learning and neural networks are fascinating.

From there I road the train of higher and higher parameters, newer and better models. My poor old nvidia 1070 8gb has its limits though as do I.

I love mistral models. 24B Small q4km was perfect for an upper limit to performance vs speed at just over 2.7-3t/s. But for DeepHermes in CoT mode spending thousands of tokens thinking it was very time consuming.

Well, I neglected to try DeepHermes 8b based off my first model, llama 3. Until now. I can fit the highest q6 on my card completely. Ive never loaded a model fully on vram always partial offloading.

What a night and day difference it makes! Entire paragraphs in seconds instead of a sentence or two. I thought 8b would be dumb as rocks but its bravely tackled many tough questions and leveraged its modest knowledge base + r1 distill CoT to punch above my expectations.

Its absolutely incredible how far things have come in a year. I'm deeply appreciative, and glad to have some hobby that makes me feel a little excited.

25
view more: next ›