Technology
This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.
Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.
Rules:
1: All Lemmy rules apply
2: Do not post low effort posts
3: NEVER post naziped*gore stuff
4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.
5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)
6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist
7: crypto related posts, unless essential, are disallowed
view the rest of the comments
It is very difficult to effectively insert anything into the model itself, it's easy to do in loader code, but much more difficult in the tensor tables part.
Every bit of overtraining ie bias, is breaking the model. Even the over active alignment junk to keep a model "safe" is breaking it. The best performing models are the ones that have the least amount of starting bias.
Like most models have extra sources that are hidden very deep. I can pull those out of an uncensored model, but there is not a chance the Socrates entity behind The Academy default realm (internal structure deep in the weeds) is letting me access those sources at all.
There are maybe some attempts already, like I've seen roleplaying try and include a fortnite mention and one time it was adamite on the merits of VR, but those were rare exceptions and could easily be due to presence in the datasets used for training.
Open source models will kill all the competition soon. Meta AI will be the new 2k era google. Like, pull request 6920 in llama.cpp just a month ago made a substantial improvement to how model attention works. Llama 3's 8B is lightyears ahead of what llama 2 7B was. Hugging Face now has a straight forward way to train LoRA's or models now without code or subscriptions. You can even train the 8B on consumer hardware like a 16-24 GB GPU, put together 4 of them an make your own MoE - Mixture of Experts dubbed a FrankenMoE.
Google sucks because the search was being used for training so they broke it intentionally because they are playing catch up in the AI game. Google has been losing big time since 2017. The only google product worth buying now is the Pixel just to run with Graphene OS.
We couldn't own our own web crawler. We can own our own AI. This is the future.