Wikipedia is a great start. You can download its entirety, roughly 100gb. Most of the basic and advanced human knowledge.
Check out kiwix to get it offline
this post was submitted on 07 Sep 2023
110 points (93.0% liked)
Asklemmy
43947 readers
448 users here now
A loosely moderated place to ask open-ended questions
Search asklemmy π
If your post meets the following criteria, it's welcome here!
- Open-ended question
- Not offensive: at this point, we do not have the bandwidth to moderate overtly political discussions. Assume best intent and be excellent to each other.
- Not regarding using or support for Lemmy: context, see the list of support communities and tools for finding communities below
- Not ad nauseam inducing: please make sure it is a question that would be new to most members
- An actual topic of discussion
Looking for support?
Looking for a community?
- Lemmyverse: community search
- sub.rehab: maps old subreddits to fediverse options, marks official as such
- [email protected]: a community for finding communities
~Icon~ ~by~ ~@Double_[email protected]~
founded 5 years ago
MODERATORS
Seconded Wikipedia. The amount of knowledge that can be gleaned in mere minutes from Wikipedia is insane. It contains enough information to do most stuff, aside from blatantly illegal things.
Luckily it isnt that easy to burn down every server as burning carpets in south america or books in other places
You can do all of Project Gutenberg too. It's only about 75gb, surprisingly.
ASCII text is hella lightweight.
load more comments
(5 replies)
You son of a bitch
Such an insightful commentary on the importance of the social contract and the irreplacibility of the individual. The only way forward is to share our personal experiences and strive for understanding. Once we know each other's value, we will never surrender our common bonds, disappoint one another, go behind each other's backs, nor do each other harm.
I feel like you're stretching the definition of "knowledge" and definitely "human knowledge" a bit there.
load more comments
(1 replies)
There isnβt. Yes Wikipedia has a lot of info but think of all the information that is in the hands of governments and corporations that are closely held secrets. Or thatβs only in the minds of a few experts on the planet.
Like sure Wikipedia can tell you what a CPU is. But to build one from scratch, from the silica to building the machines and factories, that information is spread across multiple companies and never shared with the public. And only a few experts truly know how to do every step in the process, they have vital knowledge of that process that they feel is common sense and is not written down, which they pass on to the people they mentor. If those few people die at the same time in a catastrophe the knowledge that isnβt written down dies with them.
We already lost a lot of information of old tech from not that long ago because the companies went bankrupt or the people involved all died. Like we donβt even have all the knowledge to rebuild the Saturn V rockets, because the people involved, who hold vital knowledge, are dead and the supporting infrastructure, like the sub contractors (who also had vital knowledge), is gone as well.
Perhaps https://archive.org/ is the closest you could get? With nearly a trillion web pages in its archive, I don't think I've ever come across a database of knowledge that comes close to it's collection. What's interesting is that archive.org preserves not only web pages, but several pieces of binary content such as music, movies, art and even software applications and entire operating systems. Not sure if it would be enough to rebuild our society, but it would be a great starting point for most of our essentials.
Specifically their OpenLibrary division. They had a mission to make as many books as possible digitally available and free for everyone to borrow but unfortunately they keep getting hit with lawsuits and slowly take down more and more of their collection.
Itβs βitsβ, not βitβsβ, unless you mean βit isβ, in which case it is βitβs β.
load more comments
(1 replies)
load more comments
(1 replies)
All of Wikipedia is <256 gb.
All of Wikipedia in English <64 gb.
Then archive.org for multimedia, ~10 peta bytes. Yipes.
load more comments
(4 replies)
That is pretty much exactly the goal of the Wikimedia Foundation which runs Wikipedia and its sister projects.
But by now we figured out what wikis can do well and what not. Wikis are suitable for crowdsourcing objective facts about the world (all it takes is one person to add any given fact), they are not a universal remedy for everything, especially not contentious issues or useful instructional materials.
I have made more than 100000 edits to their projects. I don't participate there anymore. The time when they were a force for good in the world is long past.
Why aren't they a force for good anymore?
Many reasons most of which you'll only understand if you pay some attention to what's going on behind their scenes.
There are reasons why nowadays pretty much everywhere else on the Internet more content is created all the time than on the Wikimedia projects.
The Wikipedias' "neutral point of view" policy used to mean "we try to treat all sides fairly", now it means "we are writing an unconditional propaganda organ for the status quo". The mainstream media that is accepted as "reliable" as Wikipedia sources just isn't that credible anymore.
Also, when I started editing there, the individual projects were mostly left alone by the WMF. Nowadays the WMF issues intransparent sanctions, up to lifetime bans from all projects, left and right.
I wish someone started an organization with the same goals as the WMF with an actually working system where people could actually enjoy participating.
load more comments
(3 replies)
I don't personally know of any but in a similar vein there are some stone monuments intended to convey information after an apocalypse like the Georgia Guidestones or the nuclear waste site warning stones. GitHub put a snapshot of all active code repositories from 2020 in arctic permafrost, and there is the arctic seed vault for preserving plant species.
Man those nuclear waste messages makes it sound like it's a cursed land
That's their intention.
The problem with such approaches will be human curiosity. Imagine today's scientists find such a site from the late paleolithic which has messages like "This site is cursed; we buried here what causes death and pestilence to us; go no further or it will do the same to you!" -- You bet they will want to see what is inside the "buried temple of death".
load more comments
(1 replies)
"On July 6, 2022, an explosive device was detonated at the site, destroying the Swahili/Hindi language slab and causing significant damage to the capstone. Nearby residents reportedly heard and felt explosions at around 4:00 a.m" the rocks got destroyed by a mere explosive and they thought it could survive a nuclear war lol
Well, if a nuke actually hits anything built to withstand nuclear war, it will break. There is nothing really that can withstand direct exposure to powerful explosives.
load more comments
(1 replies)
Check out this book: https://en.m.wikipedia.org/wiki/The_Knowledge:_How_to_Rebuild_Our_World_from_Scratch. It analyses that precise question in the first chapter. The author argues that even though Wikipedia is probably the closest thing there is, there is a clear lack of practical knowledge that will be essential in the situation that you are describing. Science progress heavily relies on industrial progress, and even if you know how to build something that doesn't mean that you can do it, as there are other things that are required first.
Damn that sounds like interesting book to read. Got to get a copy
According to my ex-, her.
Can confim
Haha you can have her. Good luck!
Please take her back. I'll even pay you.
No takesy backsys!
I think the internet as a whole is going to be the closest we'll ever come. Capitalism will make sure it's never even close to complete so it always has something to monetize.
I wouldn't say "complete" can even be sufficiently defined in this case. Every functional definition I can think of has a limiting factor.
Let's try to define knowledge. What kind of information qualifies? We can usually think of important, useful info like physics and medicine. But what about other data, like sports game stats, atmospheric sensor readings, or even something more esoteric, like the location data of every object on earth.
And even if we could have the information of every single thing at any particular time, what about when things change in the next second? And the one afterwards?
Essentially, nothing will ever be "complete". Thanks for listening to my rant on semantics.
load more comments
(1 replies)
load more comments
(1 replies)
Besides what other commenters already said, archive.org does a great job.
I'm surprised no one mentioned projects like libgen and scihub. They are much better than Wikipedia imo.
load more comments
(1 replies)
It's never going to be all knowledge, since a lot of stuff is just lost or never recorded. A ton of stuff (like this thread) are probably low on the priority list for recording as well. But the closest you'd probably get to a full catalog of human knowledge (at last text based) are the huge data sets of nearly all text data on the internet used for training LLMs. I wouldn't be surprised if there are ones soon that include video and pictures as well, since newer AI models are starting to be able to interpret those too.
I believe this is one of those data sets: https://github.com/yaodongC/awesome-instruction-dataset
Edit: here's a big data set used for a lot of gpt3 https://commoncrawl.org/
view more: next βΊ