this post was submitted on 14 Jun 2023
80 points (100.0% liked)
Technology
1928 readers
7 users here now
Rumors, happenings, and innovations in the technology sphere. If it's technological news, it probably belongs here.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Cool!
Cans has source? I've been hacking on a per-instance community scraper (that pulls a list of communities from the community directory page) for !lemmy411[email protected], in an attempt to have a regularly published community list, but I'm a bit bogged.
I’m probably going to put the source up shortly but how I decided to find communities is by scraping my instances “federated with” list from https://toast.ooo/api/v3/site
Then going through the /.well-known/nodeinfo -> /nodeinfo/v2.0.json (or whatever url the well-known gives) to check the software to make sure it’s a Lemmy instance
Then using the Lemmy rest API to paginate through the open communities for that instance
I have links to the source for my crawler (written in nodejs with redis task queuing) on https://lemmyverse.net (GitHub link top right) 🤗 It's a bit messy at the moment but has all the community crawling stuff.
Wow. Stickied THAT to [email protected] thanks!