this post was submitted on 12 Jun 2023
19 points (100.0% liked)

Selfhosted

39158 readers
388 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

Hi,

Just set up an instance on my little nuc here at home but it is sooo empty in "all communities". Are you aware of a script that e.g. extracts all community https links from lets say lemmy.world and adds them 1 by 1 in my search field so that my instance starts federating? Thanks

top 5 comments
sorted by: hot top controversial new old
[–] Orygin 5 points 1 year ago* (last edited 1 year ago) (1 children)

I may be wrong, but while it's annoying to have to add each instance one by one, it's also a feature.
If your instance's users never access those other instances/communities, federating with them will only serve to increase the load on those servers, without any tangible benefits (other than not having to search communities once before accessing them).

If I'm wrong, please someone correct me :)

[–] [email protected] 1 points 1 year ago

Not to mention it will increase your instance's storage requirements far beyond what you use. If you take @lackthought's suggestion to use browse.feddit.de, you'll only be pulling in things that you actually use.

[–] [email protected] 2 points 1 year ago (1 children)

lemmy.directory and browse.feddit.de both seem to index all lemmy communities

I’m not sure on the technical details of how they do it but you could try asking around on those instances for some advice

[–] [email protected] 6 points 1 year ago* (last edited 1 year ago) (1 children)

I'm still learning the API, but if you wanted to build your own community crawler, you could do something like this:

  • Get a list of instances. You can call /api/v3/site against your own and look at the federated_instances.linked key and query each of those. If your federation list is small, you could call that API endpoint for another instance, such as lemmy.ml or beehaw.org to get theirs.
  • For each of the domains returned, call {domain}/api/v3/community/list?type_=Local to get a JSON list of its local communities
  • Store the results however you want and then create a UI to work with them as you need.

You probably don't want to crawl and subscribe to every community on every instance. That would cause a lot of unnecessary traffic for both you and the instances you're subscribed to.

To quickly make a remote community "known" to your server, you can massage the Lemmy-UI's search endpoint to make it work a little better.

https://{your-instance-url}/search/q/[[email protected]](/c/[email protected])/type/All/sort/TopAll/listing_type/undefined/community_id/0/creator_id/0/page/1

Replacing [[email protected]](/c/[email protected]) with the remote community address you want. Can be in either [[email protected]](/c/[email protected]) or https://domain.tld/c/community formats - both will work. Calling the search page that way will wait until there is a result before rendering rather than showing "no results" and making you click search two or three times until it appears. Same process on the backend, just cleaner in the front.

You can do that through the backend API similarly, but you'd have to put your JWT in an auth URL param which I wouldn't suggest against the instance's public URL.

Dove into the API last night to see how difficult it would be to use the REST API to create something resembling RES for old.reddit.com. That's about as far as I got before life got in the way, so hopefully that's useful for someone.

[–] [email protected] 1 points 1 year ago

Wow, just wow. Thanks for the ideas... Guess I need to familiarize/play around a little bit...

load more comments
view more: next ›