I'm curious if I can migrate my instance (a single user) to a different domain? Right now I'm on a free DNS from no-ip but I might get a prettier paid domain name sometime.

So i run my own instance of 1 user. Ive subscribed to about 30 communities and They have all been pending for days. Only 2 have been marked as subscribed. Is it just an overloaded thing? Ive tried unsub and subscribing again. no luck.

For anyone else running lemmy on kubernetes-

Here is an IngressRoute CRD you can use, to leverage your built-in traefik reverse proxy.

Normally-

(ingress / ingressroute) -> (service) -> (nginx proxy) -> (lemmy / lemmy ui)

With this-

(ingress / ingressroute) -> (service) -> (lemmy / lemmy ui)

A slight optimization to better take advantage of the built in kubernetes functionality. (since, it already has a nginx and/or traefik instance running).

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: lemmy
  namespace: lemmy
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`lemmyonline.com`) && (Headers(`Accept`, `application/activity+json`) || HeadersRegexp("Accept", "^application/.*") || Headers(`Accept`, `application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"`))
      services:
        - name: lemmy
          port: http
    - kind: Rule
      match: Host(`lemmyonline.com`) && (PathPrefix(`/api`) || PathPrefix(`/pictrs`) || PathPrefix(`/feeds`) || PathPrefix(`/nodeinfo`) || PathPrefix(`/.well-known`))
      services:
        - name: lemmy
          port: http
    - kind: Rule
      match: Host(`lemmyonline.com`) && Method(`POST`)
      services:
        - name: lemmy
          port: http
    - kind: Rule
      match: Host(`lemmyonline.com`)
      services:
        - name: lemmy-ui
          port: http

Just- make sure to replace your host, with the proper instance name.

Lemmy.ml front page has been full of nginx errors, 500, 502, etc. And 404 errors coming from Lemmy.

Every new Lemmy install beings with no votes, comments, postings, users to test against. So the problems related to performance, scaling, error handling, stability under user load can not easily be matched given that we can not download the established content of communities.

Either the developers have an attitude that the logs are of low quality and not useful for identify problems in the code and design, or the importance of getting these logs in front of the technical community and trying to identify the underlying patterns of faults is being given too low of a priority.

It's also important to make each log of failures identifiable to where in the code this specific timeout, crash, exception, resource limit is encountered. Users reporting generic messages that are non-unique only slow down server operators, programmers, database experts, etc.

There are also a number of problems testing federation given the nature of multiple servers involved and trying not to bring down servers in front of end-users. It's absolutely critical that failures for servers to federate data be taken seriously and attempts to enhance logging activities and triangulate causes of why peer instances have missing data be track down to protocol design issues, code failures, network failures, etc. Major Lemmy sites doing large amounts of federation are an extremely valuable source of data about errors and performance. Please, for the love of god, share these logs and let us look for the underlying causes in hard to reproduce crashes and failures!

I really hope internal logging and details of the inner workings of the biggest Lemmy instances is shared more openly with more eyes on how to keep scaling the applications as the number of posts, messages, likes and votes continue to grow each and every data. Thank you.

Three recently created communities: [email protected] -- [email protected] -- [email protected]

cross-posted from: https://lemmyonline.com/post/10272

Since, everything done on behalf of your instance is logged, detecting if you have a large number of bots, or invalid users isn't that challenging.

These queries can be executed via docker exec -it, via remoting into the container, via pg query tools, or via pgadmin.

For listing all comments performed by users on your instance (This includes comments made remotely):

SELECT
	p.actor_id
	, p.name
	, c.content as comment
FROM public.comment c 
JOIN public.person p on p.id = c.creator_id
WHERE 
	p.local = 'true'
	AND p.admin = 'false' -- Exclude Admins
;

For listing all posts created, by users, from your instance-

SELECT
	p.actor_id
	, c.name AS title
	, c.body as body
FROM public.post c 
JOIN public.person p on p.id = c.creator_id
WHERE 
	p.local = 'true'
	AND p.admin = 'false' -- Exclude Admins
;

Lastly, here is a query to identify users who consistently upvotes or downvotes the same user over and over.

SELECT
	p.id
	, p.name
	, p.actor_id
	, cr.name as creator
	, count(1)
FROM public.comment_like l
JOIN public.comment c on c.id = l.comment_id
JOIN public.person p on p.id = l.person_id
JOIN public.person cr on cr.id = c.creator_id
WHERE 
	p.id != cr.id
	AND p.local = 'true'
	AND p.admin = 'false' -- Exclude Admins
GROUP BY p.id, p.name, p.actor_id, cr.name
ORDER BY count(1) desc
;

If- anyone has idea of other queries which can be created for detecting suspicious activity, please LMK.

Edit- added where clause to exclude admins. If your admins are spambots, you have bigger issues to worry about.

( I didn't cross-post, as I encourage comments to go all on one posting )

I’m wondering, what resources does Lemmy need? For a small instance (let’s say, me and some friends) or a large instance (think Lemmy.ml or Lemmy.world or the like…).

Also, where do y’all host your instances?

(posted here after I got directed to this community when I asked it in [email protected] )

When moving my lemmy instance to another server, how do i make shure that the instance starts up again? I used the lemmy-Ansible method to setup my instance. I use digitalocean to host my instance, and when creating a snapshot to move the server location, the instance would not work. Luckly i still had the original droplet intact. I am very new to server management, any help would suffice. Thanks in advanced!

Even though 0.18 is installed on Lemmy.ml - the code is failing internally. And without access to lemmy.ml's server logs, I'm trying to diagnose the whole design from a remote instance.

Lemmy.world, Beehaw, Lemmy.ml are all throwing "fast nginix 500" errors on their front door on a regular basis. And all are showing symptoms of replication failures sending messages and content to each other (missing posts and comments).

Even AFTER lemmy.ml was upgraded to 0.18, I was able to get stuck 'pending' subscribes on both my own personal remote instance and over at Lemmy.world:

I've been making a fool of myself as best as I can in hope somebody will step back and actually share their lemmy application error logs on where the faulty points are within the code. I highly suspect that PostgreSQL is timing out or http federation timeouts are happening within the Rust code.

cross-posted from: https://lemmy.ca/post/821266

So it seems that no instance has published a privacy policy, many users are asking about such a thing (as they should), and much confusion on how federation happens among users AND some admins. I feel this is pretty important to the survival of Lemmy to work out a privacy policy framework.

Yes, the argument that "everything on the internet stays forever" is true, but there is a big distinction between captured copies, and some of the unique data distribution / management issues that come up with a federated service. It is important to inform the user of this distinction. It is also important to inform them how early the development is.

It is going to scare the pants off some users. I'd argue an educated user on an totally public platform is far more safe than an uneducated one on a closed platform, but let the user decide that for themselves. I'd much rather scare the pants off them then have them coming for me once they get caught with their pants down and feel I didn't do enough to warn them. Can you imagine hundreds of thousands of pantless lemmings with pitchforks coming for you? Not a pretty image.

I AM NOT A LAWYER, but I have created a template based on the Mastodon privacy policy if anyone wants a basic framework to start from:

https://github.com/BanzooIO/federated_policies_and_tos/blob/main/lemmy-privacy-policy.md

I am not overly experienced with instance management yet, but I have done my best to cover all aspects of how data is federated. Please contribute in correcting any errors.

I also feel it is important for admins to disclose the current lack of SSL support in connecting to PostgreSQL and what the local admin has done to mitigate the risk.

Issues on open on the topic of privacy policies here: https://github.com/LemmyNet/lemmy/issues/721 and https://github.com/LemmyNet/lemmy-ui/issues/1347

Hey I don't run any instance so I'm not sure what's the best way to alert all instance admins - but there seems to be a massive wave of signups over the last few days, most likely by bots. I think some of the bigger instances know this and have mitigated it (my instance enabled captcha and email verification), but smaller instances like startrek.website and waveform.social are getting hammered. They've gone from 2000-4000ish users to >10,000 overnight. If this isn't fixed, it will probably be a big problem when these bot accounts are activated. It would suck if you run a smaller instance and had to defederate yourself because of thousands of spam bots.

cross-posted from: https://snkkis.me/post/1316

Hey there! I was wondering of how you might easily share your instance with others? I've created one mainly for nordic people, but I am struggling to "advertise" it to people. Have any tips for a noob at lemmy instances? Thanks in advance! Oh, and if you are by chance interested in joining, then just go to my account, it's hosted there.

cross-posted from: https://popplesburger.hilciferous.nl/post/9969

After setting up my own Lemmy server, I've been intrigued by the server logs. I was surprised to see some search engines already start to crawl my instances despite it having very little content.

I've noticed that most requests seem to come in from IPv4 addresses, despite my server having both an IPv4 and an IPv6 address. This made me wonder.

IPv4 addresses are getting more scarce by the day and large parts of the world have to share an IPv4 address to get access to older websites. This often leads to unintended fallout, such as thousands of people getting blocked by an IP ban from a site admin that doesn't know any better, as well as anti-DDoS providers throwing up annoying CAPTCHA pages because of bad traffic coming from the shared IP address. Furthermore, hosting a Lemmy server of your own is impossible behind a shared IP address, so IPv6 is the only option.

IPv6 is the clear way forward. However, many people haven't configured IPv6 for their hosts. People running their own Lemmy instances behind an IPv6 address won't be able to federate with those servers, and that's a real shame.

Looking into it

So, I whipped up this quick Python script:

import requests
import sys
import socket
from progress.bar import Bar

lemmy_host = sys.argv[1]

site_request = requests.get(f"https://{lemmy_host}/api/v3/site").json()

hosts = site_request['federated_instances']['linked']

ipv4_only = []
ipv6_only = []
both = []
error = []

with Bar('Looking up hosts', max=len(hosts)) as bar:
    for host in hosts:
        host = host.strip()

        try:
            dns = socket.getaddrinfo(host, 443)
        except socket.gaierror:
            error.append(host)

        has_ipv4 = False
        has_ipv6 = False
        for entry in dns:
            (family, _, _, _, _) = entry

            if family == socket.AddressFamily.AF_INET:
                has_ipv4 = True
            elif family == socket.AddressFamily.AF_INET6:
                has_ipv6 = True

        if has_ipv4 and has_ipv6:
            both.append(host)
        elif has_ipv4:
            ipv4_only.append(host)
        elif has_ipv6:
            ipv6_only.append(host)
        else:
            error.append(host)
        
        bar.message = f"Looking up hosts (B:{len(both)} 4:{len(ipv4_only)} 6:{len(ipv6_only)} E:{len(error)})"
        bar.next()

print(f"Found {len(both)} hosts with both protocols, {len(ipv6_only)} hosts with IPv6 only, and {len(ipv4_only)} outdated hosts, failed to look up {len(error)} hosts")

This script fetches the instances a particular Lemmy server federates with (ignoring the blocked hosts) and then looks all of them up through DNS. It shows you the IPv4/IPv6 capabilities of the servers federating with your server.

I've run the script against a few popular servers and the results are in:

Results

Server	IPv6 + IPv4	IPv6 only	IPv4	Error	Total
Lemmy.ml	1340	3	1903	215	3461
Beehaw.org	807	0	1105	74	1986
My server	202	0	312	4	518

It seems that over half (55%+) the servers on the Fediverse aren't reachable over IPv6!

I'm running my own server, what can I do?

Chances are you've already got an IPv6 address on your server. All you need to do is find out what it is (ip address show in Linux), add an AAAA record in your DNS entries, and enable IPv6 in your web server of choice (i.e. listen [::]:443 in Nginx). Those running a firewall may need to allow traffic through IPv6 as well, but many modern firewalls treat whitelist entries the same these days.

Some of you may be running servers on networks that haven't bothered implementing IPv6 yet. There are still ways to get IPv6 working!

Getting IPv6 through Tunnelbroker

If you've got a publicly reachable IPv4 address that can be pinged from outside, you can use Hurricane Electric's Tunnelbroker to get an IPv6 range, free of charge! You get up to five tunnels per account (each tunnel with a full /64 network) and a routed /48 network for larger installations, giving you up to 65k subnets to play with!

There are lots of guides out there, some for PfSense, some for Linux, some for Windows; there's probably one for your OS of choice.

Getting IPv6 behind CGNAT

Getting an IPv6 network through a tunnelbroker service behind CGNAT is (almost) impossible. Many ISPs that employ CGNAT already provide their customers with IPv6 networks, but some of them are particularly cheap, especially consumer ISPs.

It's still possible to get IPv6 into your network through a VPN, but for serving content you'll need a server with IPv6 access. You can get a free cloud server from various cloud providers to get started. An easy way forward may be to host your server in the cloud, but if you've got a powerful server at home, you can just use the free server for its networking capabilities.

Free servers are available from all kinds of providers, such as Amazon(free for a year), Azure(free for a year), Oracle(free without time limit). Alternatively, a dedicated VPS with IPv6 capabilities can be as cheap as $4-5 per month if you shop around.

You can install a VPN server on your cloud instance, like Wireguard, and that will allow you to use the cloud IPv6 address at home. Configure the VPN to assign an IPv6 address and to forward traffic, and you've got yourself an IPv6 capable server already!

There are guides online about how to set up such a system. This gist will give you the short version.

Final notes

It should be noted that this is a simple analysis based on server counts alone. Most people flock to only a few servers, so most Lemmy users should be able to access IPv6 servers. However, in terms of self hosting, these things can matter!

I'm thinking about setting up my own (bare metal) Lemmy instance to play around with it, but it seems to require PostgreSQL. Everything else on my system uses MySQL, and I don't really want to run 2 separate database services. I guess I would also be fine with using an SQLite file, but that's not ideal.

Has anyone managed to set up a Lemmy instance with MySQL instead of PostgreSQL? Are you aware of any PostgreSQL to MySQL or SQLite compatibility layers?

I encourage all instance owner/operators to run the query mentioned in the issue and see how many of these 'pending' they have on their server. (FYI, I am RocketDerp on GitHub)

Hey, just went through a few different checklists, and discovered that Lemmy does not meet GDPR requirements for notifying users for how servers handle the data. I've brought up this request on github, and I hope to get it fixed soon, but in the meantime I've compiled a list of EU address blocks and intend to add them to my firewall. Just thought you all should know.

It is currently possible, through Lemmy's API, to create accounts automatically and without limit if verification by email address or captcha is not activated. I'd advise you to activate one or both of them NOW!

After registering x number of accounts (currently I could do thousands), all you have to do is list all the existing communities for each of the account to publishes one new post per community, or more. I'll leave you to picture the mess.

(I apologise to the administrators of sh.itjust.works, I should have done the test with my own server.)

The following tutorial provides step-by-step instructions to install Lemmy, a link aggregator for the fediverse, on Ubuntu 22.04. It includes the installation of Lemmy backend, Lemmy UI (web frontend), and the configuration of a reverse proxy with TLS using Nginx.

You should be able to install Lemmy without any errors. If you have any, please let me know. I intend to extend this tutorial to Lemmy upgrades, migration and deletion, and thus update the official documentation.

After using Ansible to upgrade Lemmy following the instructions (https://github.com/LemmyNet/lemmy-ansible) I am now getting the error

Error: LemmyError { message: Some("Cannot have both private instance and federation enabled."), inner: Cannot have both private instance and federation enabled., context: "SpanTrace" }

I am not sure how to change either private to requiring approval or public OR disable federation now. I don't see anything about it in the relevant documentation (https://join-lemmy.org/docs/en/administration/configuration.html) but I could be missing something.

The docker container keeps trying to restart and fails each time. Any help with this would be really appreciated!

EDIT: solved it by ssh'ing into the postgres docker container, running psql and UPDATE local_site SET private_instance = 'f' WHERE id = 1;

Also, I changed it back to private in the admin settings and it's fine now... really weird. Going to need to remember to do this every update I guess.

At the time of this posting, both sites for over a minute were throwing 500 or 502 errors. I've seen this 'naked nginx 50x error page' many times throughout the last week from almost every Lemmy site I visit.

First off, how do we configure Nginx with a better error message when the site is down that says try again in a few minutes, etc? Someone have some config steps to share?

Has anyone else had reports of false positives from AV from users?

I host a small Lemmy instance (just me at the moment), and on the Admin page i've noticed these rate limit settings. They don't seem to be documented and their meaning is a bit unclear to me. For example, what is the difference between 'Message Rate Limit' and 'Per Second'? And are these values per user or for the entire instance?

If anyone could explain these rate limit values (or point me to the relevant documentation) I would greatly appreciate it!

cross-posted from: https://lemmy.ml/post/1216911

Some of you may have noticed that federated actions are slow to synchronize between Lemmy instances. This is most likely because of the setting "Federation worker count" under /admin. It determines how many federation activities can be sent out at once. The default value is 64 which is enough for small or medium sized instances. But for large instances it needs to be increased.

Grep the server logs for "Maximum number of activitypub workers reached" and "Activity queue stats" to confirm that this affects you. For lemmy.ml I just changed the value to 512, you have to experiment what is sufficient. The new value is only applied after restarting Lemmy. In my case changing the value through the website didnt work (maybe because its overloaded). Instead I had to update local_site.federation_worker_count directly in the database.

Edit: I had to increase the value to 160k for lemmy.ml. Now the stats arent getting logged anymore, so Im not sure if the pending queue is still building up or not.

I see many questions, thought I'd make a post :-)

So when running Lemmy in Docker, you'll see your / filesystem gets filled up quickly. That's mainly in /var/lib/docker/containers. It's the logging of the containers.

Docker will default save all logging. Also when you docker-compose stop and restart, the logging stays. Only when re-creating the containers (docker-compose down;docker-compose up -d) it will clear the logs.

Why is my container logging this much?

In the docker-compose.yml you'll see an environment variable RUST_LOG= which is set to info. This logs all informational messages.

You can set this to warn instead to only get warning and error messages. (Requires container recreate)

How to manually cleanup the log without restart

You can find the log of a container with

docker inspect --format='{{.LogPath}}' <containername>

You can then wipe it by typing '> ' (so greater than followed by space) followed by the file path+name.

(Removing it while the container runs wil not really remove it, you'll just no longer see it but it still exists and grows as long as the container runs)

How to prevent logs from growing too large

You can set a max for log file size in your docker-compose.yml using these lines for every container:

    logging:
      driver: "json-file"
      options:
        max-size: "100m"

I will post comments on individual topics