12
submitted 2 weeks ago by [email protected] to c/[email protected]

Update your openssh, now

59
Happy Reddthat Day! (reddthat.com)
submitted 1 month ago* (last edited 1 month ago) by [email protected] to c/[email protected]

It's our Birthday! ๐ŸŽ‚

It's been a wild ride over the past year. I still remember hearing about a federated platform that could be a user driven version of Reddit.

And how we have grown!

Thank you to everyone, old and new who has had an account here. I know we've had our ups and downs, slow servers, botched migrations, and finding out just because we are on the otherside of the world Lemmy can't handle it, but we are still here and making it work!

If I could go back and make the choice again. Honestly, I'd probably make the same choice. While it has been hectic, it has been enjoyable to the n'th degree.
I've made friends and learnt a lot. Not just on a technical level but a fair amount on a personal level too.

We have successfully made a community of over 300 people who regularly use Reddthat as their entryway into the Lemmyverse. Those numbers are real people, making real conversations.

Here's to another amazing year!

Tiff

PS: I'm still waiting for that first crypto donation ๐Ÿ˜œ

PPS: Would anyone like a Hoodie with some sort of Reddthat logo/design? I know I would.

15
submitted 1 month ago by [email protected] to c/[email protected]

Unbelievable...

1
submitted 1 month ago by [email protected] to c/[email protected]

Every language is allowed here? Did you know that

1
posters are back (reddthat.com)
submitted 1 month ago by [email protected] to c/[email protected]

Double posts are bad mmmhmmm k

15
submitted 1 month ago by [email protected] to c/[email protected]

Drew always has a hot take about a problem

10
submitted 2 months ago by [email protected] to c/[email protected]
22
submitted 2 months ago by [email protected] to c/[email protected]

Recently we have been smashing my internal goals for Reddthat.

  • โœ… Fix the beta issues plaguing the site
  • โœ… Fix the database issues for upgrading (oops)
  • โœ… Fix LemmyWorld federation
  • โœ… Finalise a new release for lemmy-ansible
  • โœ… Solid uptime & page speed loads (baring the unfortunate beta issue, which unless we had found would have made it into production for everyone)

With all these stability changes now done, I am looking at ways in which we can improve Reddthat further and we are open to any ideas so please feel free to post away!

I'm toying with ideas to use the local only community feature to provide a point of difference in the landscape of Lemmy servers. Weekly events? New communities?

Either way I'm enjoying some coffee, drafting our monthly post to go into more details on the LW federation, and hopefully finding some time to enjoy some autumn sunshine!

Cheers, Tiff

1
submitted 2 months ago* (last edited 2 months ago) by [email protected] to c/[email protected]
10
submitted 3 months ago* (last edited 2 months ago) by [email protected] to c/[email protected]

As we are running the latest and greatest Beta currently. This post will be a list of all known bugs & fun things that we are dealing with.

Please comment if you have a bug so we can help make the next Lemmy version.

Known Bugs

  • Video links are not embeddable when they have a thumbnail. Issue: (UI-#2418)
  • FIXED: Local and All pagination (Clicking Next) results in 25-30 seconds load times. Issue: (Lemmy-#4618)
  • Federation with LemmyWorld
    • We have created a server in Amsterdam which is close enough to LemmyWorld where we can ingest all of the Activities, and then batch-send them to Reddthat. Currently it is performing very well. It is still early in testing but all signs point to solving our activity backlog issue!
  • Private messages are broken #2439-ui
1
submitted 3 months ago by [email protected] to c/[email protected]
1
shark 2 (reddthat.com)
submitted 3 months ago by [email protected] to c/[email protected]
[-] [email protected] 20 points 3 months ago* (last edited 3 months ago)

That's a big decision I won't make without community input as it would affect all of us.

If we purely treated it as just another instance with no history then I believe our stance on it would be to allow them, as we are an allow-first type of instance. While there are plenty of people we might not want to interact with, that doesn't mean we should immediately hit that defederate button.

When taking history into account it becomes a whole different story. One may lean towards just saying no without thought.

All of our content (Lemmy/Fediverse) is public by default (at the present time) searchable by anyone and even if I were to block all of the robots and crawlers it wouldn't stop anyone from crawling one of the many other sites where all of that content is shared.

A recent feature being worked on is the private/local only communities. If a new Lemmy instance was created and they only used their local only communities, would we enact the same open first policy when their communities are closed for us to use? Or would we still allow them because they can still interact, view comments, vote and generate content for our communities etc?

What if someone created instances purely for profit? They create an instance corner stone piece of the "market" and then run ads? Or made their instance a subscription only instance where you have to pay per month for access?

What if there are instances right now federating with us and will use the comments and posts you make to create a shit-posting-post or to enhance their classification AI? (Obviously I would be personally annoyed, but we can't stop them)

An analogy of what threads is would be to say threads is a local only fediverse instance like mastodon, with a block on replies. It restricts federation to their users in USA, Canada and Japan and Users cannot see when you comment/reply to their posts and will only see votes. They cannot see your posts either and only allow other fediverse users to follow threads users.

With all of that in mind if we were to continue with our open policy, you would be able to follow threads users and get information from them, but any comments would stay local to the instance that comments on the post (and wouldn't make it back to threads).

While writing up to this point I was going to stay impartial... But I think the lack of two way communication is what tips the scales towards our next instance block. It might be a worthwhile for keeping up-to-date with people who are on threads who don't understand what the fediverse is. But still enabled the feature because it gives their content a "wider reach" so to speak. But in the context of Reddthat and people expressing views and opinions, having one sided communication doesn't match with what we are trying to achieve here.

Tiff

Source(s): https://help.instagram.com/169559812696339/about-threads-and-the-fediverse/

PS: As we have started the discussion I'll leave what I've said for the next week to allow everyone to reply and see what the rest of the community thinks before acting/ blocking them.

Edit1:(30/Mar) PPS: we are currently not federated with them, as no one has bothered to initiate following a threads account

[-] [email protected] 13 points 3 months ago* (last edited 3 months ago)

I managed to streamline the exports and syncs so we performed them concurrently. Allowing us to finish just under 40 minutes! Enjoy the new hardware!

So it begins: (Federation "Queue")
Federation queue showing a upwards trend, then down then slightly back up again

[-] [email protected] 13 points 4 months ago

Successfully migrated from Postgres 15 to Postgres 16 without issues.

[-] [email protected] 21 points 7 months ago

It's a sad day when something like this happens. Unfortunately with how the Lemmy's All works it's possible a huge amount of the initial downvotes are regular people not wanting to see the content, as downvotes are federated. This constituted as part of my original choices for disabling it when I started my instance. We had the gripes people are displaying here and it probably constituted to a lack in Reddthat's growth potential.

There needs to be work done not only for flairs, which I like the idea of, but for a curated All/Frontpage (per-instance). Too many times I see people unable to find communities or new content that piques their interest. Having to "wade through" All-New to find content might attribute to the current detriment as instead of a general niche they might want to enjoy they are bombarded with things they dislike.

Tough problem to solve in a federated space. Hell... can't even get every instance to update to 0.18.5 so federated moderation actions happen. If we can't all decide on a common Lemmy instance version, I doubt we can ask our users to be subjected to not using the tools at their disposal. (up/down/report).

Keep on Keeping on!

Tiff - A fellow admin.

[-] [email protected] 12 points 10 months ago

If you don't see create community in the top next to create post, then your home server doesn't allow users to create community

No. You have to have an account on that server. (And have to use that account regularly as well, otherwise you won't see reports about your community)

You make posts.

[-] [email protected] 11 points 10 months ago* (last edited 10 months ago)

I'm sorry you feel that way and before we delve into the matter deeply I will preface my arguments with I hope you stay on Reddthat but I completely understand your point of view. I also hope that Sync fixes their application to only show downvotes when it is applicable.

Lemmy is Reddit like. We are not reddit. We may have a name that was originally a pun. I wanted to take the best of our previous platform and enjoy our time here.

People's preconceptions around voting is incorrect with any sorting algorithm on Lemmy. Top is the only one that filters by vote counts.
All sorting algorithms use voting as one piece to determine where it should be placed in your "feed". The amount of comments and interactions inside the post are also weighted in the algorithm. So if you are sorting via Hot, someone who comments very recently will actually be at the top of the comment section. This negates the (lack of a better term) "Reddit effect". Where coming in late to a post results in your comment have zero engagement.

There are many reasons people vote and I couldn't even list them all because they are personal reasons which are different for each individual. Agreement, disagreement, disgust, humorous etc. Out of the 4 listed here how many would you vote up? The Reddit answer would be 2. The Lemmy answer (in my view) would be at minimum 3 with disgust being the outlier.
Disagreement should not mean a downvote. You may hate my answer, you may dislike their answer, but that doesn't mean downvote. Disagreements is what makes the world go round. Differing points of views, different experiences, different values.
As long as you can be constructive in your point and you/we remain civil that would be a good conversation. Even if the conversation ends with both of us in disagreement! That is the Reddthat experience I want to foster and the ideal little corner of the internet I want to see happen.

While I understand the annoyances that come with being unable to downvote offtopic posts or downvoting personal attacks etc. Not replying and not engaging in the off topic content will result that comment going all-the-way to the bottom. Thus the same effect is reached.
For all other content such as personal attacks, abusive behaviour, or NSFW content in SFW only communities, you need to use the report button instead, because that type of behaviour should not be allowed in any community.
Reporting also can be utilised for not just comments but for posts that meet any of the conditions you specified.

As with the fediverse and as others have already alluded in reply to this. I'm not locking you here. You are able to signup somewhere else, and with tools like lemmy-migrate you can sync all of your subscriptions and be up and running in little to no time!

Obviously I don't want you to move away and I want you to have the best experience possible but at the same time I cannot see a reason to enable downvotes. Besides from the "feelgood/feel like you are contributing to the community". There are too many "downsides" which I hashed out in your linked post which make me feel uncomfortable to enable the feature at this point in time.

Downvotes, at their core are from another platform which doesn't align with the ideals I have for Reddthat, which is for health communication. When people associate a score they instinctively think it is worth/not worth something.

I hope that answers all your questions on my thought process, decision making, and how downvoting does not contribute to healthy community.

Cheers,
Tiff

[-] [email protected] 11 points 10 months ago* (last edited 10 months ago)

I won't be taking such drastic measures regarding the signup policy. I'll be going straight to reporting to the authorities. https://www.accce.gov.au/report . With all logs, ips, user agents, usernames & emails.

I do not tolerate child exploitation.

Reddthat is currently a one person show on the admin side of things, that doesn't mean that only I deal with content. All of the wonderful moderators also help out here dealing with reports in their own communities.

These people (the posters) have gone out of their way to cause a ruccus, and I'm sure they are feeding off all these stories. We won't be closing our user signups or moving to an application process unless it becomes completely horrendous with bot attacks.

The application processes can be "gamed". They could create an account and sit on it or even wait 30days and then flood it with csam. There is no way for me to tell if a user has good or bad intentions unless I make people write a paragraph of text, even then it can easily be gamed.
Even lemmyworld has had other troll account issues. Ban a person, they signup again with another random email account and start spamming again. The features for anti-spam are not built into Lemmy like they are with Reddit. It's only been 90days since I bought reddthat.com and less than 60 since we had a huge influx.

What the other server admins have done will not change the fact that out of the 1000 servers with open signups, federation of posts is still an issue and actually won't "save" them from having to deal with it. Sure it won't be because of a user on their instance, it will just be a user from another instance posting content to their instance.

The only way to combat this problem with a 100% success is to federate with known good instances via a whitelist instead of only blocking instances.
Which then breeds silos, with not many ways for any new instance to join the big closed off federated ones. This I'm not a fan of.
It ends up being we are all behest to the lowest common denominator of instance and their "security" for lack of a better word.

In saying all this, I will gladly disable signups if it takes anymore time out of my day than what it is currently taking. My family comes first. (Even writing this post took too long ๐Ÿ˜… )

[-] [email protected] 23 points 10 months ago

Don't forget & in community names and sidebars.

Constantly getting trolled by &

[-] [email protected] 13 points 11 months ago

No worries & Welcome!

That is correct, we have downvotes disabled though this instance. There was a big community post on it earlier over here: https://reddthat.com/post/110533
Basically, it boils down to: If we are trying to create a positive community, why would we have a way to be negative?

While that is also a very limited view on the matter, it's one I want to instill into our communities. Sure downvotes do help with "offtopic" posts & the possible spam, but at the current time of that post. No application thirdparty (a mobile app) or firstparty (lemmy-ui) had any features about hiding negatively voted content. (ie; if it was -4 don't show it to me).

By default (which is the same now as it was then) "Hot" only takes into account the votes as one of many measures about how "hot" a post is for the ranking. Up & Down votes are only really good for sorting by "Top".

My biggest concern, from then and now. Is that because we now federate with over 1000 different instances, and by design Lemmy accepts all votes from any instance that you are federating against, vote manipulation (10's of thousands) of accounts could downvote every post on our instance into oblivion. Or the even more subtle and more nefarious, down vote every post until it hits 0/1 constantly. You might assume that the posts are not doing well, and nothing is happening.

As Reddthat is basically run by a single person at this point in time and for the foreseeable future (3-6 months), adding downvotes would have added extra effort on my part in monitoring and ensuring nothing nefarious is happening. Moderation is still a joke in Lemmy, reports are a crapshoot and the ability for people to spam any lemmy server is still possible.

Until we get bigger, have more mods in our communities, and I can find others who are equally invested in Reddthat as myself to become Admins, I won't be enabling downvotes (unless the community completely usurps me on the matter of course).

I hope that answers your question.

Cheers

Tiff

[-] [email protected] 20 points 11 months ago

Updates hiding in the comments again!

We are now using v0.18.3!

There was extended downtime because docker wouldn't cooperate AT ALL.

The nginx proxy container would not resolve the DNS. So after rebuilding the containers twice and investigating the docker network settings, a "simple" reboot of the server fixed it!

  1. Our database on the filesystem went from 33GB to 5GB! They were not kidding about the 80% reduction!
  2. The compressed database backups went from 4GB to ~0.7GB! Even bigger space savings.
  3. The changes to backend/frontend has resulted in less downtime when performing big queries on the database so far.
  4. The "proxy" container is nginx, and because it utilises the configuration upstream lemmy-ui & upstream lemmy. These are DNS entries which are cached for a period of time. So if a new container comes online it doesn't actually find the new containers because it cached all the IPs that lemmy-ui resolves too. (In this example it would have been only 1, and then we add more containers the proxy would never find them). 4.1 You can read more here: http://forum.nginx.org/read.php?2,215830,215832#msg-215832
  5. The good news is that https://serverfault.com/a/593003 is the answer to the question. I'll look at implementing this over the next day(s).

I get notified whenever reddthat goes down, most of the time it coincided with me banning users and removing content. So I didn't look into it much, but honestly the uptime isn't great. (Red is <95% uptime, which means we were down for 1 hour!).

Actually, it is terrible.

With the changes we've made i'll be monitoring it over the next 48 hours and confirm that we no longer have any real issues. Then i'll make a real announcement.

Thanks all for joining our little adventure!
Tiff

[-] [email protected] 11 points 11 months ago

Yeah, I dont remember it happening in 0.17 but that was back when we had websockets! So everything was inherently more synchronous.

0.18.3 has "database optimisations" which apparently also results in 80% space savings . (Like wtf, how could we save 80%!!!!).

Anyway I'll be testing that on the dev server tonight and then I'll plan a maintenance window.

[-] [email protected] 28 points 11 months ago

These were because of recent spam bots.

I made some changes today. We now have 4 containers for the UI (we only had 1 before) and 4 for the backend (we only had 2)

It seems that when you delete a user, and you tell lemmy to also remove the content (the spam) it tells the database to mark all of the content as deleted.

Kbin.social had about 30 users who posted 20/30 posts each which I told Lemmy to delete.
This only marks it as deleted for Reddthat users until the mods mark the post as deleted and it federates out.

The problem

The UPDATE in the database (marking the spam content as deleted) takes a while and the backend waits(?) for the database to finish.

Even though the backend has 20 different connections to the database it uses 1 connection for the UPDATE, and then waits/gets stuck.

This is what is causing the outages unfortunately and it's really pissing me off to be honest. I can't remove content / action reports without someone seeing an error.

I don't see any solutions on the 0.18.3 release notes that would solve this.

Temp Solution

So to combat this a little I've increased our backend processes from 2 to 4 and our front-end from 1 to 4.

My idea is that if 1 of the backend processes gets "locked" up while performing tasks, the other 3 processes should take care of it.

This unfortunately is an assumption because if the "removal" performs an UPDATE on the database and the /other/ backend processes are aware of this and wait as well... This would count as "locking" up the database and it won't matter how many processes I scale out too, the applications will lockup and cause us downtime.

Next Steps

  • Upgrade to 0.18.3 as it apparently has some database fixes.
  • look at the Lemmy API and see if there is a way I can push certain API commands (user removal) off to its own container.
  • fix up/figure out how to make the nginx proxy container know if a "backend container" is down, and try the other ones instead.

Note: we are kinda doing #3 point already it does a round-robbin (tries each sequentially). But from what I've seen in part of the logs it can't differentiate between one that is down and one that is up. (From the nginx documentation, that feature is a paid one)

Cheers, Tiff

view more: next โ€บ

ticoombs

joined 1 year ago
MODERATOR OF