557
submitted 2 months ago* (last edited 2 months ago) by TheDude to c/main

Hello sh.itjust.works community,

Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.

In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:

  • Dual Xeon 2.9Ghz CPUs (32 cores total)
  • 256GB ram
  • 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
  • 10gbit networking

While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.

  1. Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
  2. There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.

Here's the tentative timeline:

Task Date Expected Downtime
Migration to new server Tuesday February 27 2024 @ 8:00PM ET 90 Minutes
Upgrade to V19.3 Thursday February 29 2024 @ 8:00PM ET Up to 120 Minutes
  • If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
  • If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.

=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you

53
submitted 9 months ago* (last edited 9 months ago) by TheDude to c/agora

Please share your thoughts, concerns and opinions. Try to keep it constructive.

Link to announcement

55
submitted 9 months ago* (last edited 9 months ago) by TheDude to c/main

Hey all,

Similar to the lemmy.world announcement, we have defederated from Rammy.

Want to discuss this more? I created a [Discussion] post for us to do just that.

[-] TheDude 60 points 9 months ago

Hey all, I’m always excited to upgrade to the latest versions that become available. Unless the upgrade is related to security I will tend to wait a few days to give other instances a chance to share their experiences. Before upgrading I usually go over the issues that have been submitted to help understand the risks and solutions to mitigate them.

That being said the new version does bring in a whole bunch of performance gains (mostly related to the database) that i’m looking forward to.

Lemmy.world is due to upgrade later today and I plan on upgrading shortly after them (either Monday or Tuesday).

I will put up a notification up prior to the upgrade.

Thanks

25
submitted 10 months ago by TheDude to c/agora

Some of you may have noticed that there's a new bot in town. But this time, its one of ours.

What should we name it?

Post your entry.

Rules.

  • The 5 most upvoted entries will be brought to a vote.
  • The winning entry will become the display name of the bot.
[-] TheDude 78 points 10 months ago

Hey all,

There has been a few issues over the last couple of weeks. Here's a rundown of some of the issues we've had.

  • Broken Images: A few weeks ago we had issues with broken images. This was due to me migrating our local image storage to object storage (like AWS S3). When I did the switch it broke all old images as they needed to be migrated. Pictrs at that time did not support concurrent uploads which means the migration would have taken days or weeks to complete during which time the image service would be offline. Instead I waiting for a newer version to be released that supported concurrent connections and did the migration in about 30-40 minutes one evening.

  • 5xx server errors: Some of you may have experience a lemmy page with an error code on it. This was due to me trying to implement an additional proxy to shield and mitigate future risks. While rolling this out I hit a few blockers that caused downtime as we worked to rectify it. I'm glad to say that as of Today this has since been implemented.

In the addition to the above, the lemmyverse and especially this instance has been under bot attack almost daily. These bot attacks are eating resources and causing a query floods.

Lemmy is still very young and in its early phases. In time these issues will slowly go away.

P.S you don't have to worry about me leaving you guys hanging.

[-] TheDude 125 points 10 months ago

Honestly these type of scat spamming has been really discouraging on me. I spend time and money so that the community can have a reliable and nice instance to call home, only to have a few (not sure what to call them) people try to ruin it for everyone. If these people are reading this comment, just know that every time you make these kinds of unwanted posts you are taking my time away from my family and young kids to address the reports.

My nights lately have been mostly used for coding some extra moderation tools that will moderate these in the future.

I sincerely apologize for all of you who have been exposed to literal shit images over the last week. I’m working as fast as I can and will implement something soon.

In the meanwhile i’ve added a few new admins to help with this moderation.

[-] TheDude 77 points 10 months ago

Hey all,

As others mentioned we did not have custom emojis so we were not affected by this particular attack. I have since upgraded our UI to 0.18.2-rc.1 which mitigates this XSS vulnerability.

[-] TheDude 56 points 10 months ago

The vulnerability appeared to be from a custom emoji that they were running. SJW does not use any custom emoji so we should not be affected. In either case lemmy.world has now been restored and is back online. I’ll keep an extra eye on this instance until the patch gets released shortly.

[-] TheDude 97 points 10 months ago

If I were to do some rough math I'd say it would cost about $300/CAD per month.

My goal is once we are approved to start accepting donations that I can purchase dedicated hardware for this instance. I'd get a used server at about $2300 which would be sufficient a good amount of extra users and through it into its own dedicated shared colo at about $100/month. Factor in about $300-400 a year for drive replacements and we are left with $2300 / 12 month= 191.66 + 100/month for the shared 1u colo + a budget of $400 for drive failures throughout the year $33/month. 191.66 + 100 + 33 = $324.66/month for the first year dropping to about $133 per month after the first 12 months. It's worth noting that this method would give us double the amount of resources and quite a bit of extra storage.

Ideally we don't keep this instance on a single server forever and start to think about spreading it over multiple hosts at or after around 100K users (or less if the number of active users is high).

If someone wanted to host an instance they would not need to allocate as much resources as I have to this instance and depending on how active the instance gets could run off something a lot less powerful.

[-] TheDude 73 points 10 months ago

Hi Nahvi.

Thanks for posting this well put together post. I can tell that you put some time and thought into this and I want you to know that I recognize that.

Regarding your concern, this is a topic that has been brought up numerous times over the past month. I've since moved towards being more community driven, letting users discuss and debate on topics that are important to them. These discussion move forward towards a vote over a 7 day period. This process will change over time as we come across flaws and look for ways solve them together.

One more thing I want to make clear regarding your tl;dr question Will I be blocking other instances for personal reasons? Short answer is no, I believe the community should determine what should or should not be blocked.

That being said I want to make myself clear that there are limitations, I will continue to defederate instances that openly promote or spreads hate speech, terrorism, CP, loli, trolling and instances that become bot instances. These shouldn't be things that need to be justified, I think they fall under the "be a good human" category.

I hope this clarifies your concerns and you continue to make sh.itjust.works your home.

124
submitted 11 months ago* (last edited 11 months ago) by TheDude to c/main

This instance has been upgrade to 0.18.0.

The lemmy developers have removed the built in captcha so I had to enable email verification. They will be re-implementing this feature in the next update.

Lots of new improvements in this version. The Link i wanted to share is on the lemmy.ml instance and they appear to be down at the moment. Hopefully not related to this update.

Stay tuned.

Edit: Lemmy.ml back up, Here is the post: link of the update

[-] TheDude 65 points 11 months ago

Hey all,

This community is being reviewed and will likely be a discussion point to in our Agora community where we discuss about issues like this as a community.

Post that are breaking rules will be removed. If its a regular occurrence so will the TheDonald community. Please continue reporting inappropriate posts and its difficult to maintain visibility on everything that is going on at all times.

Thank you for bringing this to my attention.

35
submitted 11 months ago by TheDude to c/agora

Hey all,

As discussed here, I am looking for 3 new moderators to help with this community.

Please review this post and vote for the candidate you would like to see help on this community.

In no particular order, the candidates are:

I invite you to review users post history when deciding on voting.

This thread will be open for 72 hours (until Friday June 23rd, 8PM ET) at which point voting will stop. Any votes after this time will not be counted. I will try to lock this thread after that time so people do not accidentally continue voting.

Everyone gets 1 vote, including the users being considered. The top 3 will be given mod powers of this community to help with organization of discussions and votes.

Please vote by calling the persons name. For example, if I you wanted to vote for TheDude you would reply with: TheDude

24
submitted 11 months ago by TheDude to c/agora

Since making the announcement last night to become community driven, The Agora has quickly become an active place full of discussions, proposals and voting. In order to keep things going I'm looking for volunteers to help facilitate the conversations and apply the rules that have been determine by this community.

As a mod of The Agora you should be active on our main matrix channel and willing to help facilitate conversations between other mods and users.

As a mod:

  • you will still retain your right to vote just like anyone else.
  • may be dismissed by the community if its felt you are abusing your powers or pushing your own agenda.
  • will help maintain order.
  • follow the rules of the community.

Not sure you want to help mod this community but would like to nominate someone else? go ahead and nominate someone.

584
submitted 11 months ago by TheDude to c/main

Hey all

Today, I am thrilled to share two significant developments with you that will determine our shared future.

Firstly, the issue of donations. Since the inception of this instance, your most frequent request has been the ability to make contributions to support my initiative. While initially, I had never intended to accept donations, I've come to realize the value this brings in ensuring our platform's sustainability. In response to your requests, within the next week, I will be introducing several options for those of you who wish to donate. I want to emphasize that these donations are entirely optional and will directly support our instance's operational necessities - dedicated hardware, colocation fees, email services, and more.

The second announcement needs a full disclosure: it could be an extraordinary idea or potentially not so. Since the beginning of this instance, I've seen a number of insightful posts, recommendations, and ideas that the community has put forward, often superior to what I could have conceptualized myself. Even our instance's name was born from one of your suggestions.

Frequently, I'm approached by users seeking clarity on our rules and guidelines or expressing their thoughts on existing rules. In reflecting upon this, it became clear to me that I've been attempting to determine what's best for the community. But who am I to make these decisions? Just two weeks ago, I was a user among you all. Hence, I'm moving away from the traditional role of decision-making.

Instead, I want to hand over the reins of decision-making to you - the community. I'm excited to announce the creation of a community called agora where you can express your desires for the future of this instance. It's up to you to come together, discuss, and reach a consensus. If you wish to add, remove, or modify a rule, make a post, garner support from other members, and I'll implement the change. This invitation extends beyond our immediate community - I welcome input from everyone across the fediverse. Again to be clear, I gave an example of modifying rules but this applies to anything that I have the ability to do on this instance.

As the instance owner, I only have one caveat to add - any decisions made should not jeopardize the instance's existence or result in legal complications. Aside from that, I'm eager to see where this new direction takes us.

Thank you for your ongoing support, creativity, and engagement.

424
submitted 11 months ago* (last edited 11 months ago) by TheDude to c/main

Like many of you, I woke up this morning to discover that our instance, along with lemmy.world, had been unexpectedly added to the beehaw block list. Although this development initially caught me off guard, the administrators at beehaw made an announcement shedding light on their decision.

The primary concern raised was our instance's policy of open registration. Given my belief that the fediverse is still navigating its early stages, I believe that for it to mature, gain traction, and encourage adoption, it is crucial for instances to offer an uncomplicated and direct route for newcomers to join and participate. This was one of the reason I decided to launch this instance. However, I do acknowledge that this inclusive approach brings its unique challenges, including the potential for toxicity and trolls. Despite these hurdles, I maintain the conviction that our collective strength as a community can overcome these issues.

After this happened, the beehaw admins and I had a good chat about their decision. While our stances on registration policies might diverge, we realized that our ultimate goals are aligned: we both strive to foster communities that thrive in an atmosphere of safety and respect, where users can passionately engage in discussions and feel a sense of belonging.

Although the probability of an immediate reversal are slim given the current circumstances, I believe we have managed to identify common ground. It's evident that, even in separation, we can unite to contribute positively to the broader fediverse community.

In the coming weeks or months, we plan to collaborate with other lemmy instance administrators to suggest enhancements and modifications to the lemmy project. Primarily, our proposals will concentrate on devising tools and features that empower us, as instance administrators, to moderate our platforms effectively.

In the meantime, while I understand may not be ideal for everyone, users who choose to participate on the beehaw instance will be required to register a separate account on their instance.

Thank you all for continuing to make this community great!

16
submitted 11 months ago by TheDude to c/cybersecurity

Common Fortigate, do better. I really love your firewall.

[-] TheDude 70 points 11 months ago* (last edited 11 months ago)

Hey everyone, woke up this morning with this news. This news really comes at a surprise as I have not seen or heard of any trolling coming from members of this community. I also have not been approached by their admins to see how we could collaborate. In either case, I'll be attempting to reach out to their admins and discuss a path forward together.

I'll post an update with the details in the coming days.

36
submitted 11 months ago by TheDude to c/[email protected]

"Give it to me!" She yelled.
"I'm so fucking wet, give it to me now!
She could scream all she wanted.
I was keeping the umbrella.

** Full disclosure, not my joke but still one I love to share

[-] TheDude 55 points 11 months ago

I am no lawyer, however I do not see a problem with subscribing to NSFW communities outside of this instance. I'd go as far as saying that should you want to contribute to these external communities you can do so as long as you are not using the built in image hosting service to store your content. There are external platforms such as redgifs that could be used to store this content as long as it fit within their TOS.

309
submitted 11 months ago* (last edited 11 months ago) by TheDude to c/main

Good morning,

Some of you may have experienced a small hiccup this morning (eastern time). I went ahead and upgraded our lemmy instance to 0.17.4. I also took this opportunity to make some small adjustments to the instance that required a restart of services. These changes were mainly related to increasing Federation worker count in hopes to decrease synchronization times with other instances. A small fix to help slow down the storage growth was also applied (limiting log sizes that get generated).

Everything should be back up now. In the future my goal is to be more transparent with scheduling downtime.

edit: updated link

[-] TheDude 61 points 11 months ago* (last edited 11 months ago)

Hey all,

I just posted an update here https://sh.itjust.works/post/49787

I probably should have listened to my gut and not have went to bed last night.

Should be fixed now, it was related to an nginx setting that needed to be tuned.

[-] TheDude 173 points 11 months ago* (last edited 11 months ago)

Any government or governing body should be open to criticism. They are suppose to be working for the people they serve. How is anyone going to know better if no one tells them what they are doing wrong? @[email protected] you have my support

view more: next ›

TheDude

joined 11 months ago
MODERATOR OF