TheDude

joined 2 years ago
MODERATOR OF
[–] TheDude 2 points 3 months ago (1 children)

Absolutely. This is exactly why It's important that we get a donation strategy put into place sooner rather than later.

[–] TheDude 4 points 3 months ago (1 children)

This is odd. I did not upgrade pictrs (the service responsible for image hosting). There may have been an error for some specific tables rows during the database migrates. Another possibility could have been an error during the schema update that I believe the new version had to go through.

Would you be able to DM me links to other posts that you come across with broken images? I will look to see if I can identify any common trends that could tell us more.

Thank you

[–] TheDude 4 points 3 months ago (2 children)
2
test post (self.test)
submitted 3 months ago by TheDude to c/test
 

test post after upgrade

129
SJW upgrade to v19.5 (sh.itjust.works)
submitted 3 months ago* (last edited 3 months ago) by TheDude to c/main
 

Hello sh.itjust.works community,

It’s been a while since my last update, and I owe you all a more detailed one soon. However, today I’m here to give you a heads-up about the upcoming upgrade.

Over the next 7 days, I’ll be upgrading the instance to v19.5. While this update doesn’t strictly require it, I’ve decided to take this opportunity to upgrade our Postgres database from version 15 to 16 as well. This process can be time-consuming and have take up to 2 hours to complete on other instance.

I'm optimistic that we'll complete it in less time, but to err on the side of caution, I've scheduled the maintenance window for 120 minutes.

To minimize downtime, I've split the upgrade into two phases: Phase one will cover the database update, and Phase two will handle the Lemmy code upgrade.

Note: If the database upgrade is completed in under 60 minutes, I’ll proceed with the Lemmy code upgrade during the same maintenance window.

Here's the tentative timeline:

Task Date Expected Downtime
Upgrade of Postgres Database Friday September 20 2024 @ 8:00PM ET 60-120 Minutes
Upgrade to V19.5 ~~Monday or Tuesday September 23 or 24 @ 8:00PM ET~~ ~~Up to 30 Minutes~~

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

edit: 9:23EDT - Hey all, the database and lemmy codebase has been upgraded to v19.5. The database upgrade took a lot less time than expected so I went right into upgrading the codebase. A whole bunch of background tasks are still running which might make your experience slower than usual. Things should be back to normal in the next couple of hours.

[–] TheDude 6 points 4 months ago (1 children)

Hi @[email protected]

It's coming. This instance is deployed using lemmy-ansible but with some slight modifications. I need to review a few things to make sure the transition is smooth. As you also pointed out, it's also summer time which isn't helping with my free time situation.

I have a long term solution that will speed up upgrades but I only expect to get it in place by the fall. In the meanwhile, I'm planning on getting this instance upgraded to 19.5 in the coming weeks.

[–] TheDude 61 points 5 months ago (1 children)

Hey,

I'm due to make a post about the instances finances soon. I'll get one posted in the next bit.

I'm still covering costs out of pocket but my goal has been to have this instance fully funded by its members. I want this instance to stay true to its members, the fediverse and be put in a position where it can continue to thrive transparently without me. The admin team has been doing an outstanding job keeping this instance safe and moderated for everyone. I don't thank them enough but they are the true heroes of this instance.

As some may know, I've been working with a local non profit for the past 6 months to have them leveraged in accepting accepting donations for this instance. They have been slow to get things done but its progressing. I may end up doing what I was trying to avoid and create my own non-profit instead if that's what it takes. That direction does have its own challenges and would add additional responsibilities to my plate.

In the meanwhile, while some of the costs of the services have gone up slightly, I get joy knowing that I can continue providing this instance to all of you. I'm OK paying the costs to keeping the lights on until we can transition to donations.

[–] TheDude 8 points 9 months ago* (last edited 9 months ago) (2 children)

Yes, I'll get this done

UPDATE: here you go https://oldsh.itjust.works/

[–] TheDude 24 points 9 months ago (1 children)

There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I'll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.

[–] TheDude 28 points 10 months ago (1 children)

The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn't want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don't plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn't cutting it anymore I'll probably have come across another opportunities on getting a new server with better NVME support.

557
submitted 10 months ago* (last edited 9 months ago) by TheDude to c/main
 

Hello sh.itjust.works community,

Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.

In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:

  • Dual Xeon 2.9Ghz CPUs (32 cores total)
  • 256GB ram
  • 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
  • 10gbit networking

While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.

  1. Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
  2. There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.

Here's the tentative timeline:

Task Date Expected Downtime
Migration to new server Tuesday February 27 2024 @ 8:00PM ET 90 Minutes
Upgrade to V19.3 Thursday February 29 2024 @ 8:00PM ET Up to 120 Minutes
  • If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
  • If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.

=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you

[–] TheDude 7 points 10 months ago

I've started the preparation steps to minimize the maintenance window needed for this upgrade. These operations are running some background tasks and once they are done I'll make an announcement to give everyone a few days heads up of the anticipated upgrade time and date. Should be soon.

[–] TheDude 2 points 1 year ago (1 children)

Hey!

The best way is to @TheDude on our matrix channel. This action triggers notifications across all my devices, creating a barrage of alerts that almost always grab my attention, a.k.a a ddos to my sensory system.

@[email protected] explained it well. It was related to a small bug on the pictrs service that I've already reported which they quickly fixed in a newer version. I will tackle this upgrade at the same time as I get this instance up to v19

[–] TheDude 12 points 1 year ago (1 children)

@[email protected] in the perfect world community admins should be moderating their communities and the site admins should only be called upon when there is community abuse or when community mods are not doing any moderation.

As @[email protected] mentioned below @[email protected] is an alt account for same person on lemmy.world ans she has provided tons of help to this instance.

You are right about one thing, I am currently stretched really thin. If made a list of all my responsibilities I think it would shock a lot of you. That being said the amount of time it takes to run a community is astonishing and running a community well requires a lot more time than I have left to give. There is still a lot of work needed to be spent on this site.

The admins of this site help a huge amount as is mostly due to them that we are still running but I don't want to dump more onto their plates.

I could use a lot more help in all areas (system administration, community moderation, legal revision, moving to accepting donations).

If anyone is interested in helping reach out on Matrix.

53
submitted 1 year ago* (last edited 1 year ago) by TheDude to c/agora
 

Please share your thoughts, concerns and opinions. Try to keep it constructive.

Link to announcement

55
submitted 1 year ago* (last edited 1 year ago) by TheDude to c/main
 

Hey all,

Similar to the lemmy.world announcement, we have defederated from Rammy.

Want to discuss this more? I created a [Discussion] post for us to do just that.

25
submitted 1 year ago by TheDude to c/agora
 

Some of you may have noticed that there's a new bot in town. But this time, its one of ours.

What should we name it?

Post your entry.

Rules.

  • The 5 most upvoted entries will be brought to a vote.
  • The winning entry will become the display name of the bot.
124
submitted 2 years ago* (last edited 2 years ago) by TheDude to c/main
 

This instance has been upgrade to 0.18.0.

The lemmy developers have removed the built in captcha so I had to enable email verification. They will be re-implementing this feature in the next update.

Lots of new improvements in this version. The Link i wanted to share is on the lemmy.ml instance and they appear to be down at the moment. Hopefully not related to this update.

Stay tuned.

Edit: Lemmy.ml back up, Here is the post: link of the update

35
submitted 2 years ago by TheDude to c/agora
 

Hey all,

As discussed here, I am looking for 3 new moderators to help with this community.

Please review this post and vote for the candidate you would like to see help on this community.

In no particular order, the candidates are:

I invite you to review users post history when deciding on voting.

This thread will be open for 72 hours (until Friday June 23rd, 8PM ET) at which point voting will stop. Any votes after this time will not be counted. I will try to lock this thread after that time so people do not accidentally continue voting.

Everyone gets 1 vote, including the users being considered. The top 3 will be given mod powers of this community to help with organization of discussions and votes.

Please vote by calling the persons name. For example, if I you wanted to vote for TheDude you would reply with: TheDude

 

Since making the announcement last night to become community driven, The Agora has quickly become an active place full of discussions, proposals and voting. In order to keep things going I'm looking for volunteers to help facilitate the conversations and apply the rules that have been determine by this community.

As a mod of The Agora you should be active on our main matrix channel and willing to help facilitate conversations between other mods and users.

As a mod:

  • you will still retain your right to vote just like anyone else.
  • may be dismissed by the community if its felt you are abusing your powers or pushing your own agenda.
  • will help maintain order.
  • follow the rules of the community.

Not sure you want to help mod this community but would like to nominate someone else? go ahead and nominate someone.

 

Hey all

Today, I am thrilled to share two significant developments with you that will determine our shared future.

Firstly, the issue of donations. Since the inception of this instance, your most frequent request has been the ability to make contributions to support my initiative. While initially, I had never intended to accept donations, I've come to realize the value this brings in ensuring our platform's sustainability. In response to your requests, within the next week, I will be introducing several options for those of you who wish to donate. I want to emphasize that these donations are entirely optional and will directly support our instance's operational necessities - dedicated hardware, colocation fees, email services, and more.

The second announcement needs a full disclosure: it could be an extraordinary idea or potentially not so. Since the beginning of this instance, I've seen a number of insightful posts, recommendations, and ideas that the community has put forward, often superior to what I could have conceptualized myself. Even our instance's name was born from one of your suggestions.

Frequently, I'm approached by users seeking clarity on our rules and guidelines or expressing their thoughts on existing rules. In reflecting upon this, it became clear to me that I've been attempting to determine what's best for the community. But who am I to make these decisions? Just two weeks ago, I was a user among you all. Hence, I'm moving away from the traditional role of decision-making.

Instead, I want to hand over the reins of decision-making to you - the community. I'm excited to announce the creation of a community called agora where you can express your desires for the future of this instance. It's up to you to come together, discuss, and reach a consensus. If you wish to add, remove, or modify a rule, make a post, garner support from other members, and I'll implement the change. This invitation extends beyond our immediate community - I welcome input from everyone across the fediverse. Again to be clear, I gave an example of modifying rules but this applies to anything that I have the ability to do on this instance.

As the instance owner, I only have one caveat to add - any decisions made should not jeopardize the instance's existence or result in legal complications. Aside from that, I'm eager to see where this new direction takes us.

Thank you for your ongoing support, creativity, and engagement.

424
submitted 2 years ago* (last edited 2 years ago) by TheDude to c/main
 

Like many of you, I woke up this morning to discover that our instance, along with lemmy.world, had been unexpectedly added to the beehaw block list. Although this development initially caught me off guard, the administrators at beehaw made an announcement shedding light on their decision.

The primary concern raised was our instance's policy of open registration. Given my belief that the fediverse is still navigating its early stages, I believe that for it to mature, gain traction, and encourage adoption, it is crucial for instances to offer an uncomplicated and direct route for newcomers to join and participate. This was one of the reason I decided to launch this instance. However, I do acknowledge that this inclusive approach brings its unique challenges, including the potential for toxicity and trolls. Despite these hurdles, I maintain the conviction that our collective strength as a community can overcome these issues.

After this happened, the beehaw admins and I had a good chat about their decision. While our stances on registration policies might diverge, we realized that our ultimate goals are aligned: we both strive to foster communities that thrive in an atmosphere of safety and respect, where users can passionately engage in discussions and feel a sense of belonging.

Although the probability of an immediate reversal are slim given the current circumstances, I believe we have managed to identify common ground. It's evident that, even in separation, we can unite to contribute positively to the broader fediverse community.

In the coming weeks or months, we plan to collaborate with other lemmy instance administrators to suggest enhancements and modifications to the lemmy project. Primarily, our proposals will concentrate on devising tools and features that empower us, as instance administrators, to moderate our platforms effectively.

In the meantime, while I understand may not be ideal for everyone, users who choose to participate on the beehaw instance will be required to register a separate account on their instance.

Thank you all for continuing to make this community great!

 

Common Fortigate, do better. I really love your firewall.

view more: next ›