Absolutely. This is exactly why It's important that we get a donation strategy put into place sooner rather than later.
This is odd. I did not upgrade pictrs (the service responsible for image hosting). There may have been an error for some specific tables rows during the database migrates. Another possibility could have been an error during the schema update that I believe the new version had to go through.
Would you be able to DM me links to other posts that you come across with broken images? I will look to see if I can identify any common trends that could tell us more.
Thank you
Hello sh.itjust.works community,
It’s been a while since my last update, and I owe you all a more detailed one soon. However, today I’m here to give you a heads-up about the upcoming upgrade.
Over the next 7 days, I’ll be upgrading the instance to v19.5. While this update doesn’t strictly require it, I’ve decided to take this opportunity to upgrade our Postgres database from version 15 to 16 as well. This process can be time-consuming and have take up to 2 hours to complete on other instance.
I'm optimistic that we'll complete it in less time, but to err on the side of caution, I've scheduled the maintenance window for 120 minutes.
To minimize downtime, I've split the upgrade into two phases: Phase one will cover the database update, and Phase two will handle the Lemmy code upgrade.
Note: If the database upgrade is completed in under 60 minutes, I’ll proceed with the Lemmy code upgrade during the same maintenance window.
Here's the tentative timeline:
| Task | Date | Expected Downtime |
| ---------------------------- | ------------------------------------------------ | ----------------- |
| Upgrade of Postgres Database | Friday September 20 2024 @ 8:00PM ET | 60-120 Minutes |
| Upgrade to V19.5 | Monday or Tuesday September 23 or 24 @ 8:00PM ET | Up to 30 Minutes |
During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel
edit: 9:23EDT - Hey all, the database and lemmy codebase has been upgraded to v19.5. The database upgrade took a lot less time than expected so I went right into upgrading the codebase. A whole bunch of background tasks are still running which might make your experience slower than usual. Things should be back to normal in the next couple of hours.
It's coming. This instance is deployed using lemmy-ansible but with some slight modifications. I need to review a few things to make sure the transition is smooth. As you also pointed out, it's also summer time which isn't helping with my free time situation.
I have a long term solution that will speed up upgrades but I only expect to get it in place by the fall. In the meanwhile, I'm planning on getting this instance upgraded to 19.5 in the coming weeks.
Hey,
I'm due to make a post about the instances finances soon. I'll get one posted in the next bit.
I'm still covering costs out of pocket but my goal has been to have this instance fully funded by its members. I want this instance to stay true to its members, the fediverse and be put in a position where it can continue to thrive transparently without me. The admin team has been doing an outstanding job keeping this instance safe and moderated for everyone. I don't thank them enough but they are the true heroes of this instance.
As some may know, I've been working with a local non profit for the past 6 months to have them leveraged in accepting accepting donations for this instance. They have been slow to get things done but its progressing. I may end up doing what I was trying to avoid and create my own non-profit instead if that's what it takes. That direction does have its own challenges and would add additional responsibilities to my plate.
In the meanwhile, while some of the costs of the services have gone up slightly, I get joy knowing that I can continue providing this instance to all of you. I'm OK paying the costs to keeping the lights on until we can transition to donations.
There was a little hiccup this morning unrelated to the migration tonight. The Lemmy services like to be restarted every once in a while. Once we move to the new hardware I'll be able to look into implementing a better logging system and hopefully be more proactive when situations like these happen.
The biggest consumer of storage on this instance is related to the image hosting which we use an external object storage provider for. The second is the database which is no were near the 2TB capacity. 1TB SSDs are cheaper than 2TB SSDs and I also didn't want to spend more than I needed. As other mentioned if we need more space or IOPs in the future, I could accomplish this by adding more drives as a quick fix. This server does not support NVME unless I leverage its PCIe ports but I don't plan on doing that. By the time this instance gets to the point where 10 SSD drives just isn't cutting it anymore I'll probably have come across another opportunities on getting a new server with better NVME support.
Hello sh.itjust.works community,
Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.
In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:
- Dual Xeon 2.9Ghz CPUs (32 cores total)
- 256GB ram
- 4 x 1TB SSD in raid 10 (with room to add 6 more disks)
- 10gbit networking
While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.
- Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
- There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.
Here's the tentative timeline:
|Task |Date| Expected Downtime| |-----|------|----------------------| |Migration to new server| Tuesday February 27 2024 @ 8:00PM ET| 90 Minutes | |Upgrade to V19.3| Thursday February 29 2024 @ 8:00PM ET| Up to 120 Minutes |
- If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
- If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.
During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel
========================================================= Update February 29 2024 We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.
========================================================= Update February 27 2024 We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you
I've started the preparation steps to minimize the maintenance window needed for this upgrade. These operations are running some background tasks and once they are done I'll make an announcement to give everyone a few days heads up of the anticipated upgrade time and date. Should be soon.
Hey!
The best way is to @TheDude on our matrix channel. This action triggers notifications across all my devices, creating a barrage of alerts that almost always grab my attention, a.k.a a ddos to my sensory system.
@kersploosh@sh.itjust.works explained it well. It was related to a small bug on the pictrs service that I've already reported which they quickly fixed in a newer version. I will tackle this upgrade at the same time as I get this instance up to v19
@InEnduringGrowStrong@sh.itjust.works in the perfect world community admins should be moderating their communities and the site admins should only be called upon when there is community abuse or when community mods are not doing any moderation.
As @antik@lemmy.world mentioned below @MichelleG@sh.itjust.works is an alt account for same person on lemmy.world ans she has provided tons of help to this instance.
You are right about one thing, I am currently stretched really thin. If made a list of all my responsibilities I think it would shock a lot of you. That being said the amount of time it takes to run a community is astonishing and running a community well requires a lot more time than I have left to give. There is still a lot of work needed to be spent on this site.
The admins of this site help a huge amount as is mostly due to them that we are still running but I don't want to dump more onto their plates.
I could use a lot more help in all areas (system administration, community moderation, legal revision, moving to accepting donations).
If anyone is interested in helping reach out on Matrix.
Hi there
Sent you a message on the matrix chat. This should now be fixed.
Thank you
The original rules of lemmy.comfysnug.space explicitly allowed for loli content to be posted. I associate loli content with pedophilia and therefore was the reason why I had defederated with them.
Hey all,
I will be locking this voting thread as they do not conform to the Agora guidelines.
- Voting threads should run from Friday to Friday
- Voting threads should be started by a member of The Agora moderation team and must reference the discussion thread.
In addition, it appears that hexbear has reversed their decision and has removed us from their allowed list.
As Lemmy looks to add the ability for users to decide which instances they want to blocked on a per user basis, I hope these types of defederation conversations discussion become less common.
I'd like to give a thanks to the admin at hexbear carc0sa who reached out to us in the last week in order to coordinate some mutual moderation between instances.
Lastly over the last week I've read a lot of back and forth between instances and want to leave you with this.
As people we all have the right to our own opinions and beliefs. Sometimes from time to time we might come across someone who doesn't share these same opinions or beliefs. Instead of using these differences as reasons to divide us, I hope we as people can get to a point where we can use it to make us more understanding, knowledgeable, accepting and united.
As things such as Natural Disasters continue to rise globally there might be a day where the person who you disagree with is the same person who is helping you or a loved one. It won't matter if they are a leftist, rightist, white, black, LGBTQ. What will matter is whether they are a decent human being who understands that while opinions make differences they can choose to not be divide by it.
@Buddahriffic@lemmy.world Please send me a DM with the username and email address used to sign up and I'll manually verify your account.
Could you provide some additional information on what is not functioning well? Screenshots with errors are also very valuable.
Hey all,
I have made some changes to how this instance sends email that should help with the email issues people have been experiencing. If anyone continues to have issues receiving emails please let me know
Thank you
Hey,
I did some research on this.
I will not share the list of accounts as I feel its everyone right to vote and to have their own opinions whether they align with yours or not.
I will however say, Its mostly the same accounts that are down voting you. The majority of the accounts are over 2 months old with content generated by them (they do not seem like bot account). There are a few that are just 10 days old. 99% of the down votes are from other instances with feddit.uk and feddit.nl being the primary source of them.
Please DM me your username and email address used to register and i'll get it sorted out.
Thank you
Please share your thoughts, concerns and opinions. Try to keep it constructive.
Hey all,
Similar to the lemmy.world announcement, we have defederated from Rammy.
Want to discuss this more? I created a [Discussion] post for us to do just that.
Some of you may have noticed that there's a new bot in town. But this time, its one of ours.
What should we name it?
Post your entry.
Rules.
- The 5 most upvoted entries will be brought to a vote.
- The winning entry will become the display name of the bot.
This instance has been upgrade to 0.18.0.
The lemmy developers have removed the built in captcha so I had to enable email verification. They will be re-implementing this feature in the next update.
Lots of new improvements in this version. The Link i wanted to share is on the lemmy.ml instance and they appear to be down at the moment. Hopefully not related to this update.
Stay tuned.
Edit: Lemmy.ml back up, Here is the post: link of the update
Hey all,
As discussed here, I am looking for 3 new moderators to help with this community.
Please review this post and vote for the candidate you would like to see help on this community.
In no particular order, the candidates are:
- @goat@sh.itjust.works
- @Difficult_Bit_1339@sh.itjust.works
- @Apytele@sh.itjust.works
- @Seraph089@sh.itjust.works
- @sweetholymosiah@sh.itjust.works
- @annegreen@sh.itjust.works
- @imaqtpie@sh.itjust.works
I invite you to review users post history when deciding on voting.
This thread will be open for 72 hours (until Friday June 23rd, 8PM ET) at which point voting will stop. Any votes after this time will not be counted. I will try to lock this thread after that time so people do not accidentally continue voting.
Everyone gets 1 vote, including the users being considered. The top 3 will be given mod powers of this community to help with organization of discussions and votes.
Please vote by calling the persons name. For example, if I you wanted to vote for TheDude you would reply with: TheDude
Since making the announcement last night to become community driven, The Agora has quickly become an active place full of discussions, proposals and voting. In order to keep things going I'm looking for volunteers to help facilitate the conversations and apply the rules that have been determine by this community.
As a mod of The Agora you should be active on our main matrix channel and willing to help facilitate conversations between other mods and users.
As a mod:
- you will still retain your right to vote just like anyone else.
- may be dismissed by the community if its felt you are abusing your powers or pushing your own agenda.
- will help maintain order.
- follow the rules of the community.
Not sure you want to help mod this community but would like to nominate someone else? go ahead and nominate someone.
Hey all
Today, I am thrilled to share two significant developments with you that will determine our shared future.
Firstly, the issue of donations. Since the inception of this instance, your most frequent request has been the ability to make contributions to support my initiative. While initially, I had never intended to accept donations, I've come to realize the value this brings in ensuring our platform's sustainability. In response to your requests, within the next week, I will be introducing several options for those of you who wish to donate. I want to emphasize that these donations are entirely optional and will directly support our instance's operational necessities - dedicated hardware, colocation fees, email services, and more.
The second announcement needs a full disclosure: it could be an extraordinary idea or potentially not so. Since the beginning of this instance, I've seen a number of insightful posts, recommendations, and ideas that the community has put forward, often superior to what I could have conceptualized myself. Even our instance's name was born from one of your suggestions.
Frequently, I'm approached by users seeking clarity on our rules and guidelines or expressing their thoughts on existing rules. In reflecting upon this, it became clear to me that I've been attempting to determine what's best for the community. But who am I to make these decisions? Just two weeks ago, I was a user among you all. Hence, I'm moving away from the traditional role of decision-making.
Instead, I want to hand over the reins of decision-making to you - the community. I'm excited to announce the creation of a community called agora where you can express your desires for the future of this instance. It's up to you to come together, discuss, and reach a consensus. If you wish to add, remove, or modify a rule, make a post, garner support from other members, and I'll implement the change. This invitation extends beyond our immediate community - I welcome input from everyone across the fediverse. Again to be clear, I gave an example of modifying rules but this applies to anything that I have the ability to do on this instance.
As the instance owner, I only have one caveat to add - any decisions made should not jeopardize the instance's existence or result in legal complications. Aside from that, I'm eager to see where this new direction takes us.
Thank you for your ongoing support, creativity, and engagement.
Like many of you, I woke up this morning to discover that our instance, along with lemmy.world, had been unexpectedly added to the beehaw block list. Although this development initially caught me off guard, the administrators at beehaw made an announcement shedding light on their decision.
The primary concern raised was our instance's policy of open registration. Given my belief that the fediverse is still navigating its early stages, I believe that for it to mature, gain traction, and encourage adoption, it is crucial for instances to offer an uncomplicated and direct route for newcomers to join and participate. This was one of the reason I decided to launch this instance. However, I do acknowledge that this inclusive approach brings its unique challenges, including the potential for toxicity and trolls. Despite these hurdles, I maintain the conviction that our collective strength as a community can overcome these issues.
After this happened, the beehaw admins and I had a good chat about their decision. While our stances on registration policies might diverge, we realized that our ultimate goals are aligned: we both strive to foster communities that thrive in an atmosphere of safety and respect, where users can passionately engage in discussions and feel a sense of belonging.
Although the probability of an immediate reversal are slim given the current circumstances, I believe we have managed to identify common ground. It's evident that, even in separation, we can unite to contribute positively to the broader fediverse community.
In the coming weeks or months, we plan to collaborate with other lemmy instance administrators to suggest enhancements and modifications to the lemmy project. Primarily, our proposals will concentrate on devising tools and features that empower us, as instance administrators, to moderate our platforms effectively.
In the meantime, while I understand may not be ideal for everyone, users who choose to participate on the beehaw instance will be required to register a separate account on their instance.
Thank you all for continuing to make this community great!
Common Fortigate, do better. I really love your firewall.
"Give it to me!" She yelled. "I'm so fucking wet, give it to me now! She could scream all she wanted. I was keeping the umbrella.
** Full disclosure, not my joke but still one I love to share
Good morning,
Some of you may have experienced a small hiccup this morning (eastern time). I went ahead and upgraded our lemmy instance to 0.17.4. I also took this opportunity to make some small adjustments to the instance that required a restart of services. These changes were mainly related to increasing Federation worker count in hopes to decrease synchronization times with other instances. A small fix to help slow down the storage growth was also applied (limiting log sizes that get generated).
Everything should be back up now. In the future my goal is to be more transparent with scheduling downtime.
edit: updated link
I'm aware that some of you have been getting some errors loading this instance. This was a configuration that I needed to be adjusted which has since been.
Do be patient if we run into other issues, I'll be continuously working in the back-end with others to better your overall experience.
Still a lot of capacity available!
edit: For those that are interested in the configuring changes. I was using the default lemmy configuration for the nginx worker_connections value. This value needed to be raised.
Shout out to @ruud@lemmy.world over at lemmy.world for helping me out.
With r/python out for the next 48 hours, here’s a post to promote continuing discussion a language we all love. Python.
Hey all, I created this community thinking I would have time to mod and post in it but its pretty clear now that this won't be the case. If anyone is interested in taking over this community and has the time to properly moderate it please let me know!
Hello everyone!
I'd like to first start by welcoming all the new users who have decided to chose sh.itjust.works as their lemmy home instance. It's wild to think that its only been 4 days since I've decided to spin up this instance and we are already nearing 1000 registered users.
During these last few days I've welcomed users from all parts of the world who collectively have created nearly 50 new communities on this instance alone.
Today I want to be transparent and share some of my thoughts, clarify some rules, share some concerns, answer some questions and most importantly introduce the two new instance admins that have been added over the last couple of days. So let's get started!
New Admins
- @smorks@lemmy.ca - Some of you may recognized this name from the lemmy.ca instance. This is in fact the same person. I've asked smorks to help out on this instance and he accepted. As the userbase grows having someone whos living some of the similar issues will be bring great value to this instance. We are stronger together!
- @donut@sh.itjust.works - donut is a personal friend, new to the fediverse but has agreed to give me a hand should things get out of control. He's a little quiet right now but he'll open up soon enough.
Rules
As you may have noticed, there aren't a whole lot of rules. This might need to change in the future but my goal as of right now is to keep it this way. Being respectful to all users no matter which instance they come from, part of the world they grew up in, personal believe or personal opinion is a absolute must. Our world is already in shambles, we don't need another place for all of us to fight against one another.
- Be respectful
- No bigotry
- No Ads or Spamming
- No pornography
You'll may have noticed that a new rule that has been added. No pornography. When pornography is introduced a whole bunch of legal implications need to be considered amongst other things like proper moderation, DMCA notices processes, etc. 4 days ago I would not have thought this to be a problem however I am not in a position to hire legal counsel and implement the elements required to not get this instance shutdown. So for now, no pornography. If the demand grows for this in the future and there is an opportunity to fill this gap I promise I will revisit this rule.
Donations
I've had a few users ask me whether they could start donating. I'll admit I was caught off guard when that happened, thinking of how thoughtful people can be. When I started this instance 4 days ago, I did it because I wanted people to have a home away from reddit, a place where they could come and share interesting topics, debate, discuss and connect.
I'm by no means rich but I have chosen a career that puts me in proximity of things needed to run an instance like this one. It also does help that I really like servers and while some people might really like the latest limited edition sneakers from X company or the latest iPhone/Android device, etc..., I really really like servers, datacenters and networking.
I don't have a plan for accepting donations today but seeing how this community is growing, I will need to consider it soon. You know what's more expensive than running a lemmy instances for thousands of users? a failed marriage.
There are some other options that might be available such as looking for sponsors but I do want to keep this instance about the users and not about corporate so I'll need to think hard about this one.
Whatever is decided I'll make another post with the details.
Concerns
Let's quickly talk about concerns. Most of you already know that many of the reddit mods are orchestrating a protest by blacking out many popular subreddits. Most of us anticipate a large spike in traffic towards our instances during the period of June 12 to 14th. If any of you were around during the Digg days, you may remember how the migration from digg to Reddit went. The early reddit days was filled with outages, performance issues, and frustration.
A lot of people people are looking for what might be the next reddit. We don't knows if lemmy will be it but a lot of people will surely be giving it a shot. During that time many instances including this one will likely deal with some growing pains.
From a resources perspective my biggest worry for this instance is related to the storage. I have CPUs, I have RAM but when it comes to storage, well its an element that has been growing much faster than I had anticipated. I will be keeping a close eye on this however want to wave a little yellow flag so that everyone is aware until I work out a solution.
Q&A
What communities allowed to be homed here?
- As long as it follows the rules above and doesn't get any DMCA requests or jeopardize this instance from being shutdown or me in trouble. Go nuts!
What communities allowed to be federated over/subscribed to?
- Same as the above and follow rules.
What am I allowed to link to in communities or text posts?
- Same as the above and follow rules.
What kind of images am I allowed to upload to the integrated image host?
- Storage is something that will likely become a problem in the foreseeable future. You all upload a lot! That being said, don't upload copyrighted materials or anything that will get me in trouble!
A little story that goes along with this image. I asked my wife to give me a prompt of anything to have an AI generate an image. This is what she came up with:
A desert taco with ice cream and chocolate drizzle and topped with sprinkles next to a desert burrito filled with bananas and peanut butter topped with strawberries and English cream
She has since left the home and I fear I might have to pick her up at the local desert shop.
As the reddit mods gets ready for the June 12-14 black-out, there some anticipation that an influx in user base will shift over to many of the lemmy instances as user seek out a home to post their internet memes and discuss their interests.
In anticipation of this increased volume I will be growing our current instance from
- 16 CPU
- 8 GB ram
to
- 24 CPU
- 64 GB ram
This server is currently equipped with SSDs that are configured in a raid 10 array (NVMEs will come in the next gen that get deployed)
Earlier today I also configured some monitoring that I'll be watching closely in order to have a better understanding on how the lemmy platform does under stress (for science!)
I'll be sharing graphs and some other insights in this thread for everyone that is interested. Feel free to ask anything you might be interested in knowing more of!
EDIT: I'll be posting and updating the graphs in this main post periodically! Last updated: 6:21AM ET June 12th
CPU - 48 hours
Memory - 48 hours
Network - 48 hours
Load Average - 48 hours
System Disk I/O - 48 hours
TheDude @sh.itjust.works Entrepreneur, developer, seasoned system and network engineer, aspiring hobbyist pilot, husband and father.