I'm just excited to be back in the Wild West again -- all of the big players had bumps, at least this one is working to fix them.

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Lemmy has a memory leak? Or, should I say, a "lemmory leak"?

Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn't happen without you!

I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.

Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.

Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.

Seriously, you're doing a great job <3

i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.

but please, don't feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn't like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.

in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.

this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^

As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I'm remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.

As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You're doing great with such fast exponential growth

💙 Thanks for your hard work!

This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.

What was that? We're going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?

Yeah, that sounds pretty reasonable, we can work with that.

Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time

Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.

@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻

The work you're doing is greatly appreciated! It's like you invited half the internet into your house. I feel like I should've brought a cake or something

It’s a ton better this afternoon! Thank you!!

Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!

Huge respect for what you've built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here

.world is definitely running smoother than when I joined 3 days ago, back then it was impossible to comment and the lag was immense, right now I just have to occasionally reload the page, but that's nothing in comparison.

You guys are doing an amazing work! I'm broke, so here are some coins 🪙🪙🪙🪙 beans 🫘🫘🫘🫘

Cloud architect here— I’m sure someone’s probably already brought it up, but I’m curious if any cloud native services have been considered to take the place of what I’m sure are wildly expensive server machines. E.g. serve frontends from cloudfront, host the read-side API on Lambda@Edge so you can aggressively and regionally cache API responses, anything other than an SQL for the database — model it in DynamoDB for dirt cheap wicked speed, or Neptune for a graph database that’s more expensive but more featureful. Drop sync jobs for federated connections into SQS, have a lambda process that too, and it will scale as horizontally as you need to clear the queue in reasonable time.

It’s not quite as simple to develop and deploy as docker containers you can throw anywhere, but the massive scale you can achieve with that for fractions of the cost of servers or fargate with that much RAM is pretty great.

Or maybe you already tried/modeled this and discovered it’s terrible for you use case, in which case ignore me ;-)

Please keep working on it, thank you for your effort.

That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

who'd have thought memory leaks would be possible in Rust 🤯

(sorry not sorry Rust devs)

Keep up the good work!

I created an account in lemm.ee until the issues are fixed. Then I will happily go back to my lemmy.world account.

good news, a fix might be in the works: https://github.com/LemmyNet/lemmy/pull/3482

Thank you for your effort!

I am very forgiving of the bugs I encounter on Lemmy instances because Lemmy is still growing and it's essentially still in beta. I am totally unforgiving of Reddit crashing virtually every day after almost two decades.

System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Shouldn't we be discussing closing registrations?

The need to restart server every so often to avoid excessive ram usage bit is very interesting to me. This sounds like some issue with memory management. Not necessarily a leak, but maybe something like server keeping unnecessary references so the object cannot be dropped.

Anyway, from my experience Rust developers love debugging such kind of problems. Are Lemmy Devs aware of this issue? And do you publish server usage logs somewhere to look deeper into that?

How can I donate to lemmy.world?

People of lemmy.world should encourage the new users joining up on lemmy.world to other create accounts on other instances instead

To all the folks that are worried, don't be. Let me tell you, Mastodon was a wreck when Musk took over Twitter and that all got sorted out within a week or so. The mods and sysadmin are obviously working hard to get things up and running, but growing pains are growing pains. To paraphrase an old adage: "Facebook wasn't built in a day." In the beginning, Zuck and Co literally limited signups to only people with college email accounts and only added universities a few domains at a time... scaling is very difficult, but it's not impossible. The way things are going, Lemmy is going to thrive!

Would donations help in the long run? How can we donate?

Thank you for everything! Can we donate to cover the costs? If more people throw in $5, we all will benefit. Can Lemmy server scale up without needing RAM exponentially growing with the user number? I hope the system will get better optimized for bigger user base as the time goes on..

Thanks again for all the hard work on Lemmy World. It feels fast today

Thanks for giving us all these updates all the time!

It has become way smoother the last few hours. The work you guys from .world and Lemmy's devs have been doing is amazing!

Thanks team, love your work! Appreciate the updates and improvements but happy to have this place despite the bugs and quirks :)

I'm very much so okay putting up with hiccups right now. I know things will improve in the future.

Thank you and everyone for all your hard work!

I love the community here and am excited to see it grow. Thanks for giving a report, it awesome how invested y'all are and makes me want to stick with this instance even through the hardships. Keep up the good stuff!

I feel that lemmy runs smoothly the past few hours with very few hiccups and mostly on the upvoting and commenting side. I encountered no issues yet on the loading of the post.

Thank you so much for all your hard work.

The good thing about the fediverse is we can sign up on federated servers and still interact with everything on lemmy.world without the errors, then come back to our lemmy.world accounts when everything is smoother

I can't imagine how hard it is to rua server that has influx of large users. I Thank you for your hard work to run and maintain this instance. Hope it works out well and future will be a smooth sail.

Its amazing this site hasn't completely melted down yes it sucks trying to use it and having nothing load but the fact that the site has mostly worked when I want to use it and the fact you guys have less than 1% of 1% of reddits resources is mind blowing

I am sure the stress is unreal, but It is appreciated. You all stepped up when everyone was stepping down (or trying to sellout). I am hopeful this will turn into something great.

Thanks for the transparent update!

Thank you for all of your effort on this. Adjusting to rapid growth is a huge challenge, and we all appreciate the effort you and everyone else is putting in to trying to make this instance run smoothly.

The redirects to baseball games glitch is gone. I am unsure if this is because the glitch is fixed or if baseball season is over.

Can anyone give me a link to donate to help out? Ever since discovering lemmy I hardly use reddit and would like to contribute to this community more!

Permanently Deleted

Thanks u/ruud!

How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.

Thanks for the update, you're leading the way for Lemmy.

I was prepared for servers crashing and lagging, but you're keeping things moving thus far, despite receiving 85k new users in a month. Heroic stuff.

Thanks for the transparency, and communication. I think it's always better when the userbase is able to understand what is going on rather than being left in the dark wondering what is going wrong. Keep up the good work, but also be sure you guys get enough rest, and take care of yourselves too.

Thank you for your hard work! ♥️

Thank you for the transparency @ruud@lemmy.world It is rare for platforms to scale this quickly so issues like this are inevitable. Good luck for the troubleshooting!

Thanks for the transparency and keep up the good work!

Y’all doing amazing things keeping us going 🔥

This has been such a refreshing change from reddit! I'm willing to deal with occasional hiccups, and I know it will only get better with time. Keep it up!

Thanks for keeping us apprised. Hopefully you will find the resource leak.

What sort of container orchestration and database tools are you using? Maybe you need some sys admin help?

I totally appreciate all of your efforts! Thank you for being a pioneer.

around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

25 GB of RAM????? Looks like your instance is the edge case with Lemmy's scalability, when there's really big amount of active users. Looks like some problems won't fade with really high-grade hardware. But I have to appreciate, that's one gargantuan job!

Thanks for all the work you, and all the others do to welcome so many new users, and thank you for keeping us posted on your efforts to smooth things over.

From what I see the general sentiment is very positive and understanding, and it feels special to be part of a community that's more upbeat than what you see in most online places.

I expect it is federation outbound activity growing in memory usage, it's all held in RAM for every post/comment/like going out to subscribed servers. !lemmyperformance@lemmy.ml is a community for the focus on scaling Lemmy.

Thanks for making this possible in the first place 🤘

I can come whine in the discord if that'll help.

Its nice to have a place where the admins and developers aren't trying to scheme and manipulate the community to get a big payout. Thanks for all that you guys do. Curating this community is no doubt selfless hard work but we need a place like this free from the influence or potential take over by bad actors and malignant narcissists.

Amazing work team. I am already seeing improvements. Hope you are not killing yourselves though, I'm sure everyone realises how difficult it is and that it will take time to fix. We're here for the long haul! Thanks again

Thank you to you and everyone behind the scenes working hard to make Lemmy a great place. I am turkey enjoying the experience even with the bugs. Brings back a ton of nostalgia with modern mixed in.

Thank you for everything you do. With any luck, you'll get all the support you need, and have it running smooth as silk.

Yall are navigating waters that you probably have little professional experience with. Keep at it and don't get discouraged.

Something changed in the last 24hrs that fixed my account! I couldn't login on desktop at all. Tried Chrome, cleared the cache, tried Firefox, tried an Ubuntu live USB - login never worked. Jerboa could access my account that was already logged in, but trying a new login would result in a HTML error. But today login worked perfectly, zero issues at all.

Do you need any spare server hardware? @ruud@lemmy.world I would be happy to donate some!

Thanks for putting in all the hard work. Especially on a holiday. We appreciate you guys!

It’s a ton better this afternoon! Thank you!!

Appreciate the update, best of luck.

Are there any plans to make user up and down votes not viewable publicly? I know for a lot of new adopters this can be a deal breaker.

Sometimes when I post a comment (without any image) it takes forever to post and sometimes it keeps spinning and I need to cancel and post again. My internet connection is fine (100 Mbps) and it works flawlessly on other websites. Is it happening due to overloading on the lemmy server?

Thanks y'all.

Thanks! Love updates. I moved to lemm.ee for now, I'll come back to your world when you got room for my fat arse.

Thanks for your work, you are making the Internet a better place!

Thanks for the update!

If you need any Python QA test automation help I’m happy to invest. Thanks for the new home.

Keep up the great work! We believe in you! Just a reminder to everyone to help sustain lemmy.world by donating in the sidebar! If 1000 people all donated $2/month then that's 24K a year to keep the hardware running! Reddit is what happens when you rely on Corporate Overlords to provide the services you love.

Keep up the good work. It's nice to feel like we are back on the bleeding edge.

That DM sending bug was already fixed in a later rc version. Although rc.9 and I think 10 have a weird bug where you can't sign out. Fun stuff.

Encouraging to hear about how much help you're getting. A happy lemmyverse is a happy fediverse?

Thanks a LOT for your work. It's amazing seeing what people can do when they work together for a good cause.

I switched to a regional instance to take the load off for everyone, Lemmy is even better!

Again, thanks for all your hard work, and your transparency about the whole ordeal. Hopefully, the userbase in general is being patient and understanding given the situation.

Keep up the good work! Things have been really snappy the past few hours 😎

More props for you and the other admins as you ride the waves. It’s for a great cause.

Thanks for the hard work and transparency. It gives confidence that you guys will get through this.

It is hard not to root for your success in this. I'm temporarily off .world, but would love to come back when you guys get this under control more.

Keep up the good work @Ruud and you all wonderful admins, mods and lemmings for making this possible! 💙💚

That’s cool. Appreciate the work to you all!

All these updates are really much appreciated. I would love to see the conversations happening in this discord but I understand that you would want to limit that to only a small group: do you have someone making sure this is archived somewhere, and available in a few years? That would be a nice read then.

As a former programmer, I appreciate the hard work you folks are putting in so the rest of us can putter away our free time. You're doing great. Thank you.

Can we configure the website to not automatically collapse deep replies? Or at least to collapse them only past six or seven layers deep?

Nice! It feels better now.

I appreciate the transparency. Keep it up! Everyone is loving it here (⁠◍⁠•⁠ᴗ⁠•⁠◍⁠)⁠❤!

The biggest frustration is 502 errors on submit. Sometimes th8ngs go through, sometimes not

I hope hosting all these massive communities is enough a portfolio to land Ruud a crazy job offer one day that pays him stupid levels of money while allowing him plenty of time off to maintain these massive hobby projects.

Appreciate the clarity and hope things keep moving in the right direction

Just wanted to say thank you all for you efforts and dedication.

It’s really good to know that there are more people helping sorting everything out.

I’d just like to suggest publishing the workarounds and fixes public so that other small instances can get it sorted faster when they grow.

Thank you. Y'all are doing great.

I'm sure it's stressful but I know I'm here to stay. It's been a blast so far

I recall that previously we couldn't update jerboa due to it not being compatible with the last server version. Is that updated now, can we update Jerboa?

Using the connect for Lemmy app produces constant Under Maintenance or html code or 504 errors. It seems to load better on Chrome.

Thanks for your hard work. Noticing janky bits getting ironed out everyday. Has anyone had any luck making Lemmy.world work in Qutebrowser? It just hangs on loading comments/posts..Any work arounds? I'd love that..working quite well in Firefox and mobile though.

It's not perfect but becoming better everyday.

Thanks for all of the updates and work on this instance! As a fellow sys admin, I sympathize with the issues trying to get this update going.

Glad to see these progress updates. Thanks for the transparency.

I really appreciate the updates. As someone in the tech field, it's been fascinating getting the inside scoop on how it's all coming together.

Edit: I made another account off of your instance to ease up on the pressure, but so many communities and users I care about are on your instance, so I'm using it too!

It seems fine right now. Thank you.

Don't know about anyone else but I keep getting a "cannot load comments"error and the whole post goes blank.

Yup, I'm having the 2FA problem. I can't log into mobile, because no code gets generated. Luckily I'll just stay logged into a browser for now.

But thanks for all the hard work! :) You're probably getting a flux of Reddit migrants, such as myself now

Permanently Deleted

Thanks a lot for addressing these issues. It's really nice to see how this platform keeps growing and growing.

Your work is much appreciated!

Thanks for the update, glad you guys are being transparent about the issues everyone seems to be facing. Looking forward to the coming improvements. 😁

Thank you for everything you’ve done and continue to do!

Is there anything that people can do to help if they’re unable assist in troubleshooting?

ISO 8601 in the wild :o

Thanks again for the hard work that you and the others are putting in Ruud. It can't be easy trying to patch the boat while the boat keeps growing ridiculously fast at the same time. You are very much appreciated!

Thanks for all your efforts.

I'm sorry to hear you're using Discord, for me it fits right in with the likes of Reddit, Twitter and Facebook. Why not use something independant and federated like Matrix or XMPP?

Thank you for all your hard work!

Great work! We believe in you guys.

Thanks for working so hard on this and keeping lemmy going. This instance seems to have become synonymous with lemmy as a whole for some of the new comers so keeping it up and running is really helping to keep the new comers on board, this can only be good for all of lemmy in the long run!

Really appreciate the transparency about what's going on behind the scenes too. It's nice to know what my donation is doing and I hope you have time for cappuccinos.

Hi, am I correct that typing on iPad currently doesn’t work? As in the keypad won’t show up. I have to type in google or notepad and paste (the paste popup still works) to lemmy.world lol! Anyone else?

As always, thanks for the hard work everybody puts into this!

I think most of the users can understand the current situation. Take care guys and girls!

You lot are internet heroes. Thank you for putting in the work and creating this community!

Thanks for the update!

So seems that Reddit refugees influx indeed hit this instance.

Thanks u/ruud!

How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.

Thank you for your hard work and transparency

Appreciate all the work. I’ve only been on Lemmy a few days but the openness is refreshing and I think this place has been pretty cool.

In which country is this server located?

Huge heartfelt thank you to everyone involved - given the huge growth you're doing awesome!

It would be great to see plans on how to grow in a controlled and safe way and plans for improving the performance. Really hope you have the funds needed and continue to have the income to support everything needed.

Thank you again, it's wonderful to get an update!

Huge heartfelt thank you to everyone involved - given the huge growth you're doing awesome!

It would be great to see plans on how to grow in a controlled and safe way and plans for improving the performance. Really hope you have the funds needed and continue to have the income to support everything needed.

Thank you again, it's wonderful to get an update!

Anything to be said for using me cache or redis?

Connect for Lemmy, Android and yeah still happening.

Thanks for all your hard work. Wow that is a lot of CPU usage. I hope it can be tuned more. Is it mostly postgres causing the load?

Curious about the server info. Is there more info available somewhere else?

Is lemmy.world running on a single server? What's the CPU and memory capacity? You mentioned a percentage of CPU usage and an amount for ram, so that made me curious about what's available for each.

Does restarting take down all services or is the application clustered or not monolithic somehow?

Growing Pains

As much as I hope that Lemmy grows, I wouldn't want lemmy.world to suddenly get 1 million signups and multiply your headaches.

Perhaps we need more well run servers by other people to take the load.