Version update and server upgrade!
Version update and server upgrade!
The instance has been updated to version 0.19.4!
You can see the official release post and notes here: https://lemmy.ml/post/16585359?scrollToComments=true
Since this update required some manual intervention anyway to update the database version, I took the opportunity to perform some overdue server maintenance and upgrades.
I have migrated the server to a more powerful configuration, and changed the way image files are being stored and served (object storage as opposed to filesystem storage).
I am also aware of some federation issues that have affected this and other instances. I am not sure whether this upgrade will deal with these problems automatically. I will be looking into it and hopefully will implement some solution soon.
Sorry for the downtime experienced in the past few days, but hopefully with the more powerful server the experience will be smoother and will remain so for a while.
For other admins and users interested, this is a bit of information about the server
The previous configuration:
Hosting provider: serverspace.io
8 GB RAM
4 Core CPU
50 Mbps network
400 GB SSD
Cost: $55 / month
The 400 GB SSD was being used to store both the server's database and the image files, and this component in isolation had a monthly cost of $36. I have migrated the images to an object storage provider ($6/month for 500 GB), and used the resources previously used for disk space to increase the other parameters. The new configuration is a lot better and less expensive.
New configuration:
Hosting provider: serverspace.io
Object storage provider: contabo.com
16 GB RAM
6 core CPU
50 Mbps network
100 GB SSD
500 GB Object storage
Cost: $47/month
Thanks a lot to the lemmy and pict-rs devs for the hard work they put into developing the infrastructure!
Now update to version 0.19.5, please
I hope that's a joke, give this poor admin some rest!
As soon as it comes!
Well done!
Thanks!
Since the upgrade, I can see (here) that activities from lemmy.world are being fetched significantly faster. I will let it run for some time and hopefully this speed-up is enough to catch up.
Great, looks promising!
You seem also to be catching up with the backlog somehow? https://grafana.lem.rocks/d/cdfzs0dwal3pca/federation-health-time-behind?orgId=1&var-instance=All&var-remote_instance=mander.xyz&var-remote_software=All&from=now-12h&to=now
Hopefully the trend will continue, let's wait a few hours and see
Yay, down to one week delay. Looking forward to catching up.
FYI, I just tried posting from a LW account 4 minutes, the comments still don't show up: https://lemmy.world/comment/10702133
Sal, I got some error here:
couldnt_get_posts. That appears every time that I try to visualise the instance through the "Subscribed" communities.The error does not appear if I browse the instance through "Local" or "All", nor if I check the list of communities that I'm subscribed to. The error happens across devices and front-ends (tested with Linux Firefox, Android Firefox, Jerboa).
EDIT/UPDATE: I unsubscribed manually to a few communities each time, checking if the front page loaded fine. Eventually, it did.
Re-subscribing to either !technology@lemmy.world or !fediverse@lemmy.world seems to trigger the error again.
(@Shdwdrgn@mander.xyz ) I think it is related with the processing of so many activities per second while syncing with Lemmy.World.
I have tried changing some rate limits and the number of maximum database connections, but I get similar errors while moving around the site. The server is running well below 100% on all metrics, and the logs don't produce an obvious error. I will wait until we catch up with Lemmy.World and see if that fixes the issue. If it doesn't, then I will trouble shoot more in-depth.
Thank you!
There's some potential solutions to this mentioned here https://github.com/LemmyNet/lemmy/issues/4277
That's the same error I was asking about below, and wondering if it had to do with the current status of collecting all the content. If so, it sounds like it could be another day or two before we can view our front pages again?
Sorry for the double reply.
Check if you're subscribed to any of those communities: !technology@lemmy.world , !comicstrips@lemmy.world or !memes@lemmy.world. If you are, try to unsubscribe and see if your front page works again. For me it did the trick.
!fediverse@lemmy.world was also giving me problems, but apparently it's fine now, so odds are that the problem will eventually go away on its own.
For now, one option is to temp change your settings to browse by "Local" or "All". Or alternatively temporarily unsubscribe off a few comms; if you're subscribed to a small number of comms, some trial and error might help you to find which ones are causing the issue. (For me it was two LW comms).
Having the same issue here unfortunately.
Wow the new server is just flying along!
Good to hear!
Lets gooooo🔥🔥🔥
Well done!
🎉 🌞
Well done and many thanks!