CPU:

!

Memory:

!

Network:

!

Storage:

!

Cloudflare caching:

!

Summary:

All resource usage looking stable. Storage is the only one that is trending up, as expected.

Another week... another bunch of nerd graphs!

CPU:

! Not much to say here, pretty stable CPU usage wise.

Memory:

! The unusual memory growth appears to have been related to a minor Postgres configuration change I made last week, which was reverted on Thursday. Memory usage looking much more normal since.

Network:

! As with CPU usage, network traffic is looking stable.

Storage:

! Storage growth has normalized, now that we've hit an equilibrium point. Though I'll be tweaking the object storage cache retention to minimise object storage pulls.

Cloudflare caching:

! Still saving us a large volume of egress traffic. Will save even more if particular content goes viral.

Summary:

Resource utilisation on the server is looking great across the board. No skyrocketing usage as we saw initially. Storage is still looking like the first trigger for another server upgrade, but as it is now a gradual increase we'll have plenty of fore warning and its looking like this will be some time away.

Questions? 🤓

Any chance we could get a setting to toggle the display of avatars in comments?

Doh! Forgot earlier in the night, so here you are... technically Saturday.

CPU:

! The lemmy devs have made some major strides in improving performance recently, as you can see by the overall reduced CPU load.

Memory:

!

I need to figure out why swap is continuing to be used, when there is cache/buffer available to be used. But as you can see, the upgrade to 8GB of RAM is being put to good use.

Network:

!

The two large spikes here are from some backups being uploaded to object storage. Apart from that, traffic levels are fine.

Storage:

! A HUGE win here this week, turns out a huge portion of the database is data we don't need, and can be safely deleted pretty much any time. The large drop in storage on the 9th was from me manually deleting all but the most recent ~100k rows in the guilty table. Devs are aware of this issue, and are actively working on making DB storage more efficient. While a better fix is being worked on, I have a cronjob running every hour to delete all but the most recent 200k rows.

Cloudflare caching:

!

Cloudflare still saving us substantial egress traffic from the VPS, though no 14MB "icons" being grabbed thousands of times this week 😀

Summary:

All things considered, we're in a much better place today than a week ago. Storage is much less of a concern, and all other server resources are doing well... though I need to investigate swap usage.

Longer term it still looks as though storage will become the trigger for further upgrades. However storage growth will be much more slow and under our control. The recent upward trend is predominantly from locally cached images from object storage, which can be deleted at any time as required.

As usual, feel free to ask questions.

Is it possible for Voyager to display user avatars in the same way as the regular web page? Its how I recognize regulars in my communities, more so than by name.

Available here

!

Noting this here for my own reference, and any other lemmy server admins that don't happen to be database administrators by day 🙂

I am not a DBA, if I'm doing something bad/incorrect here... please post! Yes, I've reset the password and TOTP token on the example account below.

2FA flags are stored in the local_user table, however that does not show usernames. To find the person_id for the user account you want to disable 2FA for, you'll need to check the person table. I'll use my test account here as an example:

SELECT * from person where name = 'guineapig' and local = 't';

Giving: !

Note the number 781227, this is the person_id for this account on my instance. To confirm: SELECT * from local_user where person_id = '781227';

!

Yep, the 2FA string has the expected username in it. Now to disable 2FA on the account we need to NULL out both totp_2fa_url and totp_2fa_secret rows: UPDATE local_user SET totp_2fa_url = NULL WHERE person_id = 781227;

UPDATE local_user SET totp_2fa_secret = NULL WHERE person_id = 781227;

Should give output like this: !

And checking the local_user table again, both TOTP fields should be empty:

!

I'll be upgrading lemmy to 0.18.2 shortly. Expect a minute or so downtime.

The server will be undergoing a reboot shortly to add additional RAM.

This is our toaster, we call him Mr Toot... for obvious reasons.

Another week... another set of nerd graphs!

CPU:

! Some of the variations in CPU usage in the last week correlate with upgrades to the lemmy software. If you look closely, you can see a drop in CPU usage starting late on Thursday, this was when I upgraded to 0.18.1-rc.10 which includes some major DB query optimisations.

Memory:

! The spikey memory utilisation between the 28th of June and the 4th of July were due to a lemmy software issue. It resulted in huge resource consumption for varying amounts of time, before returning to normal. This specific issue seems to have been resolved in one of the upates this week.

Network:

! The unusually high traffic levels between the 1st and 4th were caused by a large community icon.. 14MB. Thankfully, Cloudflare saved us by caching this file and saved us ~800GB of traffic on this file alone before it was shrunk to a more appropriate size 🙂

Storage:

! As expected, disk space is being used up at a relatively consistent rate. The drop on the 28th was when cached object storage images were deleted due to a server reboot that was required when adding 2 vCPUs. I have a scheduled job that runs every hour to delete cached objects older than a threshold. As the disk used by the cache increases (or the disk gets closer to being full), I'll decrease the age threshold.

Cloudflare caching:

! Cloudflare caching is saving us a lot of egress traffic. It is saving us less since the 14MB "icon" was shrunk, however this shows the value in the service. Even now, it is saving ~40GB of egress traffic per day.

Summary

Everything is looking pretty good right now, thanks mostly to the ongoing improvements in lemmy making things run more smoothly. In the next week or so I'll be upgrading the storage on the VPS to cater for the ever increasing database. Admins are already discussing ways we can manage the increasing storage requirements, I'm optimistic we'll have good solutions before we hit a VPS scaling limit.

As always, happy to answer any questions.

About to upgrade aussie.zone to lemmy 0.18.1-rc.10 and lemmy-ui to 0.18.1-rc.10. This includes some substantial database improvements.

I'll unpin this once down... expect 2-3 minutes downtime.

As reported to the lemmy devs here there is no sanity checking of links in posts currently in lemmy. Please be careful in the links you click!

Further discussion and context from the reporter here.

If you'd like to volunteer as a moderator for this community, please reply here saying so.

If you'd like to volunteer as a moderator for this community, please reply here saying so.

If you'd like to volunteer as a moderator for this community, please reply here saying so.

If you'd like to volunteer as a moderator for this community, please reply here saying so.

If you'd like to volunteer as a moderator for this community, please reply here saying so.

If you'd like to volunteer as a moderator for this community, please reply here saying so.

Why turned the rain back on? After such a nice weekend... its back to winter ☹