Recommended hard drive monitoring (Ubuntu server) options?
I've had a home server for years, at first using Windows Server, then Unraid, and now using Ubuntu server. I've long known that I should keep a close eye on my spinning rust, but I never really knew the best way to have that monitoring quietly automated in the background, only sending me a message when something bad shows up. If it matters at all (I assume it doesn't) I am using ZFS on Ubuntu (but not using ZFS as root. It's mounted in e.g ~/user/storage. My primary drive is an SSD)
What are you all using for hard drive monitoring? What are you using for notifications and (generally) how are you linking those two together?
So just as an FYI to those who trust these sorts of things, SMART technology is a self reporting thing. The hard drive is more than capable of lying to the data in that system if it protects the manufacture from responsibility of replacing faulty drives. Whats more, it's actually pretty rare that SMART reports and issue before the drive just sorta... dies someway or another.
It's not useless technology, but it's pretty damn close. I don't even both with any of my setups. I test it by monitoring if the server has issues reading/writing. SMART wont tell me anything before that will.
Source: Was a firmware engineer on hard drives for 10 years.
SMART value monitoring helped me finding faulty drives, not only once. And drives are tested before adding to a production system.
Certainly system drives are separate from data drives. The latter can be perfectly monitored by SMART values.
A 10-line Python script that parses the zpool status and looks for errors
A systemd one-shot service that runs it and puts the output into node_exporter
A systemd timer that runs the service every 5 minutes
A Prometheus container that among other things has an alert for changes in the zpool health data
Result
Once this is setup, it's trivial to add other checks like smartmontools, etc. If you add Alertmanager, you can get emails and other alerts sent to you while you're not looking.
I just have a shell-script running daily that runs the smartctl short test and checks certain S.M.A.R.T. values and pings healthchecks.io when successful. Easy and seems to do the job.
smartd + email to mailrise which sends notifications to a self hosted gotify instance. I then get notifications through the gotify mobile app and on the web interface.