this post was submitted on 08 Jul 2024
22 points (92.3% liked)

Selfhosted

38789 readers
366 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 1 year ago
MODERATORS
 

I'm curious what people are using to monitor their backups? I have Borg running on a cronjob, but checking logs periodically is getting tedious, so I'd like to automate that if possible

top 11 comments
sorted by: hot top controversial new old
[–] [email protected] 14 points 1 month ago* (last edited 1 month ago) (1 children)

I recommend a dead man's switch like Healthchecks.io, which can be selfhosted for free. Whenever you have something that's regularly occurring, add an extra callout to your unique Healthchecks callout UUID as part of the automation, and Healthchecks will send you a notification if something misses its callout schedule. You can also attach whatever data (e.g. a log) to the callout so you can look back through the run history. IIRC Borg will give you a non-zero return code if it detects problems, so you can send e.g. https://hc-ping.com/your-uuid-here/$? and a non-zero code will signal a notification as well (more examples here).

Also, Borgmatic is really easy to use for managing Borg repos. There's a lot of configuration options (including Healthchecks.io integration) but you can delete like 90% of it for normal usecases.

[–] [email protected] 0 points 1 month ago* (last edited 1 month ago)

I use healthchecks for all my scripts and making sure my docker containers and services are up. If anything goes down or errors then I get sent a telegram message immediately.

[–] [email protected] 5 points 1 month ago* (last edited 1 month ago)

If you want to do this, what you probably want is to pump your logs into a log drain, something like betterstack is good. They then allow you to set up discrepancy thresholds and can send you emails when something seems to be out of the ordinary. There's probably a self hosted thing that works the same way but I've never found a simple setup. You can do the whole Prometheus, influxdb, grafana setup but imo it's too much work, and then you still have to set up email smtp separate from that.

[–] [email protected] 4 points 1 month ago

Use unit files and add on failure a notification per mail or whatever.

You still should test the backups periodically.

[–] [email protected] 3 points 1 month ago (1 children)

I just use ’’’borg list [mypath] | tail 1’’’ in my zshrc on my desktop to see the last backup date. I guess you could make it work on server if you mail that output to yourself or something similar. Maybe add it to your cronjob?

[–] [email protected] 2 points 1 month ago

something that important I'd put into the login banner as well.

[–] [email protected] 3 points 1 month ago* (last edited 1 month ago)

I check the return code from the borg backup job and issue a notification to my phone via NTFY if there was a problem.

Please note that return code 1 (warning) will be issued if a file changes while borg is backing it up so it's very common if you backup log files for example. Which is why I only notify for code 2 or greater.

You can also do it the other way and simply issue a notification no matter what happened, and just list the return code as-is. This is probably better since you also get confirmation that the backup job is actually running.

[–] [email protected] 3 points 1 month ago

+1 for healthchecks.io, and I also use ntfy.sh so every morning I can wake up to a warm, fuzzy, “backup successful” notification.

[–] [email protected] 2 points 1 month ago

I use Borgbase as one of my backup destinations and you can set it to send an email if they haven't received a new backup for a number of days.

[–] [email protected] 2 points 1 month ago (1 children)

Are you looking for something for a headless server or something for a system with a GUI (eg a desktop)?

[–] [email protected] 2 points 1 month ago

Headless. It's all running fine in the background. For now...