Alerts and monitoring

Salvo warns when something needs attention, on three channels: e-mail, webhook and Zabbix. All three use the same rules as the panel's Alerts screen.

When Salvo warns

Level	Situation
error	a job finished with an item without a new backup
error	an account covered by a job has no new backup beyond the limit
error	a destination with less than 5% free
error	a job with a schedule that did not run on time
warning	a destination with less than 10% free, or filling up in less than 7 days
warning	an interrupted run

A problem is announced when it starts, when it is solved, and once a day while it lasts, in a single message.

E-mail

/var/lib/salvo/salvo alert-channel --type email --to ops@example.com --levels erro,aviso

Without --smtp, the message goes through the server's own mail server (sendmail). With an SMTP server:

/var/lib/salvo/salvo alert-channel --type email --to ops@example.com \
  --smtp smtp.example.com:587 --user alerts@example.com --password-from /root/smtp-password

The password comes from a file, never from an argument (arguments show in ps).

Webhook

/var/lib/salvo/salvo alert-channel --type webhook --url https://hooks.example.com/salvo --secret-from /root/webhook-secret --levels erro

Salvo sends a JSON POST. With a secret, the request carries the signature in the X-Salvo-Assinatura: sha256=<HMAC> header (HMAC-SHA256 of the body), so the receiver can check it came from Salvo.

Zabbix

Salvo keeps <state>/zabbix.json up to date (permission 0644, no secrets). The Zabbix agent reads it with a UserParameter:

zabbix_agentd.d/salvo.conf

UserParameter=salvo.estado,cat /STATE/zabbix.json

Field	Meaning
atualizado, atualizado_unix	when the file was written; alert if it stops changing (runner stopped)
problemas, erros, avisos	current problems, by level
contas_sem_backup	accounts without a recent backup
jobs_com_falha	jobs with a failure
jobs_atrasados	jobs that did not run on schedule
destinos_com_pouco_espaco	destinations short on space
titulos	the titles of the problems, separated by ;

Test and check

/var/lib/salvo/salvo alert-channel --list   # the channels
/var/lib/salvo/salvo alert-test             # sends a test message to every channel
/var/lib/salvo/salvo alerts                 # the current problems
/var/lib/salvo/salvo zabbix                 # shows zabbix.json