Grafana Alerting vs Prometheus Alertmanager: Where Homelab Alerts Should Live

A homelab with Prometheus and Grafana has two places where alerts can live, and most people do not realise it until they have half their rules in one and half in the other. Prometheus evaluates alert rules from files and hands them to Alertmanager, which decides who gets told and how. Grafana has its own complete alerting system, with rules built in the web interface, its own contact points and its own silences. Both work. Running both by accident is how you end up with duplicate notifications, silences that do not silence anything, and a 3 a.m. page you cannot find the source of.

I treat every green dashboard as a claim to verify, and alerting is where that matters most: an alert that never fires looks exactly like a system that never breaks. Here is how the two approaches differ, and how to choose one home for your alerts.

How each one works

Prometheus and Alertmanager. Alert rules are written in PromQL in YAML files that Prometheus loads. Prometheus evaluates them on a schedule and sends any that fire to Alertmanager. Alertmanager is the dispatcher: it groups related alerts into one notification, suppresses alerts that are caused by another (inhibition), routes them to the right receiver based on labels, and handles silences. Its configuration is also a YAML file. Everything is text, versionable and reproducible.

Grafana Alerting. Rules are created in Grafana’s web interface, or provisioned from files, and can query any data source Grafana knows: Prometheus, Loki logs, InfluxDB, a SQL database, and more. Notifications go to contact points such as email, Slack, Discord, Telegram, Pushover or a webhook, chosen by notification policies that match on labels, much like Alertmanager’s routing. Silences and mute timings are managed in the same interface. Under the hood, Grafana runs its own built-in Alertmanager, and it can also send alerts to an external one.

The case for Alertmanager

Alertmanager is the older, simpler and more battle-tested of the two, and its strengths matter in a homelab more than people expect.

It does not depend on Grafana. If Grafana crashes, its database is corrupted, or an upgrade goes wrong, Prometheus keeps evaluating rules and Alertmanager keeps sending notifications. Your alerting survives your dashboards. With Grafana Alerting, a broken Grafana means no alerts, and a broken Grafana is exactly the kind of thing you would want to be alerted about.

It is all in text. Alert rules and routing sit in files next to your Compose definitions. You can review changes, see history in Git, and rebuild the whole monitoring setup from scratch. Many community rule sets, for node_exporter, containers and common services, are published in this format and can be dropped in.

Inhibition works well. If the switch is down, you do not need fifteen alerts about every device behind it. Alertmanager’s inhibition rules suppress the consequences when the cause is already alerting. Grafana Alerting has grouping and routing, but this kind of suppression has historically been more limited.

The cost is the editing experience. Writing PromQL in YAML without a preview means testing thresholds by trial and error, and routing trees are easy to get subtly wrong. Tools like amtool help, but it is a text-file workflow through and through.

The case for Grafana Alerting

Grafana Alerting’s biggest advantage is reach. Because rules can query any data source, you can alert on things Prometheus never sees: a log pattern in Loki, such as a backup container printing “failed”; a reading from Home Assistant stored in InfluxDB; a row count in a database. In a homelab with several data sources, one alerting system covering all of them is genuinely simpler.

The editing experience is also much friendlier. You build a query, see the graph, set a threshold, and preview which series would fire before saving. For anyone who does not live in PromQL, that alone saves hours. Mute timings, such as “no non-critical alerts between 23:00 and 07:00”, are easy to set up and easy to understand.

And if you want text files after all, Grafana supports provisioning alert rules, contact points and policies from files, so you can keep them in Git. It takes more setup than Prometheus rule files, but it is possible.

If you already use Loki for logs, Grafana Alerting is the natural way to alert on them. Choosing between a live log viewer and a searchable log store is covered in Dozzle vs Grafana Loki.

The mistake to avoid: both, half-configured

The most common homelab setup is not a choice at all. Someone imports a community dashboard with alert rules, someone else follows a Prometheus tutorial that includes Alertmanager, and now alerts come from both places. The symptoms are familiar:

• The same problem produces two notifications, with different wording, from different systems.
• You silence an alert in one place and it keeps arriving from the other.
• When something fires at night, you are not sure where the rule is defined, so you cannot tune it.

Pick one place where rules live, and move everything there. If you choose Grafana, delete or disable the Prometheus rule files. If you choose Alertmanager, remove Grafana-managed rules and keep Grafana for dashboards only. A clear rule of thumb: one source of truth for rules, one place to silence them.

Who watches the watcher?

Whichever you choose, the alerting pipeline itself can fail silently: Prometheus stops, Alertmanager loses its configuration, Grafana’s container restarts in a loop, or the notification service’s token expires. No alerts arrive, and silence looks exactly like health.

The standard defence is a dead man’s switch: a rule that is always firing, often called “Watchdog”, routed to an external service that expects to hear from it regularly and alerts you when it stops. Healthchecks.io and similar services do exactly this; they are built for checks that must keep arriving, as covered in Healthchecks.io vs Cronitor. It takes ten minutes to set up and closes the most dangerous gap in any home monitoring system.

Also send alerts somewhere you will notice at the right times. A push service on your phone, such as ntfy or Pushover, works well for urgent alerts; email suits warnings that can wait. Route by severity, so that a disk at 80% does not wake you, but a UPS on battery does.

Alerts worth having in a homelab

Whichever system you choose, a handful of rules catch most real problems:

• A host or critical container is down for more than a few minutes.
• A disk is predicted to fill within a day, rather than “over 90%”, which fires too late on big disks and too early on small ones.
• A container is restarting repeatedly.
• Backups have not succeeded in their expected window.
• The UPS is on battery, or its battery is low.
• Certificates expire within two weeks.
• The Watchdog rule, sent to an external dead man’s switch.

Resist the urge to alert on everything. Every alert you ignore teaches you to ignore the next one. If a rule fires and you never act on it, delete it or turn it into a dashboard panel instead.

Prove an alert reaches you

A rule that has never fired is a rule you have not tested. After setting up either system, break something on purpose and watch the notification arrive:

• Test the notification path. Grafana’s contact points have a test button that sends a sample notification. With Alertmanager, the amtool command can send a test alert through your routing tree.
• Test a real rule. Stop a non-critical container and confirm the “container down” alert fires, reaches your phone, and resolves when you start it again.
• Test the silence. Silence that alert, repeat the test, and confirm nothing arrives. Then remove the silence.
• Repeat a few times a year, and after any upgrade of Prometheus, Alertmanager or Grafana. Tokens expire and configurations drift.

Ten minutes of deliberate breakage is the only way to know the system you built will speak up when it matters.

Picking one home for your alerts

If your metrics live in Prometheus, your homelab is managed as code, and you want alerting that keeps working even when Grafana does not, keep your rules in Prometheus and let Alertmanager route them. Use Grafana purely for dashboards. This is the more robust setup, and the one most community rules assume. If you are deciding between Prometheus and another collector, Zabbix vs Prometheus covers that earlier choice.

If you alert on more than Prometheus metrics, such as logs, Home Assistant data or databases, or you would rather build and tune rules visually, use Grafana Alerting as your single source of truth, and protect it with an external dead man’s switch, since it depends on Grafana staying healthy.

Either way, the decision that matters most is having one place for rules, one place to silence them, and one check that tells you when the whole thing has gone quiet.