Monitoring Strategy
How the homelab is monitored, alerted, and observed.
Monitoring Stack Overview
| Component |
Location |
Purpose |
| Uptime Kuma |
VPS |
External health checks (HTTP, TCP, ping) |
| ntfy |
VPS |
Push notifications (alerts, backup status) |
| VictoriaMetrics (Papa) |
Docker VM |
Time-series metrics database |
| vmagent |
Docker VM |
Prometheus-compatible metrics scraper |
| Grafana (Papa) |
Docker VM |
Dashboards and visualization |
| Dozzle (Ysyry) |
Docker VM |
Real-time container log viewer |
| Glances |
NAS |
System resource monitor (HA integration) |
| Watchtower |
Docker VM |
Auto-update monitoring (Sunday 4 AM) |
Coverage Matrix
Which monitoring layer protects which host. Audited 2026-05-27.
| Host |
Host metrics¹ |
cAdvisor² |
Container up/down³ |
tailscaled self-heal⁴ |
External liveness⁵ |
| docker-vm |
✅ host binary |
✅ |
✅ 8 critical + 2 monitored |
✅ |
✅ |
| NAS |
✅ container |
❌ |
✅ 3 critical + 1 monitored |
✅ |
✅ |
| Proxmox |
✅ host binary (PR #102) |
n/a (no Docker) |
n/a (no Docker) |
✅ self-heal only⁶ |
✅ (Ping / HACS) |
| VPS |
❌ gap |
❌ |
❌ labels inert⁷ |
❌ |
✅ (it hosts Uptime Kuma) |
- Host metrics —
node_exporter scraped by vmagent → HighCpuUsage, HighMemoryUsage / HypervisorMemoryPressure, DiskSpaceLow / DiskSpaceCritical, HostDown.
- Per-container metrics — cAdvisor →
ContainerHighCpu, ContainerHighMemory, ContainerRestarting. Only runs on docker-vm.
- Container up/down —
scripts/critical-containers-watchdog.sh (cron */5), non-aging textfile metrics → CriticalContainerDown / MonitoredContainerDown. See Failure Detection & Self-Heal.
- tailscaled self-heal —
scripts/tailscaled-watchdog.sh restarts a wedged tailscaled.
- External liveness — Uptime Kuma (35 monitors) + out-of-band UptimeRobot — the only layer that survives a full host outage.
- Proxmox self-heals a stuck tailscaled but has no textfile path, so
TailscaledStuck / TailscaledWatchdogStale don't fire for it.
adguard + unbound carry com.cronova.critical=true on the VPS, but the watchdog is not deployed there — nothing reads the labels. VPS liveness is covered externally instead (see Known Gaps).
Metrics Pipeline
node_exporter (Docker VM :9100) ──┐
node_exporter (NAS :9100) ────────┤
node_exporter (Proxmox :9100) ────┤
VictoriaMetrics (:8428) ──────────┤
vmagent (:8429) ──────────────────┤
Grafana (:3000) ──────────────────┼──► vmagent ──► VictoriaMetrics ──► Grafana
vmalert (:8880) ──────────────────┤ (30s scrape) (90d retention) (papa.cronova.dev)
Alertmanager (:9093) ─────────────┤ │
cAdvisor (:8080) ─────────────────┤ vmalert ──► Alertmanager ──► ntfy
Home Assistant (/api/prometheus) ──┘ (30s eval) (group/dedup) (push)
Config: docker/fixed/docker-vm/monitoring/prometheus.yml
Scrape Targets
| Job |
Target |
Labels |
| node-docker-vm |
host.docker.internal:9100 |
instance: docker-vm |
| node-nas |
100.82.77.97:9100 |
instance: nas |
| node-proxmox |
100.78.12.241:9100 |
instance: proxmox (Tailscale IP — LAN :9100 is blocked) |
| victoriametrics |
victoriametrics:8428 |
instance: victoriametrics |
| vmagent |
vmagent:8429 |
instance: vmagent |
| grafana |
grafana:3000 |
instance: grafana |
| cadvisor |
cadvisor:8080 |
instance: docker-vm |
| vmalert |
vmalert:8880 |
instance: vmalert |
| alertmanager |
alertmanager:9093 |
instance: alertmanager |
| home-assistant |
host.docker.internal:8123/api/prometheus |
instance: home-assistant |
VictoriaMetrics
- Image:
victoriametrics/victoria-metrics:latest
- Port: 8428 (localhost only)
- Retention: 90 days
- Memory limit: 1GB
- Data volume:
vm-data
Grafana
- Image:
grafana/grafana:latest
- Port: 3000 (localhost only, behind Caddy + Authelia)
- URL:
https://papa.cronova.dev
- Plugin:
victoriametrics-metrics-datasource
- Dashboards provisioned via
grafana/provisioning/dashboards/json/
Grafana Dashboards
| Dashboard |
Grafana ID |
Source |
What it shows |
| Node Exporter Full |
1860 |
node-docker-vm, node-nas |
Host CPU, RAM, disk, network |
| VictoriaMetrics Single |
10229 |
victoriametrics |
TSDB health, ingestion rate, storage |
| cAdvisor Docker |
19792 |
cadvisor |
Per-container CPU, memory, network |
| vmagent |
12683 |
vmagent |
Scrape stats, target health, remote write |
| Grafana Internals |
3590 |
grafana |
API response times, sessions, memory |
| Homelab Overview |
— (custom) |
all targets |
Host health, containers, network, monitoring health |
Failure Detection & Self-Heal
The layer that survives multi-day outages — built out Mar–May 2026 after several silent failures (the 7-day Frigate NFS outage, the 8-day forgejo-runner death, backup sidecars dead 6 weeks). Unlike the old cAdvisor ContainerStopped rule (which used last_over_time(...[24h]) and aged out — going silent on long outages), these textfile metrics are re-emitted every run and do not age out.
Container watchdog — scripts/critical-containers-watchdog.sh
- Runs on docker-vm and NAS, cron
*/5, from /opt/homelab/bin/. Emits node_exporter textfile metrics (critical_container_up, monitored_container_up, critical_container_watchdog_last_check_timestamp_seconds).
- Two label tiers:
| Label |
Tier |
Down → |
Alert (vmalert) |
Channel |
com.cronova.critical=true |
urgent |
>5m |
CriticalContainerDown + direct host ntfy POST |
cronova-critical |
com.cronova.monitored=true |
warning |
>30m |
MonitoredContainerDown (metric path only) |
cronova-warning |
- The direct ntfy POST is pinned DNS-independent (public-IP-first, tailnet fallback via
curl --resolve) so it delivers even when tailscaled / DNS is wedged.
CriticalContainerWatchdogStale (heartbeat >30m) catches the watchdog itself dying — otherwise a dead watchdog is its own silent failure.- Labelled today: docker-vm — 8 critical (authelia, caddy, frigate, homeassistant, immich-server, mosquitto, pihole, vaultwarden) + 2 monitored (forgejo-runner, forgejo-dind); NAS — 3 critical (forgejo, restic-rest, samba) + 1 monitored (syncthing).
tailscaled self-heal — scripts/tailscaled-watchdog.sh
- Deployed via
ansible/playbooks/tailscaled-watchdog.yml; restarts tailscaled when BackendState != Running for ≥10m.
- docker-vm + NAS: full (self-heal +
tailscaled_watchdog_* metric → TailscaledStuck / TailscaledWatchdogStale). Proxmox: self-heal only (no textfile path).
autoheal sidecar
willfarrell/autoheal (maintenance stack) restarts any container labelled autoheal=true that goes unhealthy. Currently guards frigate — paired with a mount-asserting healthcheck (stat -f -c %T /media/frigate == nfs) so a lost NFS bind triggers a restart instead of binding silently to an empty mount.
Uptime Kuma Monitors
Uptime Kuma runs on the VPS and monitors all services via Tailscale mesh. 35 monitors managed via scripts/setup-uptime-kuma.py (single source of truth). Alerts route to ntfy topics by priority tier.
Critical (60s interval, ntfy urgent)
| Monitor |
Type |
Target |
| Headscale |
HTTP |
https://hs.cronova.dev/health |
| Vaultwarden |
HTTP |
https://vault.cronova.dev/alive |
| Pi-hole DNS |
TCP |
100.68.63.168:53 |
| Caddy (Docker VM) |
HTTP |
https://cronova.dev |
| OPNsense Gateway |
Ping |
192.168.0.1 |
| Uptime Kuma |
HTTP |
https://status.cronova.dev |
| ntfy |
HTTP |
https://notify.cronova.dev |
| Caddy (VPS) |
TCP |
100.77.172.46:443 |
| VPS Pi-hole |
TCP |
127.0.0.1:53 |
| cronova.dev |
HTTP |
https://cronova.dev (900s interval) |
Warning (60-300s interval, ntfy high)
| Monitor |
Type |
Target |
| Home Assistant (Jara) |
HTTP |
https://jara.cronova.dev (300s) |
| Frigate (Taguato) |
HTTP |
https://taguato.cronova.dev/api/version (60s) |
| Forgejo |
HTTP |
http://100.82.77.97:3000 (60s) |
| NAS Samba |
TCP |
100.82.77.97:445 (300s) |
| Restic REST |
Keyword |
http://100.82.77.97:8000 (keyword: "Unauthorized", expect 401) |
| Coolify (Tajy) |
HTTP |
https://tajy.cronova.dev (300s) |
| Authelia (Okẽ) |
HTTP |
https://auth.cronova.dev (300s) |
| Javya |
HTTP |
https://javya.cronova.dev (60s) |
| Javya API |
HTTP |
https://javya-api.cronova.dev/health (60s) |
| NAS |
Ping |
100.82.77.97 (60s) |
| Docker VM |
Ping |
100.68.63.168 (300s) |
| Watchtower |
Ping |
100.68.63.168 (60s) |
Info (300-900s interval, ntfy default)
| Monitor |
Type |
Target |
| Jellyfin (Yrasema) |
HTTP |
https://yrasema.cronova.dev/health |
| Grafana (Papa) |
HTTP |
https://papa.cronova.dev |
| Immich (Vera) |
HTTP |
https://vera.cronova.dev |
| Syncthing |
HTTP |
http://100.82.77.97:8384/rest/noauth/health |
| Glances |
Keyword |
http://100.82.77.97:61208/api/4/cpu (keyword: "total") |
| Pi-hole Fixed |
TCP |
100.68.63.168:53 (300s) |
| DNS - cronova.dev |
DNS |
cronova.dev via 1.1.1.1 |
| Beryl AX |
Ping |
100.102.244.131 (120s, may be offline) |
| Beryl AX - Admin |
TCP |
100.102.244.131:80 (120s) |
| hermosilla.me |
HTTP |
https://hermosilla.me/ (900s) |
|
HTTP |
` (900s) |
ntfy Notification Architecture
URL: https://notify.cronova.dev (VPS, Caddy reverse proxy)
Topics
| Topic |
Purpose |
Priority |
cronova-critical |
Service down, data loss risk |
Urgent (wakes phone) |
cronova-warning |
Degraded performance |
High |
cronova-info |
Backups completed, maintenance |
Default (silent) |
cronova-test |
Testing notifications |
Low |
Auth
- Anonymous access: deny-all
- Service tokens for automation (backup sidecars, scripts)
- User
augusto has full read/write on all topics
Integration Points
| Source |
Topic |
Trigger |
| Uptime Kuma |
cronova-critical / cronova-warning |
Service down/degraded |
| Backup sidecars |
cronova-critical / cronova-info |
Backup failure / success |
scripts/backup-notify.sh |
Per-service routing |
Backup event notifications |
Subscribe on Phone
Android/iOS ntfy app → Subscribe to:
https://notify.cronova.dev/cronova-critical
https://notify.cronova.dev/cronova-warning
Container Log Monitoring — Dozzle (Ysyry)
- URL:
https://ysyry.cronova.dev (Caddy + Authelia)
- Real-time Docker log viewer for all containers on Docker VM
- No persistent storage — live view only
- Useful for debugging container startup issues, watching Frigate detections, checking backup logs
Auto-Update Monitoring — Watchtower
- Schedule: Sunday 4:00 AM (label-enabled, opt-in via
com.centurylinklabs.watchtower.enable=true)
- Image:
nicholas-fedor/watchtower:1.14.2 (maintained fork — official containrrr is abandoned/Docker 29+ incompatible)
- Behavior: Rolling restarts, old image cleanup
- Excluded from auto-update (manual only): vaultwarden, frigate, headscale
| Integration |
Source |
What It Monitors |
| System Monitor |
Docker VM |
CPU, RAM, disk usage |
| Glances |
NAS (100.82.77.97:61208) |
NAS system metrics |
| Proxmox VE (HACS) |
Oga (100.78.12.241) |
Host and VM status |
| Frigate |
MQTT (mqtt-net) |
Camera events, detection counts |
Monitoring Checklist
Weekly
- [ ] Check Uptime Kuma dashboard — all monitors green
- [ ] Review ntfy alert history — any unexpected alerts
- [ ] Spot-check Dozzle for container error logs
Monthly (1st Sunday)
- [ ] Spot-check NAS restic repos for snapshot freshness (until the alerting plan —
docs/plans/backup-success-alerting-2026-04-22.md — lands and this becomes automatic)
- [ ] Check Grafana dashboards — disk usage trends, RAM pressure
- [ ] Verify vmagent scrape targets are all up (
/targets endpoint)
- [ ] Review Watchtower update logs
- [ ] Check NAS Purple 2TB usage (~94%, Frigate retention — monitor closely; task #33)
Quarterly
- [ ] Manual backup restore drill per
docs/guides/backup-test-procedure.md (pending Task #18 to automate)
- [ ] Review and update Uptime Kuma monitors for new/removed services
- [ ] Test ntfy notification delivery (all priority levels)
- [ ] Review VictoriaMetrics retention and disk usage
Known Gaps & Limitations
Surfaced by the 2026-05-27 coverage audit. None are active outages — they are blind spots to close deliberately.
- VPS host resource metrics — no node_exporter. The VPS (1 GB Vultr) is not a vmagent scrape target, so there are no
HighCpuUsage / HighMemoryUsage / DiskSpaceLow / HostDown alerts for it. Service liveness is covered (Uptime Kuma + UptimeRobot), but slow resource exhaustion (disk filling, memory pressure on a constrained box) would go unnoticed until a service actually fails. Remediation: add vultr to node_exporter_hosts, run node-exporter.yml, add a node-vps scrape target — but keep :9100 tailnet-only (the VPS is internet-facing; verify UFW drops it on the public interface, mirroring the port-53 posture the external-dns-probe guards).
- VPS critical labels are inert.
adguard + unbound carry com.cronova.critical=true but no watchdog runs on the VPS, so the labels do nothing. Today this is redundant (Uptime Kuma covers their liveness externally), and headscale + ntfy — the tailnet coordinator and the alert sink — aren't labelled at all. Closing gap #1 enables a VPS watchdog that would make these meaningful.
- ntfy delivery is self-referential. ntfy runs on the VPS; Uptime Kuma (its nearest watcher) runs on the same VPS. A full VPS outage takes both down together, so ntfy death is only detectable by the out-of-band UptimeRobot probe. Alertmanager has no meta-alert for "cannot reach my receiver." Keep the UptimeRobot check on
notify.cronova.dev as the backstop.
- No per-container metrics off docker-vm. cAdvisor only runs on docker-vm, so
ContainerHighCpu / ContainerHighMemory / ContainerRestarting don't fire for NAS or VPS containers. The watchdog covers up/down there; resource-pressure on those containers is not alerted. Lower priority.
References