# Build Resilient Home DNS with Pi-hole **Pi-hole** is the single most outage-sensitive service in most homelabs — when it goes down, every device loses name resolution at once, and it looks like the whole internet is down. This is a defense-in-depth pattern that makes it genuinely resilient: two Pi-holes behind a floating VIP, plus the parts most failover guides skip. Battle-tested in a real homelab. --- ## Why this exists Pi-hole is the worst single point of failure in a homelab. When it dies, **every** device on the network loses name resolution — no streaming, no smart home, no browsing — and the failure looks like "the whole internet is down" to everyone in the house. It's the one service where an outage is *immediately, universally noticed*, usually by someone who can't fix it. The community's answer is well-established: **two Pi-holes and a floating VIP managed by keepalived (VRRP)**. That part is not new — there are many good guides (see [Prior art](#prior-art--further-reading)). This writeup assumes you can follow one of those for the base VIP, and focuses on **the five things those guides almost universally skip** — the parts that turn a VIP demo into something you'd actually trust your household to: 1. Hand out **only** the VIP over DHCP (and *why* a "backup" DNS server makes things worse). 2. A health check that **can't false-pass**. 3. **Local self-heal** so a crashed daemon comes back on its own. 4. An **independent monitor whose alert path doesn't depend on DNS**. 5. **Testing the failure**, not the happy path. Together they form a layered system where each layer catches what the others can't. --- ## The base pattern (one paragraph) Two Pi-holes (`192.168.1.51` and `192.168.1.52`). A third, floating **VIP `192.168.1.53`** (mnemonic: `53` is the DNS port). keepalived runs on both; the higher-priority node (`.51`, MASTER) holds the VIP, and if it fails the VIP moves to `.52` (BACKUP) in ~3 s. Clients are pointed at the VIP and never talk to a real Pi-hole IP directly. That's the foundation. Everything below is what makes it hold up in the real world. ``` Clients ──DNS──▶ VIP 192.168.1.53 (DHCP hands out only .53) │ │ keepalived VRRP keeps the VIP on the healthy, higher- │ priority node; on failure it floats to the other in ~3 s. ┌──────────────────┴──────────────────┐ ▼ ▼ Pi-hole #1 192.168.1.51 Pi-hole #2 192.168.1.52 MASTER, prio 150 BACKUP, prio 100 self-heal: keepalived self-heal: keepalived Restart=on-failure Restart=on-failure Independent watcher — a third box that hosts NEITHER Pi-hole: every 60 s dig @192.168.1.53 ──(debounced)──▶ alert on a DNS-free path ``` --- ## 1. Hand out only the VIP — drop the "backup" DNS server The instinct — and what most guides show — is to hand out **two** DNS servers via DHCP: `dns1 = VIP`, `dns2 = one of the real Pi-holes`. It feels like belt-and-suspenders. **It quietly defeats the entire point of the VIP.** DNS clients do **not** reliably treat `dns2` as "only use if `dns1` times out." Many resolvers (glibc with `rotate`, systemd-resolved, plenty of IoT stacks) **round-robin or spray** across the whole list. So a meaningful fraction of your devices end up talking to a *real* Pi-hole IP directly — bypassing the VIP. When that box dies, those clients get the exact **per-query timeout stall** the VIP was built to eliminate, and they get it for the *entire* outage (a directly-named dead resolver stays dead), not just the ~3–6 s it takes the VIP to move. We measured the stall on the naive two-server setup: **~45× slower** (5 fresh lookups = 15,381 ms with a dead `dns1` vs 341 ms healthy). The fix is counter-intuitive but correct: **hand out the VIP and nothing else.** The VIP *is* the redundancy; a second address just gives clients a way to route around it. > If you're nervous about "no fallback," that anxiety is real but it points at a different gap — > see [layer 4 (the monitor)](#4-an-independent-monitor-whose-alert-path-doesnt-depend-on-dns) and > [self-heal](#3-local-self-heal). The answer is to make the VIP *reliable*, not to give clients a > way to bypass it. ## 2. A health check that can't false-pass keepalived should move the VIP based on **"is this box actually answering DNS?"**, not "is the box powered on?" That means a `vrrp_script` that queries the local resolver and only succeeds on a real answer. Everyone knows this much. The trap is in the check itself. The obvious version — ```sh out=$(dig +short pi.hole @127.0.0.1 2>/dev/null); [ -n "$out" ] # WRONG ``` — **reports healthy while the resolver is dead.** When port 53 refuses the connection, `dig +short` prints `;; communications error … connection refused` to **stdout, not stderr**, so `2>/dev/null` doesn't suppress it and the output is non-empty. The check "passes" on a corpse, the VIP never moves, and DNS stays black. The fix is to validate that the answer **is an IPv4 address**, which the error text can't satisfy: ```sh dig +short +time=1 +tries=1 pi.hole @127.0.0.1 2>/dev/null | grep -qE '^[0-9]{1,3}(\.[0-9]{1,3}){3}$' ``` We only caught this because we tested the *failure* (see [layer 5](#5-test-the-failure-not-the-happy-path)) — a naive check "passes" in the happy path forever. ## 3. Local self-heal keepalived's packaged systemd unit ships **`Restart=no`**. So if the keepalived *process* crashes, systemd leaves it dead — that box now has no VRRP, and if it happened on both boxes the VIP would go unheld and DNS would go dark, with nothing trying to recover. A one-line drop-in fixes it: ```ini # /etc/systemd/system/keepalived.service.d/override.conf (on BOTH boxes) [Service] Restart=on-failure RestartSec=5 ``` Now a crashed keepalived self-heals **locally** — no network, no coordinator, no waiting on a human. (Check your Pi-hole's `pihole-FTL` unit too; on current releases it already ships `Restart=on-failure`.) systemd's default start-limit (5 starts / 10 s) still halts a genuine *crash-loop* — e.g. a bad config a restart can't fix — which is correct: the loop stops and [the monitor](#4-an-independent-monitor-whose-alert-path-doesnt-depend-on-dns) pages you. Note there's nothing to "un-heal": a restarted daemon just re-reads its normal config, and the VIP returns to the master automatically via VRRP **preemption** (the higher-priority node reclaims it on recovery). The health-check priority penalty un-applies itself. Recovery is entirely hands-off. ## 4. An independent monitor whose alert path doesn't depend on DNS VRRP failover handles a single box dying. Self-heal handles a crash. Neither catches the tail cases — a config error that kills keepalived on **both** boxes, both nodes down, or a resolver that's "up but wrong." For those you want to be **told**. Three design rules make the monitor trustworthy, and all three are routinely gotten wrong: - **Run it somewhere that hosts neither resolver.** A monitor co-located with a Pi-hole shares that box's fate; put it on a third machine so its uptime is uncorrelated with what it watches. - **Check the symptom, not a proxy.** Ping to the VIP tells you the address is *claimed*; it does **not** tell you DNS works. Do a real `dig @VIP` and validate a real answer (same discipline as the health check). - **🔑 The alert path must not depend on DNS.** This is the one almost everyone misses. You are monitoring *DNS being down* — so every hop in the alert must work **with DNS down**. Use **raw IPs**, never hostnames. And be aware that **mobile push notifications themselves depend on DNS**: the notification server has to resolve its push relay's hostname, so in a real DNS outage the push may never send. The only channel guaranteed to work during a DNS outage is a **local** one (flash a light, sound a speaker) — device control on your own LAN needs no name resolution. Our implementation: a 60-second systemd timer on the cluster's third node runs the `dig @VIP` check, **debounces** (alert only after 3 consecutive failures ≈ 3 min, so a healthy ~6 s VIP failover never pages), and on sustained failure POSTs to a Home Assistant **webhook at a raw IP** (`http://:8123/api/webhook/…`). HA then pushes to a phone, drops an in-app notification, and logs a breadcrumb; a recovery event clears it. The webhook is a **low-power trigger token** (its only capability is to fire that one notification) rather than a full API credential on the watcher — least privilege. ## 5. Test the failure, not the happy path A health check or a self-heal that has only ever been observed *working* is untested. A check that passes when things are fine tells you nothing; you have to see it **fail on the actual fault**. So: kill the resolver and confirm the VIP moves. Kill keepalived and confirm systemd restarts it. Point the monitor at a dead IP and confirm it actually alerts (and that the alert *arrives*). Prove the broken state first, then prove the fix — otherwise you've built a smoke detector you've never held a match to. (This is how we found the stdout-false-pass bug in §2 and confirmed the debounce in §4.) --- ## The layered result Defense in depth: each layer catches a different failure class, and the ones a layer *can't* fix fall through to the next. | Failure | Caught by | Outcome | |---|---|---| | One Pi-hole box powers off / reboots | **VRRP failover** | VIP moves to survivor in ~3 s; most clients never notice (DNS cache) | | A resolver is *up but not answering* (FTL dead, box on) | **Health-checked VRRP** | Priority drops, VIP moves to a box that actually resolves | | keepalived (or FTL) process **crashes** | **Local self-heal** (`Restart=on-failure`) | Daemon back in ~5 s, locally, no human | | Bad config kills keepalived on **both** boxes; both nodes down; "up but wrong" | **Independent monitor** | You get paged (self-heal correctly gives up on a crash-loop) | | The alert itself during a DNS outage | **Raw-IP path + local channel** | Alert still fires because it never needed DNS | The failure that has *no* automatic fix — a bad config pushed to both boxes — is the one the monitor exists for, and it's the one a human should handle anyway (the fix is "revert," not "restart"). Everything above it recovers on its own. --- ## Hardening roadmap (where to take it next) Honest about the current build's edges, roughly easiest-first: - **A local, DNS-free alert channel.** Have the monitor's automation also flash a light or make a speaker announce — the one alert guaranteed to work mid-outage (see §4). *Highest value for the effort.* - **Dead-man's-switch on the monitor.** Today it's failure-triggered, so a *dead monitor* is silent. Have it check in every run to a heartbeat (e.g. a hosted dead-man service, or a "last seen" timestamp your home-automation platform alarms on if it goes stale). - **A second, independent vantage.** One watcher is itself a single point. A second checker on a different box (or your gateway) removes it. - **Active remediation (carefully).** On a sustained alert, have the watcher *try* a fix — restart keepalived + the resolver on both boxes — before escalating. Bounded (one attempt), and mind the access path/credentials. This covers "up but wrong," which a local crash-restart can't. - **Escalation ladder.** time-sensitive push → iOS *critical* alert (pierces silent/DND) → local siren → notify a second person. - **Split-brain awareness.** With unicast VRRP on a trusted LAN this is low-risk, but if the two boxes ever can't see each other they can both claim MASTER (duplicate IP). Keep them on the same L2 and consider alarming on a detected address conflict. - **Scheduled game-day.** Periodically (and automatically) kill the resolver on the *backup* and confirm recovery — proves the resilience still works months later, catching silent drift. - **Config-as-code + observability.** Keep the keepalived/self-heal/monitor configs in version control (we do), and export keepalived state / VIP holder / restart counts to a dashboard so a crash-loop is visible *before* it becomes an outage. --- ## Gotchas, condensed - `dig +short` prints connection errors to **stdout** → a naive non-empty check false-passes on a dead resolver. Validate an IPv4. - keepalived ships **`Restart=no`** — a crash is not auto-recovered until you add the drop-in. - A DHCP **`dns2` pointing at a real resolver** lets clients round-robin past the VIP and inherit the failover stall. VIP-only. - **Ping ≠ DNS.** A held VIP can front a dead resolver; check with a real query. - Your **alert path depends on DNS** unless you force raw IPs — and **mobile push depends on DNS** regardless, so add a local channel for true outage coverage. - **Focus / Do Not Disturb** silences alerts; send the outage alert **time-sensitive** (or iOS *critical*) so it breaks through. - A monitor **co-located** with a resolver shares its fate. Third box. - **Test the fault, not the happy path** — a check only ever seen passing is unverified. --- ## Prior art & further reading The keepalived + Pi-hole VIP base pattern (often paired with gravity-sync for list replication) is well documented; credit where due: - [High Availability Pi-Hole Setup using Keepalived — dchan.tech](https://dchan.tech/books/pi-hole/page/high-availability-pi-hole-setup-using-keepalived) - [How I achieved true DNS failover with multiple Pi-holes — Jerimiah Ham](https://medium.com/@jerimiahham/how-i-achieved-true-dns-failover-with-multiple-pi-holes-359b576a11ce) - [matayto/pihole-keepalived (GitHub)](https://github.com/matayto/pihole-keepalived) - [Redundant Synchronized Pi-Hole with keepalived and gravity-sync — Delightly Linux](https://delightlylinux.wordpress.com/2024/03/04/redundant-synchronized-pi-hole-with-keepalived-and-gravity-sync/) - [Pi-hole failover cluster with keepalived and unbound — Vipin PG](https://vipinpg.com/blog/building-a-pi-hole-failover-cluster-with-keepalived-and-unbound-recursive-dns-for-zero-downtime-ad-blocking) - [Pi-hole failover using Gravity Sync and Keepalived — David's Homelab](https://davidshomelab.com/pi-hole-failover-with-keepalived/) - [Redundancy for DNS: Keepalived + Gravity Sync — Adyanth Hosavalike](https://adyanth.site/posts/redundancy-keepalived-gravitysync/) - [High-Availability Pi-hole DNS with Keepalived VRRP — Homelab Journal](https://mareox.github.io/homelab-journal/tutorials/ha-pihole-keepalived-vrrp/) **What this writeup adds** to that body of work: VIP-only DHCP (and why a backup server hurts), a health check that can't false-pass, local self-heal, and an independent monitor whose alert path survives the very outage it reports — assembled as one layered system, with the reasoning and the failure-first testing that most guides skip. --- ## How it's deployed This runs on a 3-node cluster (Proxmox, in our build): the two Pi-holes are LXCs pinned to **different** nodes, so losing one node can't take out both, and the monitor runs on the third node — sharing no fate with either resolver. Keep the keepalived configs, the self-heal drop-in, and the monitor's timer/script in version control; the failure-first tests (§5) are what prove the whole thing actually recovers.