Contents (4 sections)
Clear-cut failures are a luxury: everything is broken, everyone sees it, you look, you find. This one was of another kind. Our continuous integration pipeline was failing intermittently (about one build in three) when resolving our own domains. Same command, same servers: sometimes it works, sometimes no such host. And to muddy the waters, a second message sometimes appeared: connect: network is unreachable on an IPv6 address. Post-mortem of one of the most instructive failures of the year, because everything in its diagnosis is reusable.
The false culprit: always date the clues #
The IPv6 message pointed to an obvious suspect: an AAAA record pointing to an address that our build machines, on IPv4 only, could not reach. Case closed? This is where the reflex that saved the diagnosis came in: date the clue. That AAAA record had existed for two and a half months, and the CI had run without a hitch throughout that period. Something present long before the symptoms cannot be their cause; it was only diversifying the error messages (A lookup fails → “no such host”; only the AAAA answers → IPv6 attempt → “unreachable”). First lesson, a general one: faced with an intermittent symptom, establish the timeline (date of the last success, date of each change) before any hypothesis.
The method that settled it #
- Count failures per domain in the internal resolver logs: the SERVFAILs affected only two of our zones (hundreds of occurrences) and none of ten external domains tested. So the failure had a scope, and a scope is already a signature.
- Compare public resolvers: the same query succeeded 12 times out of 12 on one… and 3 times out of 12 on another, known for strict DNSSEC validation. A domain that fails only on validating resolvers is a neon arrow pointing at the chain of trust.
- Check the DS:
dig DS <zone>at the parent. Bingo: a DS record published for zones that were serving unsigned answers.
The mechanism: a broken promise #
DNSSEC works through a chain of trust: the parent zone (.com) publishes a DS record (the fingerprint of your zone’s key) that promises resolvers: “answers from this zone are signed, and here is how to check”. A validating resolver that finds this DS then requires signed answers. If it receives unsigned data (our case, during a DNS server migration window when the old and new sets coexisted in caches), it does not return the answer “anyway”: it responds SERVFAIL, by design. It is a security refusal, not a bug. Hence the perfect intermittency: our internal resolvers alternately queried a lenient resolver and a strict validating one: about one lookup in seven failed, just enough to bring down a build now and then without ever breaking anything outright.
The golden rule of DNSSEC: to enable, sign the zone, check, then publish the DS. To disable, remove the DS, wait for its TTL, and only then unsign. Never the other way round, and never a DS that survives a change of authoritative servers.
The fix, and the check that matters #
The fix came down to one action: remove the DS records published at the registrar for the zones concerned, then check, on every zone in the portfolio, that dig DS returns nothing and that the validating resolver answers 12 out of 12. One last trap along the way: your registrar’s interface or API may display a DNSSEC status (“disabled”) that proves nothing about what the parent actually publishes. Only a DS query to DNS is authoritative. The CI failures stopped the same day.
This failure ticks every box in our definition of reliability: invisible on dashboards, intermittent, and solved by method rather than by luck. That is the level of detail at which a properly managed infrastructure must be kept, DNS included, the poor relation nobody looks at until everything is on fire. A resolution mystery on your side? We like this kind of puzzle.
