\documentclass[conference, a4paper, twoside]{IEEEtran} \usepackage{url} \usepackage{graphicx} \usepackage{verbatim} \usepackage[latin1]{inputenc} \newenvironment{code}{\footnotesize\verbatim}{\endverbatim\normalsize} \begin{document} \title{Monitoring DNSSEC zones: what, how and when?} \author{% \IEEEauthorblockN{Stéphane Bortzmeyer} \IEEEauthorblockA{AFNIC\\ Immeuble International\\ \\78181 Saint-Quentin-en-Yvelines\\ France\\ \texttt{bortzmeyer@nic.fr}} } %end /author \maketitle \begin{abstract} Unlike the traditional DNS, where zones, once properly configured, stay OK forever, DNSSEC introduces much more possibilities of failure. A zone can be perfect at one time and, without human intervention, stop working later. Permanent and automatic monitoring is therefore a necessary part of DNSSEC operations. But why do we still see DNSSEC problems such as signature expirations or missing keys and why are they often detected by a post on public mailing lists rather than by monitoring? As of today, only a minority of top-level domains operators can say they never had a DNSSEC validation failure. Because monitoring is not perfect today, we need to think about how to monitor and what to monitor. Among various issues, there is a pressing need for a time-aware monitoring, which keeps track of the situation of the past, to be able to deduce the possible current state of DNS caches. We developed a new tool to monitor the timing of key rollover events and ran it against several domains, showing that timing errors are not an exception. \end{abstract} \section{Introduction} The Domain Name System (DNS) allows clients, the resolvers, to query servers about resource records of various types\footnote{The two types most often queried are A -~IPv4 addresses~- and MX -~mail servers for a domain.} Records for a same name and of a same type make a resource record set and some operations (such as signature, see later) handle a set as atomic. DNS relies heavily on caching, for performance, and the time during which you may cache is indicated by the server, as a Time To Live (TTL). Because DNS has several vulnerabilities, allowing to insert fake data (\cite{rfc3833} and \cite{rfc5452}), the DNS SEcurity Extensions (DNSSEC) have been developed to allow cryptographic signing of resources record sets (\cite{rfc4033}, \cite{rfc4034} and \cite{rfc4035}). This makes modification of data easily detectable. DNSSEC use assymetric cryptography and the public keys are published in the DNS itself. Signatures are also published as resource records, with a normal TTL. They also have an expiration date, to prevent replay. Good cryptographic practices suggest to replace the keys from time to time\footnote{Although it is not clear that the risks and problems associated with this practice are worth it. But everyone does it.} Such key rollovers are quite delicate since old keys and signatures staying in the caches at the resolvers may easily prevent proper validation of data. Hence the importance of timing in key rollovers (\cite{rfc4641}) and the existence of several tools\footnote{Such as OpenDNSSEC \url{http://www.opendnssec.org/}.} to automate this process. % TODO add a nice time graph ilustrating a rollover As a result, breakage of domains by DNSSEC validation errors happened in many signed TLD and other domains. See for instance the use of wrong keys in .UK in september 2010\footnote{\url{http://blog.nominet.org.uk/tech/2010/09/24/dnssec-incident-report/}}, the expiration of signatures in .GI in january 2011\footnote{\url{http://dnssec-deployment.org/pipermail/dnssec-deployment/2011-January/004719.html}} or the use of an invalid signature for a NSEC3 record in .FR in february 2011\footnote{\url{https://www.dns-oarc.net/files/workshop-201103/DNSSEC_Key_Deletion_Issue-Vincent_Levigneron-afnic.pdf}}. \section{Monitoring} Network and systems monitoring is typically done today for basic services (host up/down, DNS servers replies/does not reply) but not always for more complicated things such as signature expiration. Until recently, ``simple'' failures like a signature expiration were still able to sneak under the radar and be undetected until an embarassing message appears on public mailing lists, such as OARC\footnote{DNS Operations, Analysis and Research Center \url{https://www.dns-oarc.net/}}'s \url{dns-operations}, asking ``Does anyone know what is going on with .EXAMPLE?''. Of course, we can hope that these ``simple'' failures will be less and less common with time, when tools get better and system administrators more aware of the issue. But we cannot be confident that this category of bugs has completely disappeared. So, what are today the best practices in DNSSEC monitoring? Most DNS operators use a tool which perform automatically various tests at regular intervals, and raise an alarm if a test has failed repeatedly. The best known of these tools is Nagios\footnote{\url{http://www.nagios.org/}} but there are many others. All these tools come with predefined tests but DNSSEC testing is not yet part of this set. In that case, the system administrator installs plugins, which perform the tests, and the tool just acts as a scheduler, running the test at specified times, and triggering alarms if needed. The Nagios API for these plugins is widely used and is recognized by many programs. Since no monitoring tool comes with ``out of the box'' DNSSEC tests, many plugins were developed and are available on the network such as \url{http://dns.measurement-factory.com/tools/nagios-plugins/check_zone_rrsig_expiration.html}, \url{http://dave.knig.ht/check_full_zone_rrsig_expiration/}, \url{https://dnssec.surfnet.nl/?p=562}, \url{http://svn.durchmesser.ch/trac/check_dnssec/}, etc. What are the specific DNSSEC tests to perform? The most obvious is that the authoritative name servers all serve the DNSKEY and transmit signed data. As far as we know, no TLD ever went the DNSSEC way with a missing name server but anecdotal evidence indicates that it happened with lower zones. After that, of course, the first test which comes to mind is signature expiration which is still, at the date of the paper, the most common failure. But testing that the authoritative name servers does not serve expired signatures, while a first step, is not sufficient: when the signature is expired, it is already too late! Even if some resolvers (for instance Unbound) accept signatures for some time after the expiration, you cannot rely on it, specially since caching means that the expired signatures may stay for some time in the resolvers. You therefore need to be proactive, something that most of the above tools do, by testing that a signature is ``about'' to expire (``about'' being configurable). In the spirit of ``you can debug DNSSEC with only dig and date'', the Unix shell script at \url{http://www.bortzmeyer.org/files/check-sig.sh} illustrates this method: it adds a given amount of days to the current time, retrieves the signatures, and test if their expiration time is below the above time. If not, it warns that the signature is too close from expiration. Such a very simple tool can easily be integrated into a monitoring solution. Other common tests are general consistency (that the key used for signing is in the keyset, that the signatures are valid, etc), and, for some tools, that the chain of trust from a given trust anchor (such as the key of the DNS root) can be followed. These tools are developed for the specific purpose of monitoring a DNSSEC server. But the simplest of all tests is to set up a validating resolver and to test your zones through it. Validating resolvers, by default, returns an error code (SERVFAIL, for Server Failure) is validation fails. You then just have to test for this code. To be sure that the trust anchor has not been deleted from the configuration, you can even test that the reply comes with the AD (Authentic Data) bit set, indicating a positive validation\footnote{With the NSEC3 and opt-out option, it does not work for negative answers, such as ``this domain does not exist'', for which the AD bit is not set.}. The strength of this method is its simplicity: no programming, except a very small wrapper which sends the request and test the result. While it is a very useful method (and we use it to monitor continously the .FR name servers, using the Unbound resolver), its main weakness is that the result depend on the state of the resolver's cache. For instance, if the key used for the signatures is no longer in the authoritative name servers but is still in the cache, thanks to its TTL, this test will wrongly report a success. Addressing this weakness is covered in the next section. Some tools integrate many different tests and therefore try to offer a comprehensive testing of the DNSSEC setup. They often provide a Web interface and sometimes a command-line interface, suitable for automatic periodic testing. The best known tools today are: \begin{itemize} \item IIS' DNScheck \url{http://dnscheck.iis.se/} \item Verisign's DNSSEC debugger \url{http://dnssec-debugger.verisignlabs.com/} \item Sandia Laboratories' DNSviz \url{http://dnsviz.net/} \item AFNIC's Zonecheck \url{http://www.zonecheck.fr/} \end{itemize} Some can be used for monitoring (such as Zonecheck at AFNIC which is run periodically against .FR). \section{Time-aware monitoring} There is apparently today no tool that can check that a DNSSEC zone is always consistent. By ``consistent'', we mean: \begin{enumerate} \item every RRSIG still in the caches of the resolvers is made by a key which is currently in the zone, \item and that the key used for the current signatures is in all the key sets still cached. \end{enumerate} As a result, it is quite possible that some DNSSEC zones have sometimes ``windows of vulnerability'' where, for instance, a cached RRSIG has no key in the authoritative servers, but these windows are not detected because there are very few validating caches: if they are all lucky, nobody will notice the problem. .FR had the problem once\footnote{\url{http://operations.afnic.fr/en/2010/11/22/dnssec-validating-resolution-problem.html}}. It was detected because it lasted too long but a similar problem of a short duration could have been unnoticed. To see if these problems occur, we need a new tool. Such a tool cannot be a one-shot testing tool like are most of the DNSSEC Nagios plugins discussed in the previous section. It needs to have a memory, to remember the signatures that were present before, as long as their TTL is not over. We developed such a tool. Its algorithm is: \begin{enumerate} \item Runs the tool periodically (a good interval is a small fraction of the TTL of RRSIG and DNSKEY RRsets). \item When the tool runs for the first time, it stores in a database various RRSIGs, the keys they use and the TTL. Same thing with the DNSKEY record. \item Each time the tool runs, it: \begin{itemize} \item checks that, for every RRSIG still possibly in the caches, the authoritative name servers \emph{currently have} the key (or the complete key chain, if we want to test from the DS in the parent), \item checks that, for every DNSKEY still possibly in the caches, the \emph{current} signatures use only keys in this set. \item updates the database \end{itemize} \end{enumerate} What resource record type to use to get the signatures? For a general-use tool, SOA, NS and DNSKEY are the only one we are sure they exist. But signatures on DNSKEY are special (they are often done with two keys or, of only one, by the KSK, not the ZSK). Currently, we record signatures on both the SOA and the DNSKEY but, for the analysis, we use only the signatures on the SOA. It may not be sufficient (for instance, if every resource record set is signed independently, some signatures may be right for one type but not for the other) but there is little choice. If you accept to modify the code for specific domains, you can use a record which is present in these domains\footnote{For instance there is often a TXT record at the apex, with various human-readable information, to help debugging.} It would be interesting too to have the signatures of the ``non-existing domain or data'' records, NSEC or NSEC3. Again, there is no perfectly sure way to get them for any domain: you cannot query them directly so you have to query a domain that you believe does not exist (for instance by using a long random name\footnote{It will not work if the zone contain wildcards.}) or a type which is so uncommon that it probably does not exist. It is not perfect but it would extend the monitoring to these NSEC* records. The security of a delegation, with DNSSEC, is done with a DS\footnote{``Delegation Signer'', a cryptographic hash of the public key of the child zone} resource record in the parent zone. This DS is signed by the parent. The same issues may occur when replacing it with a new one and it would be interesting to monitor it as well. One may wonder if we should use the TTL or the signature expiration date in the calculations. \cite{rfc4033}, sections 6 and 8.1 say the resolver must use the first value (either the expiration of the TTL or the expiration of the signature). In practice, signature durations are almost always much larger than TTL so the current version of the tool takes only TTL into account. The current tool was developed in Python\footnote{http://www.python.org/}, using the excellent dnspython package\footnote{http://www.dnspython.org/} to retrieve and parse DNS resource records. The signatures and keys are stored in a SQLite\footnote{http://www.sqlite.org/} database. In the database schema, there is one table for the signatures and two for the keys, one for the keys themselves and one for the key sets. That is because some information is attached to the key, such as the cryptographic algorithm which was used, and some is attached to the key set, such as the TTL. The caching works per set, not per record, hence the need to store the set itself. Two instances of the program are running, one for monitoring of the keys (addition and deletion), one as a Nagios plugin, to be used from a monitoring system (most of existing monitoring tools can use plugins written for Nagios). Various utility programs allow to display things such as the history of keysets for a given zone: \begin{code} % ./keyset-zone.py cz #1 of cz.: ['34702', '14568*'] (first 2011-01-31 21:23:50Z, last 2011-03-08 07:16:47Z) #2 of cz.: ['34702', '14568*', '400'] (first 2011-03-08 08:14:50Z, last 2011-03-15 16:14:57Z) #3 of cz.: ['14568*', '400'] (first 2011-03-15 17:16:28Z, last 2011-03-22 14:16:44Z) \end{code} You can see above a key rollover in two steps. On March 8th, the ZSK 400 is introduced, on March 15th, the ZSK 34702 is deleted. Since it relies on a SQL database, you can also explore the tables with regular SQL commands, here for the signatures: \begin{code} sqlite> SELECT first_seen,last_seen,ttl FROM Signatures WHERE type=6 AND name='192.in-addr.arpa.' AND key_tag=20918 ORDER BY last_seen DESC; 2011-03-28 17:29:30|2011-03-28 20:17:31|86400 2011-03-28 13:22:23|2011-03-28 16:25:05|86400 2011-03-28 09:19:59|2011-03-28 12:28:09|86400 \end{code} And here for the keysets, then for the keys: \begin{code} sqlite> SELECT first_seen,last_seen,ttl,id FROM Keysets WHERE name='192.in-addr.arpa.' ORDER BY last_seen DESC; 2011-03-29 09:38:45|2011-03-31 08:30:30|14400|J/dCsFib6kxRer/O/eh1ZbI/Un8= 2011-03-21 21:39:09|2011-03-29 08:38:16|14400|NgM4JKT7QacTgX+ZF7bNo2owKjQ= sqlite> SELECT first_seen,last_seen,key_tag FROM Keys WHERE name='192.in-addr.arpa.' ORDER BY last_seen DESC; 2011-03-01 15:34:17|2011-03-31 08:30:30|39318 2011-03-21 21:39:09|2011-03-31 08:30:30|60494 2011-03-01 15:34:17|2011-03-29 08:38:16|20918 \end{code} The tool is published under a free software licence\footnote{BSD three-clause, allowing pretty much every use.} at \url{https://github.com/bortzmeyer/key-checker}. \section{Results} The program started to run at the end of January 2011. 54 domains are monitored, both top-level domains and ``important'' domains at the second or third level. As suspected, several of them experienced rollover problems, which apparently have not been detected by other means. It seems to indicate that the tools currently use to manage the keys are far from perfect. Here are examples of the two possible problems, new keys used too soon and old keys deleted too soon, with the guinea pig domain \url{office--enregistrement.fr}. Here are the keysets before and after a careless key rollover: \begin{code} % ./keyset-zone.py office--enregistrement.fr ... #3 of office--enregistrement.fr.: ['15856*', '1611', '56550', '36216'] (first 2011-02-08 11:33:00Z, last 2011-02-27 14:28:42Z) #4 of office--enregistrement.fr.: ['15856*', '37155'] (first 2011-02-27 15:22:41Z, last 2011-03-22 13:21:35Z) \end{code} The KSK 15856 stayed but the entire set of three ZSK was replaced by a new ZSK, without concern for the proper key rollover timing. And here is the analysis: \begin{code} % ./examine-history.py office--enregistrement.fr ERROR: signature of zone office--enregistrement.fr. last seen at 2011-02-27 14:28:42 (with a TTL of 3600) while the key 1611 was retired at 2011-02-27 15:22:41 ... \end{code} Here, the signatures were made by the key 1611 and the key was deliberately withdrawn fom the DNSKEY set\footnote{The monitor program runs once every hour so the times are approximate.} without waiting for the signatures and the caches to expire. Second example: \begin{code} % ./examine-history.py office--enregistrement.fr ... ERROR: signature of zone office--enregistrement.fr. first seen at 2011-02-27 15:22:41 while the last keyset before key 37155 was last seen at 2011-02-27 14:28:42 and its TTL was 3600 \end{code} Here, in the same rollover, the new key was immediately used for signatures, while the former key set was still possibly in some caches. Do this sort of problem actually occur in the wild? Unfortunately, yes. See for instance this rollover: \begin{code} % ./examine-history.py noaa.gov ERROR: signature of zone noaa.gov. last seen at 2011-02-18 11:17:47 (with a TTL of 86400) while the key 23826 was retired at 2011-02-18 21:26:20 \end{code} Because of the TTL, the key should have been published until February 19th at noon. Because of the early retirment of key 23826, unvalidatable signatures have been available in caches for up to 14 hours. In the same rollover, another problem occured: \begin{code} ERROR: signature of zone noaa.gov. first seen at 2011-02-18 21:26:20 while the last keyset before key 52668 was last seen at 2011-02-18 20:27:06 and its TTL was 86400 \end{code} The new key 52668 was immediately used while it should have been published but not used for 24 hours\footnote{Contacted, the NOAA DNS administrator diagnosed a bug in the recent version 1.6 of the program they use, rollerd \url{https://www.dnssec-tools.org/wiki/index.php/Rollerd}. Previous version had no such issues.} From the sample of tested domains, it seems that the first error (key retired while signatures were still in some caches) is the most common. Here is another example: \begin{code} % ./examine-history.py isoc.org ERROR: signature of zone isoc.org. last seen at 2011-03-01 21:31:08 (with a TTL of 86400) while the key 41414 was retired at 2011-03-02 10:30:13 \end{code} The tool was again too eager to delete the key (12 hours while the TTL was 24 hours). The problem can also happen to TLD. Here, with the ccTLD of Bulgaria: \begin{code} % ./examine-history.py bg ERROR: signature of zone bg. last seen at 2011-03-19 18:14:39 (with a TTL of 345600) while the key 2817 was retired at 2011-03-20 20:12:40 \end{code} The TTL was huge (345600 seconds) and therefore the signature could have been in the caches until March 23rd. But the key 2817 was retired before. Some cases are more complicated and more difficult to analyze. For instance, this reported problem is questionable: \begin{code} % ./examine-history.py isc.org ERROR: signature of zone isc.org. first seen at 2011-02-27 01:08:42 while the last keyset before key 21693 was last seen at 2011-02-27 00:06:07 and its TTL was 7200 \end{code} That's because \url{isc.org}\footnote{Warning: the previous example was \url{isoc.org}, this one is \url{isc.org}, just one letter is different.} uses a rare technique, double signature where every resource record is signed with two keys. The key 21693 was published on February 27th and immediately used for actual signatures. Since the TTL of the DNSKEY set is two hours, it seems to mean that the signatures may have been unvalidatable for up to two hours, if the DNSKEY RRset was already in the cache. But another key was in the old keyset, 26982, and this key was also used for signing. It means that every record had a correct signature, with 26982, and a possibly invalid one, with 21693. What should be the behaviour of a validating DNS resolver in such a case? \cite{rfc4035}, section 5.3.3 (``Checking the Signature'') says a validating resolver is free to choose its policy when there are multiple signatures so, in theory, an unlucky resolver could pick the wrong one. But this rule seems to be applicable only when the two signatures have a proper chain of trust from a trust anchor. If a resolver misses key 21693, it must tries the signatures made by the other key so the warning of our tool was probably spurious in that case. \begin{table}[!t] % increase table row spacing, adjust to taste \renewcommand{\arraystretch}{1.3} \caption{Summary of all the significative incidents detected with the first tests of this tool} \label{glitches_table} \centering\begin{tabular}{|l|c|c|c|} \hline \textbf{Zone} & \textbf{Date} & \textbf{Glitch} & \textbf{Window of vulnerability}\\ \hline \cline{1-4} weather.gov&2011-04-07&used too early&18h \\ \hline \cline{1-4} isoc.org&2011-03-29&retired too early&11h \\ \hline \cline{1-4} 192.in-addr.arpa&2011-03-28&retired too early&14h \\ \hline \cline{1-4} my&2011-03-26&retired too early&24h \\ \hline \cline{1-4} bg&2011-03-19&retired too early&72h \\ \hline \cline{1-4} isoc.org&2011-03-01&retired too early&11h \\ \hline \cline{1-4} noaa.gov&2011-02-18&used too early&24h \\ \hline \cline{1-4} noaa.gov&2011-02-18&retired too early&24h \\ \hline \end{tabular} \end{table} \section{Conclusion and future work} This short run of the new key rollover analysis tool showed that the problem is still, at the beginning of 2011, a reality. There is not yet an indication that DNSSEC problems are entirely behind us. While we can hope that, in five or ten years, these problems will be no only solved but also forgotten, except by a few old timers, DNSSEC today's situation forces us to do a lot of monitoring to catch the issues before someone else spot them. With time, new and better key management tools will be developed, and tested, but it is a long-term project. To improve the monitoring, we hope to develop of a vizualisation system of the database, interactive, able to show the various periods, allowing to zoom to see discrepancies, etc. It would make analysis easier and more spectacular. \section*{Acknowledgment} Thanks to the DNS administrators who, warned of the issue, took the time to discuss it with me. For tool development, thanks to Joe Abley. For discussions and brainstorming about the new tool, thanks to Kim-Minh Kaplan, Patrik Wallström and Duane Wessels. Presented at the SATIN conference\footnote{\url{http://conferences.npl.co.uk/satin/}} on April 4th, 2011. \bibliographystyle{alpha} \bibliography{biblio} \end{document}