Custom-written DNS servers: Learn to love them

A cryptic e-mail message from one of my readers telling me that blog.ioshints.info does not resolve into an IP address (yet again, thank you Igor!) set just the right mood for my Monday morning. As I had about 30 minutes before rushing off to a day-long countryside family event, all I could do was run a few quick tests (of course it worked for me) and take my iPad with me (luckily, it’s a 3G model and the coverage in Slovenia is decent).

During the day, Igor was able to supply more details: the DNS hosting company I was using suddenly decided to return NXDOMAIN error code on every CNAME record unless the query type was CNAME, while providing the results in the response at the same time. Here’s a sample of their bizarre reasoning:

$ dig @ns1.domaindiscover.com xml.ioshints.info
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 14206
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;xml.ioshints.info.             IN      A

;; ANSWER SECTION:
xml.ioshints.info.      0       IN      CNAME   ghs.google.com.

Correct answer is obviously somewhere along the lines of the following printout (I will not waste my time reading the RFCs to figure out all the potential correct answers ;). The authoritative name server supplies just the CNAME, not the final answer (it has no authority to do it) and obviously has to return the success code (as it found the name we were looking for).

$ dig @ns.nil.si blog.ioshints.info
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59148
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 12
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;blog.ioshints.info.            IN      A

;; ANSWER SECTION:
blog.ioshints.info.     0       IN      CNAME   ghs.google.com.

Be liberal in what you accept

The worst part of the puzzle was that I was able to reach the blog throughout the day (ensuring the DNS TTL of the CNAME record in the server I was using did expire) even using different Service Providers (and DNS servers). In the afternoon, as I was finally able to get hold of a reasonable PC (for whatever reason, I don’t think iPad is the proper platform to write longer texts) I tried to figure out the difference between Igor and myself and polled my Twitter friends.

Only a few of them had problems (weird) and Ethan was kind enough to help me with my troubleshooting efforts. It looks like most recursive DNS servers (including Google’s 8.8.8.8 and OpenDNS) don’t bark at NXDOMAIN error if the answer is included in the reply and happily continue the resolution process, while a few servers decide NXDOMAIN means what it should mean and return an error to the client.

Let’s fake authority

Digging further, Igor uncovered another stupidity: the programmers working for DomainDiscover decided to make their life extremely easy and generate fake SOA records. This is what they claim the authority for google.com is:

$ dig @ns1.domaindiscover.com xml.ioshints.info

... deleted ...

;; AUTHORITY SECTION:
google.com.         3600    IN      SOA     ns1.tierra.net.
                    hostmaster.tierra.net. 2010080701 7200 1800 604800 28800

One must wonder what these guys will do when DNSSEC comes knocking on their doors. I decided I don’t want to be there when that happens and moved my DNS server today.

Conclusions

I am sad to be forced to leave DomainDiscover. They were working perfectly for the last 5+ years I was using them, but obviously the load they were handling (and the pricing wars in the domain registration business) pushed them to cut a few corners too many.

Unfortunately, we can only expect to see more failures like this one as the industry goes through endless cycles of doing more with less.

1 comment:

  1. This recently happened to me as well, because the name server thought he was authoritative for the entire internet. Its one of those configuration mistakes that only show up if you have CNAMEs pointing to other zones not hosted on the same name server.
    And to make it more annoying to troubleshoot, some DNS servers like BIND can still resolve the records while others such as the one included in windows returns NXDOMAIN.

    How does this happen? In my case the DNS server was PowerDNS with a mysql backend. Someone or some poorly written script, entered a blank line in the database, which caused PowerDNS to behave like this.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.