ISP Intercepting DNS Lookups

The core function of DNS (Domain Name System) is translation of human-friendly textual computer names into the numeric IPaddresses computers use internally. It's often called "name service" rather than the geekier internal name DNS.

DNS Background Information

When there are only a few computers on an ISP connection, as is typical in most small offices and homes (SOHO), each individual computer directly contacts a DNS server somewhere in the internet. In the special case where there are tens or hundreds of computers funnelling through a single ISP connection, the computers may contact your own caching DNS server (also on your network), which in turn contacts a DNS server somewhere in the internet as necessary. Contacting "root" DNS servers directly is not only more complex and error prone, but will probably provide poorer performance.

Besides the core function of translation of names into addresses, DNS services may also provide:

• load balancing and/or geographic redirection, so although you request a single well-known computer name each request is sent to a nearby lightly loaded computer
• improved performance for just a small number of computers
• improved security/filtering
• silent fully automatic correction of typos with no delay for any additional request
• capturing and redirection of does not exist (NXDOMAIN) responses to a search service that delivers related content and/or finds a similar website and/or suggests corrections

Load balancing and/or geographic redirection is intrinsic to the way DNS works, and is provided equally by all DNS servers. Improved performance for just a small number of computers is a realistic expectation only if your "normal" DNS service (almost always provided by your ISP) is quite slow (which is seldom the case). Improved security/filtering comes from having the DNS translate a name to the correct IPaddress only for computers thought to be harmless, and for "bad" computers to instead translate their name to a bogus IPaddress. This makes the "bad" computer essentially unreachable.

This sort of security/filtering is very easy to set up and maintain, but can only provide controls at the very coarse level of whole sets of computers or entire websites. (For example, some DNS services are able to filter out all of www.geocities.com, but not just www.geocities.com/joespages.) This rudimentary type of filtering may nevertheless be sufficient, especially for public access networks such as in a school. Automatic silent correction of typos is provided by only a few DNS services (OpenDNS is the only known example as of spring 2011). Capturing and redirection of does not exist (NXDOMAIN) responses to some sort of lookup service (often with advertisements) is provided by many ISPs. As some users are highly annoyed by this (even though other users find it quite convenient), ISPs generally provide some way to (at least partially) opt out of this behavior.

DNS service is usually provided through your ISP. Currently the main providers of alternate DNS service are Google and OpenDNS. (Many other public DNS services are also available. A websearch for public DNS list will locate many documents, such as this list of public DNS servers.) Performance can be hard to measure, requiring both an accurate way to measure program execution time with high resolution, and clever procedures to avoid having the measurements invalidated by remembered responses to repetitious requests. The very misleading ping tests may suggest the Google service is significantly faster. However (in spring 2011) more meaningful testing indicates that OpenDNS is actually at least as fast (and perhaps even faster), and also that even many ISP services are just as fast. The best way to meaningfully (and easily) measure DNS speed is with a comprehensive automated program, such as this free one for Windows systems. (Note that [as of the end of 2014] this particular program will generate meaningful results only on home networks that are configured in the Per-Computer DNS style, but will generate legitimate-looking but in fact meaningless results on home networks configured in the Router-Based DNS style. If the measured performances of the DNS servers are all almost the same, suspect your network is using the Router-Based DNS configuration style and the results are meaningless.)

Currently both security/filtering and automatic typo correction are available only from the OpenDNS service but not the Google service. According to one way of looking at the situation, replacement of not found (NXDOMAIN) responses with a page containing a user-friendly message is done both by the OpenDNS service and many ISP-provided DNS services. But the two are so different it probably makes more sense to consider them separately. The message provided by the OpenDNS service aims to let the user know exactly what's going on, as well as to provide larger institutions with the opportunity to customize/brand the error response. On the other hand in this situation ISP-provided DNS services typically provide a page heavily influenced by advertisements which masquerades as some sort of intelligent response from either the local network attachment or the browser itself.

There are quite a few various ways to access these alternate DNS services: either by changing an IPaddress, or using the DNSCrypt program (only for OpenDNS as of the end of 2014), or changing a port number (in individual programs or in each computer's operating system or on the inner side of your router or on the ISP side of your router). Changing port numbers is not possible on most smartphones and internet-capable cellphones, nor on most individual computers; it's only possible with typical caching DNS servers (DNSMasq, Microsoft DNS Service, BIND, etc.). And changing the port number is only useful when accessing OpenDNS, which provides (largely undocumented) service on port 5353 as well as port 53 (Google public DNS is not available on any alternate port at all). OpenDNS now prefers the DNSCrypt method, which includes a checkbox for using port 443 rather than port 53 if necessary, and no longer publicizes the availability of their service on port 5353 (which has apparently been retained for backward compatibility). Since it's seldom a useful thing to do, changing port numbers will not be discussed in any more detail here. (As port 5353 is often used for Multicast DNS, network tracers that interpret packets may mildly misinterpret the DNS traffic that's been shifted to port 5353.)

Measuring DNS Performance

Contrary to common usage, the ping test does not provide meaningful results. It tells you how quickly an empty internet exchange with the server occurs, not how fast that server will send back an actual DNS response. (In other words ping measures only the network time, but not the processing time.)

To measure actual DNS response time (rather than ping response time), initially it seems that current Windows systems will require installation of some kind of alternative or enhancement or power tool such as this one. It turns out to also be possible to make such a measurement on a standard Windows system without any additional software at all, although the way to do this is rather geeky and difficult and not at all well known. The needed command is

cmd /v:on /c "echo !TIME! & nslookup www.nist.gov test.dns.server & echo !TIME!"

For test.dns.server substitute the IPaddress of the alternate DNS server you're testing (use the server's IPaddress rather than a name so your test can't accidentally involve looking up two names). The command will display two times to the nearest hundredth of a second (HH:MM:SS.hh). It's up to you to calculate the difference between these two times. Use the time difference from the very first test of each possible DNS alternative. It's quite likely that if you rerun the same command, the answer will come out of some cache the second time and arrive much quicker than it did the first time. This may or may not be what you want to measure, depending on whether your computer use generates a lot of repeat DNS requests or mostly novel requests.

The Problem: Apparent Interception

The problem is that many ways of accessing an alternate DNS service don't really have any effect; although you specify an alternate DNS service and your name translation requests appear to go to the alternate servers, requests really still use the ISP's preferred DNS service. This is quite difficult to diagnose. Not even network packet sniffing will diagnose this situation, as the packets still use port 53, and have been modified so both requests and responses are modified so they appear to go to and come from the correct desired alternate DNS service.

One of the better ways to diagnose the situation is to have a modified DNS server that can tell you via a separate channel whether you really just accessed it; but few DNS servers provide such a separate channel. The only generic way I know to diagnose this situation is to make a test DNS request to a DNS server you know does not exist, and see if you get a valid IPaddress answer anyway, indicating the request was in fact intercepted or hijacked. It's likely after you make a DNS change you will at first think everything is okay, but then later you will begin to notice unexplained inconsistencies of either results or features.

The behavior I was seeing was as though all port 53 traffic were being intercepted by the ISP and redirected to the server (host computer) the ISP preferred it to go to, despite my specification of a different server. My initial suspicion was this interception was happening somewhere inside my ISP's network. The problem was to thoroughly understand, then prevent or control this interception.

What's Really Going On  (and Why)

"Interception" and redirection of all port 53 packets is indeed happening, and in many cases the device doing it did indeed originally come from the ISP ...yet my initial suspicion was not fully correct. In my case ISP service is provided by Verizon, more specifically it's their FiOS optical broadband service. The device doing the interception turns out to be the router (NATbox) that Verizon supplies to its customers, and which is on the user premises and controllable by the user and ultimately owned by the user.

The first important implication of this is that any change to any individual computer (or individual computer programs) will have no effect. All it may do is confuse you. HowTo recipes that describe how to modify the DNS settings in an operating system will make the system look different yet will not actually have any effect. The second important implication is that all the computers that share the ISP connection will always be forced to use the same DNS service. If an alternate DNS service is selected, it will affect all computers.

Most modern routers accept port 53 packets addressed directly to them (as though they were a full DNS server) and redirect them to some outside actual DNS server (often defaulting to servers belonging to the ISP). Many routers (not just those supplied by an ISP) go even further, actually intercepting and rewriting all port 53 packets (not just packets addressed directly to them). This situation even has names: a home network's DNS style will be either Per-Computer DNS (Individual) style, or Router-Based DNS (Shared) style. But users seldom made a conscious choice, and in fact often don't even know which style their home network is using. The Router-Based DNS style has a couple possible advantages. One possible advantage is that DNS configuration changes need to be made only once in only one spot, yet they will affect all the computers that might possibly ever be used in the home (including computers that are powered off -or not even present- when the DNS configuration changes are made). A second possible advantage is users are prevented from circumventing SOHO (Small Office, Home Office) network restrictions (including DNS filtering) by using a different DNS server.

Unfortunately this router choice of DNS configuration styles is sometimes fixed and other times configured in an unwieldly variety of (sometimes incomplete) different ways, is almost completely undocumented, and is usually not even reflected in the router's internal log. (The D-Link DI604 is an example of a router where the Router-Based DNS style cannot be turned off no matter what.) In fact it often can't even be determined by reading whether a particular router includes the possibility of a Router-Based DNS configuration style at all, making it more difficult to choose which router to buy. Worse, the person who answers that router's support telephone number is likely to not know what to call the behavior, not know how to configure the behavior even on that router, not know how to find out if the behavior is currently active, not know what the possible advantages of the behavior are, and not even know their router is capable of that behavior at all (let alone that it does it by default).

Router documentation probably covers specification of a DNS server IPaddress, but gives the impression the address is used only by the router itself for its own internal functions (such as logging). Unless there's more than one place to specify a DNS server address, this impression is probably misleading; likely all DNS requests from all individual computers will be intercepted and redirected to this address (and likely DHCP responses will include this IPaddress too).

ISPs often arrange to use three closely related behaviors:

1. default configuration of routers such that home networks use the Router-Based DNS configuration style, and furthermore use the ISP's preferred DNS servers, even though users never made a conscious decision and don't understand what's actually happening at all
2. capture of NXDOMAIN (no such domain) error responses, and replacement with suggestions (and often a lot of advertising)
3. whenever a newly booted computer asks for an IPaddress, it's also given the IPaddresses of the ISP's preferred DNS servers

Since these behaviors are closely related and quite often appear together, from a business perspective they're often lumped together and collectively called intercepting or hijacking. But from a technical perspective they remain separate, and technically terms like intercepting or hijacking should refer specifically to the first of the three items only.

Redirecting all port 53 packets to Verizon's internal DNS servers is the default configured behavior of the Actiontec routers currently supplied to users by Verizon. Fortunately (but apparently by accident?) most HowTos for using an alternate DNS server on Verizon FiOS have as a side effect the (partial?) disabling of this behavior. However -like most routers- configuration of an Actiontec router in this area is complex, confusing, and very poorly documented, so any slight deviation from the procedure to compensate for an unusual SOHO network configuration is likely to inadvertently leave enabled the behavior of intercepting and redirecting all port 53 packets to an undesired DNS service.

Router behavior may be further complicated by having two DNS server IPaddress configurations, one on the LAN/inner/local side and another one on the WAN/outer/ISP side. Sometimes when the inner side interception behavior is disabled, requests will not be sent to the original address as you might expect, but rather will be handed off to the ISP side, which redirects them anyway (perhaps even to a different place).

Determining DNS Style

Execute the command on the first line below in a DOSbox on a Windows system. (This test assumes your Internet connectivity is working. If your Internet connectivity has failed, the results of this test may be incorrect.)

C:\WINDOWS\system32>nslookup www.census.gov 1.2.3.4
DNS request timed out.
    timeout was 2 seconds.
*** Can't find server name for address 1.2.3.4: Timed out
Server:  UnKnown
Address:  1.2.3.4

DNS request timed out.
    timeout was 2 seconds.
Non-authoritative answer:
Name:    e3604.dscna.akamaiedge.net
Address:  23.56.94.175
Aliases:  www.census.gov, www.census.gov.edgekey.net

The command generates a request to translate the name www.census.gov into an IPaddress, and sends it to the specific DNS server whose IPaddress is 1.2.3.4 ...but it's known there's not really any DNS service at all available at that IPaddress! If you nevertheless get some sort of correct IPaddress answer (such as the 23.56.94.175 shown in the example above), your network is currently using the Router-Based DNS configuration style, and DNS traffic is being silently intercepted and redirected by your router. (Ignore possible messages such as timeout and reply from unexpected source and Can't find server name and Non-authoritative; all that really matters is whether or not there's a valid IPaddress for www.census.gov anywhere in the response.) If you do not get a valid IPaddress for www.census.gov anywhere in the answer, your network is currently using the Per-Computer DNS style of configuration.

What To Do

Assuming your network is using the Router-Based DNS configuration style (which causes apparent packet interception, which is likely what brought you here in the first place), simply continue with the same configuration style, but make it work for you rather than against you to solve your apparent DNS interception problem.

Leave your computer OSs and programs unchanged. Access your router (probably via an internal web interface), find the place to enter DNS server address(es), and put in the IPaddress(es) supplied by the alternate DNS service you wish to use. Check if your router has more than one slot for a DNS server address, and if so change all of them to the primary and secondary and tertiary IPaddress(es) supplied by the alternate DNS service you wish to use. If there are more slots than you have addresses, change the extra ones to blank (or 0.0.0.0 or whatever your router accepts) so that none of the original contents remain. (Do not duplicate IPaddresses by entering the same address into more than one slot.)

Possible Further Problems

It's rumored (I haven't seen this myself, and am virtually certain it's not really true) that some versions of some routers even selectively intercept and modify DNS responses based not on the port number (as is typical) but rather on the contents of the packets; in particular these routers change a returned NXDOMAIN (does not exist) to instead report a positive outcome then point at a server the router company controls.

If the rumor is true and this happens to you and you find it quite objectionable, you might be able to get the router manufacturer's support to tell you how to turn off this behavior. But it's likely that your only option will be to buy a different router.

It's also possible that your ISP intercepts and modifies packets inside their network rather than in your router, but to date the hints I've heard that suggest such a thing is actually happening are too sketchy to be conclusive. If your ISP really does intercept DNS packets inside its network rather than in your router, this analysis will be incorrect and the suggestion above of What To Do will likely not resolve the problem.

What About Public Wireless And Cellphones?

If you either use public access wireless networking, or have a smartphone or other internet-capable cellphone, you will likely have no control over what DNS service you use. Usually the port number cannot be modified on either individual computers or smartphones or cellphones, and on smartphones and cellphones not even the server IPaddresses can be overridden in many cases. (Not-so-smart yet still internet-capable cellphones behave similarly to smartphones in this respect.) All these devices get a system IPaddress (and also the IPaddresses of the DNS servers to use) in the DHCP response from the local network infrastructure when they first connect to a Wi-Fi network. This occurs purely as a result of using DHCP, not because of any sort of intercepting (which may or may not happen separately later).

If a smartphone allowed DNS server IPaddresses to be overridden with local static information even though the IPaddress of the smartphone itself was obtained via DHCP, it might be possible to reasonably circumvent the forced use of the network's preferred DNS server(s) when connected via Wi-Fi. But although such partial IPaddress configuration has appeared in some Blu-ray devices, on MacOS, in recent versions of Windows, and probably elsewhere, I'm not aware of it having appeared on smartphones. (Besides, if the responsible ISP intercepts and modifies port 53 packets inside their own network infrastructure, even specifying alternate DNS server addresses this way would not work.)

The infrastructure's DHCP servers —which are controlled by the provider— virtually always supply the IPaddresses of the DNS servers on either the provider's or the ISP's premises. And the mobile device has no choice but to rely on those servers to perform all needed DNS lookups. The supplied servers usually belong either to the ISP or to the owner of the commercial establishment. (Some handhelds offer an option to opt out of the usual DHCP and instead specify static IPaddresses, including whatever DNS servers you desire. [This option may be quite well hidden - for example on my HTC Droid Eris after going to Wireless & networks" -> Wi-Fi Settings, you have to press the Menu button then select the Advanced icon that appears on the screen, and finally flick the screen to scroll down.] But while such an option is occasionally useful for a cellphone that stays in one location and is set up by a geek, it's of little use for cellphones that move around to different networks; with a completely static IPaddress configuration it's normally impossible to make a useful connection to whatever network is nearest at the moment.) Furthermore, it's likely that some networks also intercept DNS requests internally, which would make it even more impossible to control what DNS service you use.

So to put it baldly, it's almost certain that there's simply nothing at all that you can do (short of rooting and customizing your phone) about which DNS service a handheld device uses when outside your home or office. Unless you're a uber-geek or risk-loving (or both), attempting to use an alternate DNS service while utilizing public network access is nothing more than a recipe for a headache.