Securing proxy server back ends

Some people think that in June 2008 security researcher Dan Kaminsky discovered the subset of these problems that apply specifically to proxy DNS servers, and they talk of the "Kaminsky DNS vulnerability". In fact, Kaminsky didn't discover anything that wasn't already known and published years before.

The Frequently Given Answer on proxy server IP addresses mentioned the very problem that Kaminsky supposedly "discovered". It was published in 2007, which is at least a year before Kaminsky announced his discovery and (possibly coincidentally) also the approximate date that Kaminsky is reported, by journalists and web loggers, to have "begun looking into the problem" in the first place.

The Frequently Given Answer made no claim to primacy, for the simple reason that it was based upon mailing list and newsgroup discussions, also all public, that occurred for several years before its publication, and that in their turn were instigated by Daniel J. Bernstein's documenting these particular flaws in the DNS protocol, which he did on his DNS forgery WWW page back in 2002, after postings to various public DNS-related mailing lists that he made on the subject in July 2001.

So proper credit belongs to Bernstein, if anyone, not to Kaminsky. (That is for DNS/UDP protocol hijacking, at any rate. Brecht Claerhout's explanation of how to hijack a TCP/IP connection by forging responses, with worked examples, was published in 1996, and is still in circulation today. Steve Bellovin discussed TCP/IP session hijacking even earlier, in 1989 in an article in Computer Communication Review. All of these involve the same basic protocol design weaknesses, of not-very-random or easy-to-blindly-forge port/sequence/ID numbers, that Kaminsky "discovered" in the DNS protocol almost two decades later.)

You've come to this page because you've asked a question similar to the following:

What are the security issues for the back ends of proxy servers?

This is the Frequently Given Answer to that question.

The considerations, relating to impersonation of back-end servers, apply to both proxy DNS servers and proxy HTTP servers, as well as to proxy servers for several other protocols. All of these protocols involve the proxy server performing a back-end request/response transaction, with one (or several) content servers, the results of which it then serves up to its clients (and caches, if it is a caching proxy server).

Note: For many protocols these problems are also security problems for the front end of a proxy server. Attackers can impersonate responses from proxy servers to clients just as they can impersonate responses from content servers to proxy servers. However, these problems are mitigated by proper selection of the IP addresses that proxy servers listen on.

The security problems

The problems are that attackers can forge IP traffic, purporting to come from the content servers that the back ends of the proxy servers are attempting to talk to. Such forged traffic contains the correct "source" IP addresses and UDP/IP or TCP/IP port numbers, and contains false data that the attacker wishes the proxy server to then serve up to the proxy server's actual front-end client(s), and add to its cache. Attackers can thus insert false data into the caches of caching proxy servers and provide false data to the clients of proxy servers.

For TCP/IP traffic, this involves forging the correct TCP sequence numbers in the IP datagrams. For UDP/IP traffic, there are no sequence numbers in the actual transport protocol, unlike the case with TCP. For both transport protocols, forgery also involves forging the appropriate portions of the protocols layered on top of TCP/IP or UDP/IP.

Unfortunately, for many protocols, this is not a large "space" that an attacker has to guess the right numbers within.

Examples:

For the DNS protocol, DNS/UDP/IP forgery involves forging the correct UDP/IP port numbers and the correct 16-bit message IDs in the actual DNS protocol queries and responses. Thus the potential number of possible responses is only 65536 (the number of possible 16-bit DNS protocol message IDs) multiplied by the number of back-end UDP/IP ports in use by the proxy server.

For some proxy DNS server softwares, this is just one back-end DNS/UDP/IP port, leading to a mere 65,536 possibilities (which a brute force attacker can exhaustively cover in a matter of seconds on, say, a 10Mib/s link). This is the case for several versions of BIND, for example, which send their queries from port 53. For Microsoft's DNS server, with the June 2008 fix for this very issue that increases to 163,840,000 possibilities (2,500 ports with a 16-bit message ID).
For NTP, NTP/UDP/IP forgery involves forging only the correct UDP/IP port numbers, since there are no transaction IDs in NTP itself at all. Thus the potential number of possible responses is 65536, and in practice is less for many systems, which disallow the use of port numbers 1023 and below as source ports for service clients.

Severity of the problems

These problems have some very severe consequences if unaddressed:

An attacker who can send forged NTP responses to one's NTP clients can (depending from what operating systems one uses) slow down, stop, or even reset to an earlier time the system clocks on one's machines. Many caching proxy servers use the system clocks of the operating systems upon which they run to determine when cached data have expired. An attacker who successfully forges NTP responses can thus control when cached forged data that xe has injected into a proxy server expires, and possibly prevent it from ever expiring.
An attacker who can send forged DNS responses to one's resolving proxy DNS servers can control where most of one's systems and softwares think the rest of Internet even is in the first place. Xe can pretend to be Google, or Microsoft, or the BBC, or Xinhua, or the U.S. Department of Defence. With some careful crafting of security certificates (which commonly encode domain names, not actual IP addresses), xe can even make secured TCP/IP services appear to be secure, even though connecting to fraudulent Internet servers for those services, except to the more eagle-eyed user.
An attacker who can send forged HTTP responses to one's proxy HTTP servers can control what one thinks the World Wide Web looks like. (An attacker who did this for the back-ends of the proxy HTTP servers run by large organizations or some Internet Service providers could influence rather a lot of people's view of the World Wide Web.)

The classes of attackers

Attackers are classified as either "blind" or not. "Blind" attackers have no access to the requests that the proxy server back-ends make, whereas non-blind attackers can snoop the network traffic coming from the back ends of the proxy servers, and know what requests are being made to content servers.

Good practice

In an ideal world, the actual protocols would be secure. A client making a query would be able to include a (very hard to guess) nonce and a public key, which the server would use to sign the response. Blind attackers wouldn't be able to guess the nonce in any reasonable amount of time (even if they knew the preceding sequence of nonces), and so wouldn't be able to generate properly signed forged responses. There are HTTP variants that do this, and Dan Bernstein has suggested a scheme for modifying the DNS protocol to incorporate this. No-one has yet made much of a fuss over the lack of security in NTP, albeit that Dan Bernstein proposed (back in 1997) a replacement protocol that can be made secure. (It is left as an exercise to the reader to determine the odds of Dan Kaminsky, in another few years' time, announcing the discovery of the "Kaminsky NTP vulnerability" long after it was publicly discussed in a Frequently Given Answer and documented by Bernstein. ☺)

In reality the protocols actually in use are not secure. Therefore one has to adopt some best practices that reduce, but do not wholly prevent, the problems.

Isolate back-end IP traffic

If an attacker is on the same physical network segment, or even network, as the back-ends of one's proxy servers, xe can obtain total control over one's proxy service.

A non-blind attacker has full access to the original, outgoing, back-end requests from the proxy servers, and so can craft fraudulent responses with all of the correct message IDs, port numbers, TCP sequence numbers, and so forth. (There's a whole load of hooplah about DNSSEC that people try to interject at this point, when this point is made with respect to proxy DNS servers. The simple truth is that non-blind attackers can do all sorts of things that DNSSEC can, by design, do nothing about, such as replaying old, properly signed, responses with out-of-date IP addresses that the attacker has since taken over; or replaying the signed responses for one continent to clients in another continent and then intercepting the cross-continental traffic. To protect against non-blind attackers, one requires confidentiality of the back-end transaction between proxy server and content server, a quality that DNSSEC explicitly, per its own documentation, does not provide. A security extension like DNSSEC for any other protocol would share DNSSEC's deficiencies in this regard, for the same reason.)

Best practice is to always ensure that no machines (that one does not trust completely) can snoop the UDP/IP and TCP/IP traffic between the back-ends of one's proxy servers and the content servers on the rest of Internet. Keep the transactions confidential (as far as one's own borders, at any rate) by preventing the network traffic from flowing along the wrong wires, or out into the wild blue yonder.

Wireless network traffic is visible to anyone within radio range, remember. So one must not have any wireless network connections at any point between one's proxy servers and Internet at large.

Ethernet datagrams, remember, are visible to all machines on any given network segment. At the very minimum, therefore, one should use something such as a separate, dedicated, network segment (for the IP traffic between the proxy server back ends and one's border routers, and hence the rest of Internet at large) to which only trusted machines are attached, so that this IP traffic does not appear on the network segments to which untrusted machines are attached.

Even using a dedicated network segment does not prevent attackers from doing things such as ARP/RARP spoofing in order to obtain the outgoing request traffic from proxy servers. Not only a separate network segment, but a separate network, is required in order to prevent such local, non-blind, attacks against proxy servers.

Use good IP addresses for front ends

Even a blind attacker can better xyr chances if xe can request service from one's proxy servers' front ends.

A blind attacker can more easily know when to blindly generate forged content server responses if xe can send queries to the front end of the proxy server. A blind attacker who can send a GET request to one's proxy HTTP server knows when the back-end of that server is more likely to send a back-end GET request to content HTTP servers. A blind attacker who can send a DNS query to one's proxy DNS server knows when the back-end of that server is more likely to send a back-end DNS query to content DNS servers.

If the IP addresses on which proxy servers listen are not reachable by Internet at large, as is best practice, blind attackers do not have this capability, and have to rely upon more indirect mechanisms.

Such indirect mechanisms still exist. A blind attacker can get local clients to make DNS lookups or to make HTTP requests, via one's proxy DNS servers and proxy HTTP servers, in various ways. Xe can do a wide range of things, from sending junk electronic mail with embedded hyperlinks in it to people using local machines (in the hope that those people use some of the badly-designed Mail User Agents in existence), to putting up WWW pages with embedded images and frames whose URLs contain domain names whose content DNS or content HTTP service the attacker controls (in the hope that people on local machines will browse those WWW pages). Xe could even just buy the appropriate WPAD domain name for a country and bask in the happy knowledge that Microsoft, Mozilla, Google and others design their WWW browsers to automatically make DNS queries and HTTP requests for domains that an attacker can easily control.

© Copyright 2009 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.