I Don’t Need no Stinking Firewall ... or Do I?
Brian Johnson started a lively “I don’t need no stinking firewall” discussion on NANOG mailing list in January 2010. I wanted to write about the topic then, but somehow the post slipped through the cracks… and I’m glad it did, as I’ve learned a few things in the meantime, including the (now obvious) fact that no two data centers are equal (the original debate had to do with protecting servers in large-scale data center).
First let’s rephrase the provocative headline from the discussion. The real question is: do I need a stateful firewall or is a stateless one enough?
Quick Overview
Firewalls or routers with packet filters?
Stateless firewalling is implemented quite easily with access lists on routers, switches, and virtual switches. It usually works at line speed. Stateful firewalls implemented in routers are usually suitable for low-speed remote offices; you should use dedicated firewall devices in data centers.
For more details read also The Spectrum of Firewall Statefulness.
Technology issues
Stateful firewall is the only option if you’re trying to tightly protect applications that use dynamic port numbers, including everything from peer-to-peer applications (including SIP) to RPC-based applications (let’s try not to call them broken ... how about unpredictable applications).
If your applications use only well-known fixed port numbers (let’s call them fixed-port applications), you don’t have to inspect the application data stream and can match the applications with access lists; stateless solutions seem appropriate.
However, some stateful firewalls can add value even in fixed-port environments: they can delay the commitment of server resources to TCP sessions with TCP SYN cookies (also available in numerous server operating systems) and check the validity of TCP sessions.
Last but not least, stateful firewall in front of a server can block TCP fingerprinting attempts. Sometimes you simply don’t want the attacker to know too much about your infrastructure.
Design choices
There are two extremes you can be facing in a data center:
Unified large-scale infrastructure using fixed-port applications, including Google, Yahoo, Facebook, Twitter and a few others. You would expect to see from tens to thousands of almost-identical servers with standardized and identically configured operating system in these environments. It’s quite manageable to harden and patch these servers and combined with fixed-port applications these environments usually offer, it makes perfect sense to be satisfied with router ACLs (and that was the case the proponents of “get rid of the firewall” line of thinking were promoting in the NANOG discussion).
Dynamic hodgepodge of servers, operating systems and application encountered in a usual enterprise data center. Server hardening is mission impossible (as you have so many different operating systems and/or versions of the same operating system), patching is the responsibility of individual server administrators (and they might not even be a unified team) and the applications are a nightmare from the security perspective.
For example, I was not able to find anything from Microsoft telling me how to configure my firewall to support OWA for Exchange Server 2010; the cynical response I got from a (non-Microsoft) security engineer a few days ago was “nobody can figure out which ports it uses”.
Now what?
Obviously you’re the only one who can decide where your Data Center environment is and which measures you’d like to implement, but as a generic rule, the more exposed (and the worse managed) a server is, the more you should lean toward a stateful firewall in front of it – no wonder most enterprise data centers make heavy use of stateful firewalls.
A thought provoking post :)
Server hardening may be "mission impossible" but it is still essential. Very rarely is every server provisioned in a separate broadcast domain, if you rely on a big-iron firewall for your security you are going to be left at risk from the zero-day that hits the machine next to you. Or even from the compromised administrator's laptop that connects without going through the firewall.
I think there is an argument for defence in depth, putting a dedicated resource between you and your greatest attack vector to reduce your load is always going to be a good idea. If your risk is great it may even be worth adding an additional firewall level from a second vendor in series. If a stateless firewall can capture the largest threats without blocking valid content then it may be the right choice, either on its own, or in addition to the stateful firewall.
Of course managing layered firewalls like this comes with its own configuration management issues that could put you at more risk of outage than running without the firewalls..
Sure some apps are hard to put behind static ACLs, but equally there are plenty of examples of where stateful firewalls get their application code wrong and can break networks just as badly as a badly written ACL (an example that immediately springs to mind is the SMTP code in early PIXes. For years the first command I would enter in a default PIX config was "no fixup smtp". Memory leaks in SIP ALGs is a more contemporary and widespread issue.)
Discounting endpoint security as "mission impossible" says to me that we need to find a new way to handle it because we sure as hell can't give up on it. Of course I don't have an answer for *how* we deal with it or I would be running away to the patent office right now ;)
More proxies, more high availability, more active active is my motto, and I have yet to meet a vendor who does not agree 8-)
Furthermore, even the largest firewalls available have state-tables which are easily exhausted either by legitimate flash crowds or by layer-7 attack traffic crafted in such a way as to appear perfectly legitimate, passing all firewall rules and 'inspectors'. Stateful firewalls are DDoS chokepoints, they can withstand far fewer sessions than the hosts behind them, and they always fall over during DDoS attacks.
And of course, the firewalls themselves are merely special-purpose computers running code - and there are bugs in that code, as the list of security flaws and patches for every commercial firewall vendor demonstrates. Add this to the fact that the code written for these devices receives much less scrutiny than OS/application/service code, and one can see that in addition to the availability threat they pose, firewalls also greatly broaden the attack surface and potentially provide obscure vectors for attackers to compromise these blackbox devices.
None of the large-scale Internet properties with which readers are familiar put stateful firewalls (or so-called 'IPS') in front of their servers - at least, the ones which stay up and running.
Stateful firewalls have no place whatsoever in front of servers, period. There's no state to inspect, and they greatly reduce the security posture of organizations which make the mistake of placing them in front of servers.
Stateless ACLs in hardware-based routing/switching platforms which can handle millions of packets-per-second in hardware are the BCP for enforcing communications policy for servers.
This NANOG presentation succinctly demolishes the myth that firewalls (and other stateful inspection devices) should be placed inline in front of servers:
http://www.nanog.org/meetings/nanog48/presentations/Monday/Kaeo_FilterTrend_ISPSec_N48.pdf
http://meetings.apnic.net/__data/assets/pdf_file/0010/13987/rokusaddosjul09preso.pdf
http://webcast.apnic.net/meetings/28/apops-10-roland-dobbins.mov
Obviously I can't possibly disagree with anything you wrote; your facts are absolutely correct. However, you're focusing at a particularly exposed and demanding market segment: large content providers. As I wrote in the post, they are one extreme of the whole spectrum, the other one being a mid-sized enterprise (don't want to go down to SMB) with very mixed server environment and application requiring dynamic ports.
While the lessons learned in high-volume environment are definitely worth considering (and as I wrote, they are best served by router-based ACLs), not everyone has the same amount of knowledge, the team size that these content providers enjoy, the same requirements or the same exposure. These lessons are thus not directly applicable to other environments, as they can sometimes do more harm than good (see the other end of the spectrum).
What you're talking about are F1 racing cars of the content business. That's where the technology is stressed, where we learn how to improve it and how to design it better (one can only hope the same knowledge transfer would work in the IT industry); but the methodologies used there cannot be applied directly to a commodity car (which will also not be exposed to most of the stresses an F1 racing car is).
as you might have realized, the readers of my blog vary significantly in their exposure to particular technologies. While Roland questions the validity of my arguments, you're asking "what's the point". Obviously you need someone to explain the basics of stateless versus stateful firewalls. The articles in Wikipedia are not bad and look to be the first hit on Google when you're searching for "stateful firewall" (they refer to stateless firewalls as "packet filters") ...
http://en.wikipedia.org/wiki/Stateful_firewall
http://en.wikipedia.org/wiki/Packet_filter
... if you need more, just let me know; I might be able to help you.
Now, Bro, how about some common decency ... like using your name like those that argue with the content of my post ... or politely telling me "hey, Ivan, I would appreciate if you could explain the difference between stateful and stateless firewalls, I was not able to find a decent document anywhere".
Although, what do you do to protect against TCP SYN DoS attacks?
Roland says Cisco should stop selling/recommending stateful firewalls which are put infront of servers which receive unsolicited connections.
In other situations, stateful firewalls are a perfect fit, f.e. permit inside -> outside traffic, while blocking the outside -> inside traffic.
Put another way - what is the advantage of putting of stateful firewall in front of public facing DNS server?
In an enterprise environment where the number of connections between various services grows exponentially, PVLANs could become a nightmare.
As for server hardening: I'm far away from saying it can't be done; I'm just saying that not everyone knows how to do it ... unfortunately.
Being in higher ed, I really can't imagine NOT having a firewall. Now, NAC...whole different story.
You mentioned TCP SYN cookie support in stateful firewalls. Frankly, I think this is an awful feature. It requires the firewall to intercept/terminate the TCP session, and can lead to lots of crazy failure scenarios. Imagine, for example, if the server (or an intermediate router) doesn't know the path back to the client. TCP 3 way handshake appears to succeed, but then the application never responds. What has really happened is that the client has never established 2-way communication with the server. ...and for what benefit? TCP SYN cookie support is easy to implement. A server that doesn't have it probably shouldn't be facing the wrath of the Internet.
Also, about TCP fingerprinting... You said that a stateful firewall *can* prevent fingerprinting. I'm not sure that's true -- you'd need much more than just state awareness to prevent fingerprinting. You'd need to strip funny TCP options, rewrite sequence numbers... You're getting much closer to a full-blown TCP proxy than you are a stateful firewall. Not that it's a big deal (netcat can do it), it's just not something that I think of as a firewall function. Do firewall vendors market this as a feature?
I haven't used the CUBE personally, but I was on a project where we were using a dedicated SBC appliance from a top SBC vendor but we replaced it with a ASA because we also wanted to control some other protocols like FTP, TFTP, etc ...
Fingerprinting is irrelevant, anyways. If one really wishes to obfuscate at that level, there are kernel patches which perform this functionality.
Putting a stateful inspection device at a point in the topology *where there is no state to inspect in the first place* doesn't make sense at any scale. It is insane, IMHO.
Firstly, yes, TCP/IP stacks should be hardened.
Secondly, S/RTBH is a great, low-cost mitigation mechanism for all kinds of packet-flooding attacks.
Thirdly, intelligent DDoS mitigation systems (IDMS) can handle layer-3 through layer-7.
[Full disclosure; I was heavily involved with the former Cisco (nee Riverhead) Guard IDMS when I was at Cisco, and now work for a vendor of IDMS systems.]
;>
And, Pavel, you're entirely correct - stateful firewalls have an important role to play in separating enterprise client access networks, home networks, and other endpoint networks from 'The Internet', in order to keep unsolicited connections off machines which are primarily workstations.
But they shouldn't be wedged into broadband access networks due to scaling and near-infinite user application diversity, and they need to be removed from mobile wireless data networks, as they cause more problems than they solve (their presence on these networks is primarily a legacy of unfamiliarity with Internet-scale connectivity requirements at the time many of these networks were initially constructed).
;>
;>
In terms of being forced to deploy stateful firewalls by various subspecies of auditors - the nonsensical PCI DSS 'Web application firewall' requirement can be met via mod_security. In other scenarios, education in order to help the auditors get past the 'firewalls = security' disinformation is the long-term solution.
...And I didn't mean to suggest that fingerprinting prevention was a feature worth asking for!
I just meant to explain that stateful inspection doesn't suggest anti-fingerprinting in my mind. There are so many fingerprinting vectors that it'd be tough for a man-in-the-middle to stop them all, and mere "stateful inspection" stops none of them.
As a matter of considerable amusement, one large commercial firewall vendor has implemented a 'stateful inspection bypass' feature - meaning that they expect you to spend $100K USD on their 10gb/sec stateful firewall, then *disable the stateful inspection which was the justification for buying it in the first place*, heh. This is a tacit admission that stateful inspection has no role in front of servers, as there's no state to inspect.
The DDoS vector and increase in the overall compromise attack surface these devices bring with them are very real threats, and are additional reasons for not deploying firewalls in front of servers; but even if these weren't concerns (and they're HUGE concerns), the fundamental illogicality of inserting a stateful inspection device in the topology where there is in fact no state to inspect is the larger point.
If one is forced to install firewalls in front of one's servers despite the obvious disadvantages and security risks of doing so, the firewalls and everything behind them should be protected via reaction mechanisms such as S/RTBH, flowspec, and/or IDMS.
With (godawful) SYN cookie support on the firewall, the firewall MIGHT die trying to proxy the incoming TCP handshakes. But it won't fall over because of state table exhaustion. The only risk is in generating the cookie-ized ISN in the outgoing SYN/ACK.
Do you say "more quickly" because state table exhaustion takes some time, while interrupt/CPU issues can be switched on and off instantly?
This is correct as the security problem is not solved in the network. It is like debating which is better a sour egg or a rotten egg.
The network also has a role to play in terms of mitigating DDoS attacks via reaction mechanisms such as S/RTBH and/or IDMS.
Availability is an area in which the network can contribute, but layer-7 must be designed, implemented, and operated properly, or what's done in the network doesn't matter.
The environment I protect is an enterprise environment, with a mix of off-the-shelf and custom-written software: Ivan's dynamic hodgepodge.
Stateful firewalls were the answer to a failure of application *and* operating system authors to properly secure their code. These machines, hence, could not be trusted to manage their own connections properly, and so a separate purpose-built appliance was needed.
It really comes down to economics. The marketplace (or the bazaar) has largely proven that we are unwilling to spend more time & money on software to ensure it is well-audited, free of security holes. We collectively prefer new features in our software.
Anyway, the stateful firewalls that I prefer give me everything I need: centralized management of a well-organized security policy. (Not to mention fulfillment of the auditors' requirement that we have such a firewall in place.)
Moreover, I'm not convinced that the app & OS authors have gotten any better at writing their networking code, code that is perhaps not shielded by a stateless ACL.
DDOS is also less of a concern for me. I don't have a high profile, and to date I've never been attacked in that manner. It may well happen someday, and if it does, I've got BGP and stateless ACL tools to deal with it.
As Ivan wrote: different data centers, different solutions.
-Brian
Thank you for great post.
I am interesting what are you thinking about statefull firewall inside enterprise data center, where all resources are owned by enterprise, statefull firewall between different tears of web application, and stetefull firewall between different application groups, lets say there is MS environment where every MS server has joined AD, and to segment these different application servers with stetefull firewall ?
Thank you,
Giorgi
http://www.ipspace.net/Replacing_the_Central_Firewall