I Don’t Need no Stinking Firewall ... or Do I?

Brian Johnson started a lively “I don’t need no stinking firewall” discussion on NANOG mailing list in January 2010. I wanted to write about the topic then, but somehow the post slipped through the cracks… and I’m glad it did, as I’ve learned a few things in the meantime, including the (now obvious) fact that no two data centers are equal (the original debate had to do with protecting servers in large-scale data center).

First let’s rephrase the provocative headline from the discussion. The real question is: do I need a stateful firewall or is a stateless one enough?

Quick Overview

Firewalls or routers with packet filters?

Firewalls or routers with packet filters?

Stateless firewalling is implemented quite easily with access lists on routers, switches, and virtual switches. It usually works at line speed. Stateful firewalls implemented in routers are usually suitable for low-speed remote offices; you should use dedicated firewall devices in data centers.

For more details read also The Spectrum of Firewall Statefulness.

Technology issues

Stateful firewall is the only option if you’re trying to tightly protect applications that use dynamic port numbers, including everything from peer-to-peer applications (including SIP) to RPC-based applications (let’s try not to call them broken ... how about unpredictable applications).

You can limit the dynamic port range for some of these applications and allow all ports in that range through the firewall… while hoping that some other service on your server won’t grab one of those ports and expose itself unnecessarily.

If your applications use only well-known fixed port numbers (let’s call them fixed-port applications), you don’t have to inspect the application data stream and can match the applications with access lists; stateless solutions seem appropriate.

However, some stateful firewalls can add value even in fixed-port environments: they can delay the commitment of server resources to TCP sessions with TCP SYN cookies (also available in numerous server operating systems) and check the validity of TCP sessions.

You might think that there are no vulnerabilities left in TCP that could be exploited. A long while ago, everyone thought it was impossible to establish one-way spoofed TCP sessions even though there were known vulnerabilities in TCP sequence number generation… until Kevin Mitnick proved them wrong.

Last but not least, stateful firewall in front of a server can block TCP fingerprinting attempts. Sometimes you simply don’t want the attacker to know too much about your infrastructure.

Design choices

There are two extremes you can be facing in a data center:

Unified large-scale infrastructure using fixed-port applications, including Google, Yahoo, Facebook, Twitter and a few others. You would expect to see from tens to thousands of almost-identical servers with standardized and identically configured operating system in these environments. It’s quite manageable to harden and patch these servers and combined with fixed-port applications these environments usually offer, it makes perfect sense to be satisfied with router ACLs (and that was the case the proponents of “get rid of the firewall” line of thinking were promoting in the NANOG discussion).

Dynamic hodgepodge of servers, operating systems and application encountered in a usual enterprise data center. Server hardening is mission impossible (as you have so many different operating systems and/or versions of the same operating system), patching is the responsibility of individual server administrators (and they might not even be a unified team) and the applications are a nightmare from the security perspective.

For example, I was not able to find anything from Microsoft telling me how to configure my firewall to support OWA for Exchange Server 2010; the cynical response I got from a (non-Microsoft) security engineer a few days ago was “nobody can figure out which ports it uses”.

Now what?

Obviously you’re the only one who can decide where your Data Center environment is and which measures you’d like to implement, but as a generic rule, the more exposed (and the worse managed) a server is, the more you should lean toward a stateful firewall in front of it – no wonder most enterprise data centers make heavy use of stateful firewalls.

38 comments:

  1. Hi Ivan,

    A thought provoking post :)

    Server hardening may be "mission impossible" but it is still essential. Very rarely is every server provisioned in a separate broadcast domain, if you rely on a big-iron firewall for your security you are going to be left at risk from the zero-day that hits the machine next to you. Or even from the compromised administrator's laptop that connects without going through the firewall.

    I think there is an argument for defence in depth, putting a dedicated resource between you and your greatest attack vector to reduce your load is always going to be a good idea. If your risk is great it may even be worth adding an additional firewall level from a second vendor in series. If a stateless firewall can capture the largest threats without blocking valid content then it may be the right choice, either on its own, or in addition to the stateful firewall.

    Of course managing layered firewalls like this comes with its own configuration management issues that could put you at more risk of outage than running without the firewalls..

    Sure some apps are hard to put behind static ACLs, but equally there are plenty of examples of where stateful firewalls get their application code wrong and can break networks just as badly as a badly written ACL (an example that immediately springs to mind is the SMTP code in early PIXes. For years the first command I would enter in a default PIX config was "no fixup smtp". Memory leaks in SIP ALGs is a more contemporary and widespread issue.)

    Discounting endpoint security as "mission impossible" says to me that we need to find a new way to handle it because we sure as hell can't give up on it. Of course I don't have an answer for *how* we deal with it or I would be running away to the patent office right now ;)
  2. Of course no one needs a stinking firewall, all you need is NAT.
  3. Ofcourse we need statefull firewalls AND stateless router ACL's in front of everything. It's a great idea to implement an additional source of issues in front of an OS with issues carrying apps with issues. If not in obscurity & complexity, where would the fun in our jobs be?
    More proxies, more high availability, more active active is my motto, and I have yet to meet a vendor who does not agree 8-)
  4. Stateful inspection makes zero sense when every incoming connection is unsolicited - as is the case with Web servers, DNS servers, et. al.

    Furthermore, even the largest firewalls available have state-tables which are easily exhausted either by legitimate flash crowds or by layer-7 attack traffic crafted in such a way as to appear perfectly legitimate, passing all firewall rules and 'inspectors'. Stateful firewalls are DDoS chokepoints, they can withstand far fewer sessions than the hosts behind them, and they always fall over during DDoS attacks.

    And of course, the firewalls themselves are merely special-purpose computers running code - and there are bugs in that code, as the list of security flaws and patches for every commercial firewall vendor demonstrates. Add this to the fact that the code written for these devices receives much less scrutiny than OS/application/service code, and one can see that in addition to the availability threat they pose, firewalls also greatly broaden the attack surface and potentially provide obscure vectors for attackers to compromise these blackbox devices.

    None of the large-scale Internet properties with which readers are familiar put stateful firewalls (or so-called 'IPS') in front of their servers - at least, the ones which stay up and running.

    Stateful firewalls have no place whatsoever in front of servers, period. There's no state to inspect, and they greatly reduce the security posture of organizations which make the mistake of placing them in front of servers.

    Stateless ACLs in hardware-based routing/switching platforms which can handle millions of packets-per-second in hardware are the BCP for enforcing communications policy for servers.

    This NANOG presentation succinctly demolishes the myth that firewalls (and other stateful inspection devices) should be placed inline in front of servers:

    http://www.nanog.org/meetings/nanog48/presentations/Monday/Kaeo_FilterTrend_ISPSec_N48.pdf
  5. Roland, you bring up many good points! It is often difficult to find a nice balance between strong protection measures and resource utilization. Strong gates do not a castle make.
  6. For an example of how firewalls fall over during DDoS attacks, as well as a list of best current practices (BCPs) for protecting servers, see my APNIC presentation here:

    http://meetings.apnic.net/__data/assets/pdf_file/0010/13987/rokusaddosjul09preso.pdf

    http://webcast.apnic.net/meetings/28/apops-10-roland-dobbins.mov
  7. Thanks for the comment, Roland ... it's been a while since we last "spoke" and it's nice to hear from you.

    Obviously I can't possibly disagree with anything you wrote; your facts are absolutely correct. However, you're focusing at a particularly exposed and demanding market segment: large content providers. As I wrote in the post, they are one extreme of the whole spectrum, the other one being a mid-sized enterprise (don't want to go down to SMB) with very mixed server environment and application requiring dynamic ports.

    While the lessons learned in high-volume environment are definitely worth considering (and as I wrote, they are best served by router-based ACLs), not everyone has the same amount of knowledge, the team size that these content providers enjoy, the same requirements or the same exposure. These lessons are thus not directly applicable to other environments, as they can sometimes do more harm than good (see the other end of the spectrum).

    What you're talking about are F1 racing cars of the content business. That's where the technology is stressed, where we learn how to improve it and how to design it better (one can only hope the same knowledge transfer would work in the IT industry); but the methodologies used there cannot be applied directly to a commodity car (which will also not be exposed to most of the stresses an F1 racing car is).
  8. Dear Guest,

    as you might have realized, the readers of my blog vary significantly in their exposure to particular technologies. While Roland questions the validity of my arguments, you're asking "what's the point". Obviously you need someone to explain the basics of stateless versus stateful firewalls. The articles in Wikipedia are not bad and look to be the first hit on Google when you're searching for "stateful firewall" (they refer to stateless firewalls as "packet filters") ...

    http://en.wikipedia.org/wiki/Stateful_firewall
    http://en.wikipedia.org/wiki/Packet_filter

    ... if you need more, just let me know; I might be able to help you.

    Now, Bro, how about some common decency ... like using your name like those that argue with the content of my post ... or politely telling me "hey, Ivan, I would appreciate if you could explain the difference between stateful and stateless firewalls, I was not able to find a decent document anywhere".
  9. Your views intrigue me and I would like to subscribe to your newsletter.

    Although, what do you do to protect against TCP SYN DoS attacks?
  10. Sweet - I'm going to tell my Cisco SE that we no longer want to purchase those ASA5580s, under the advisement of a CSCO solutions architect. :-P
  11. Your network, your risk, your decision :-P
  12. Notice that Roland does not say that Cisco should stop selling/recommending firewalls.
    Roland says Cisco should stop selling/recommending stateful firewalls which are put infront of servers which receive unsolicited connections.

    In other situations, stateful firewalls are a perfect fit, f.e. permit inside -> outside traffic, while blocking the outside -> inside traffic.

    Put another way - what is the advantage of putting of stateful firewall in front of public facing DNS server?
  13. If you have numerous servers in a multi-tenant environment, there's always PVLAN or layer-3 isolation with P2P interfaces (assuming your aggregation gear knows what's above L2).

    In an enterprise environment where the number of connections between various services grows exponentially, PVLANs could become a nightmare.

    As for server hardening: I'm far away from saying it can't be done; I'm just saying that not everyone knows how to do it ... unfortunately.
  14. ....nothing, Roland operates under assumtion that servers should protect themselves via implementing TCP cookies by default....moving the problem into the server space (which is actually where it belongs, but we spoiled them)
  15. Ah, if only that was an acceptable answer for management.
  16. My network, my risk, state auditor's decision :'(


    Being in higher ed, I really can't imagine NOT having a firewall. Now, NAC...whole different story.
  17. Ivan,

    You mentioned TCP SYN cookie support in stateful firewalls. Frankly, I think this is an awful feature. It requires the firewall to intercept/terminate the TCP session, and can lead to lots of crazy failure scenarios. Imagine, for example, if the server (or an intermediate router) doesn't know the path back to the client. TCP 3 way handshake appears to succeed, but then the application never responds. What has really happened is that the client has never established 2-way communication with the server. ...and for what benefit? TCP SYN cookie support is easy to implement. A server that doesn't have it probably shouldn't be facing the wrath of the Internet.

    Also, about TCP fingerprinting... You said that a stateful firewall *can* prevent fingerprinting. I'm not sure that's true -- you'd need much more than just state awareness to prevent fingerprinting. You'd need to strip funny TCP options, rewrite sequence numbers... You're getting much closer to a full-blown TCP proxy than you are a stateful firewall. Not that it's a big deal (netcat can do it), it's just not something that I think of as a firewall function. Do firewall vendors market this as a feature?
  18. Talking about stateful firewalls, would SBCs like the Cisco UBE be considered specialized SIP firewalls? I assume they must be stateful even though SIP uses a well known udp port 5060 it is only used to set up audio or video sessions that can use any dynamically assigned udp ports.

    I haven't used the CUBE personally, but I was on a project where we were using a dedicated SBC appliance from a top SBC vendor but we replaced it with a ASA because we also wanted to control some other protocols like FTP, TFTP, etc ...
  19. Concur 100%. SYN-cookies should be handled on the hosts themselves.

    Fingerprinting is irrelevant, anyways. If one really wishes to obfuscate at that level, there are kernel patches which perform this functionality.
  20. Respectfully (and good to 'speak' with you, too, heh), these issues are even *more* important for the smaller fry. They get absolutely crushed with even legitimate flash crowd traffic, much less direct or collateral DDoS.

    Putting a stateful inspection device at a point in the topology *where there is no state to inspect in the first place* doesn't make sense at any scale. It is insane, IMHO.
  21. Actually, it's a bit more nuanced than that.

    Firstly, yes, TCP/IP stacks should be hardened.

    Secondly, S/RTBH is a great, low-cost mitigation mechanism for all kinds of packet-flooding attacks.

    Thirdly, intelligent DDoS mitigation systems (IDMS) can handle layer-3 through layer-7.

    [Full disclosure; I was heavily involved with the former Cisco (nee Riverhead) Guard IDMS when I was at Cisco, and now work for a vendor of IDMS systems.]
  22. To clarify, I'm no longer with Cisco - but I was 'infamous' in certain quarters within Cisco for espousing these very same views when I was there, so my strong recommendation against stateful firewalls in front of any server, anytime, anywhere has been consistent over the years.

    ;>

    And, Pavel, you're entirely correct - stateful firewalls have an important role to play in separating enterprise client access networks, home networks, and other endpoint networks from 'The Internet', in order to keep unsolicited connections off machines which are primarily workstations.

    But they shouldn't be wedged into broadband access networks due to scaling and near-infinite user application diversity, and they need to be removed from mobile wireless data networks, as they cause more problems than they solve (their presence on these networks is primarily a legacy of unfamiliarity with Internet-scale connectivity requirements at the time many of these networks were initially constructed).
  23. Due to the pessimal communications models of grotesqueries such as SIP, H.323, et. al., things like SBCs and IP PBXes and so forth should be shielded from a policy perspective as best they can via stateless ACLs in hardware, their inherent stateful inspection mechanisms must be enabled, and operators must be prepared to maintain their availability in the face of attack via mechanisms such as S/RTBH, flowspec, and/or IDMS.
  24. NAT in front of servers is even worse than stateful firewalls - guaranteed collapse in the event of even a minimal flash crowd, much less directed or collateral DDoS.
  25. Preach it, brother!

    ;>
  26. Thanks for the kind words, and I concur with your cited metaphor 100%!

    ;>
  27. Concur, the whole concept of trusting endpoint self-assessment reporting is inherently flawed - I argued heavily against it on this basis when it was first posited, and it's with no pleasure that I note that events have borne out this view.

    In terms of being forced to deploy stateful firewalls by various subspecies of auditors - the nonsensical PCI DSS 'Web application firewall' requirement can be met via mod_security. In other scenarios, education in order to help the auditors get past the 'firewalls = security' disinformation is the long-term solution.
  28. So, is it correct to assume that your main point is: "stateful firewalls are not good DDoS mitigation devices"? I can easily agree with that.
  29. I guess that if you're going to ignore Roland, and install a stateful firewall in front of servers, then putting the (godawful) TCP SYN cookie mechanism in the firewall is helpful. It helps the firewall survive in a place where it doesn't belong.

    ...And I didn't mean to suggest that fingerprinting prevention was a feature worth asking for!

    I just meant to explain that stateful inspection doesn't suggest anti-fingerprinting in my mind. There are so many fingerprinting vectors that it'd be tough for a man-in-the-middle to stop them all, and mere "stateful inspection" stops none of them.
  30. No - my main point is that it makes zero sense to insert a stateful inspection device in front of servers, where there's no state to inspect in the first place, as every incoming connection is unsolicited. Stateless ACLs in hardware-based routers/layer-3 switches are the correct way to enforce network access policies.

    As a matter of considerable amusement, one large commercial firewall vendor has implemented a 'stateful inspection bypass' feature - meaning that they expect you to spend $100K USD on their 10gb/sec stateful firewall, then *disable the stateful inspection which was the justification for buying it in the first place*, heh. This is a tacit admission that stateful inspection has no role in front of servers, as there's no state to inspect.

    The DDoS vector and increase in the overall compromise attack surface these devices bring with them are very real threats, and are additional reasons for not deploying firewalls in front of servers; but even if these weren't concerns (and they're HUGE concerns), the fundamental illogicality of inserting a stateful inspection device in the topology where there is in fact no state to inspect is the larger point.
  31. Actually, the SYN-cookie feature will end up causing the firewall to die more quickly under duress due to processing/interrupt overhead and memory exhaustion.

    If one is forced to install firewalls in front of one's servers despite the obvious disadvantages and security risks of doing so, the firewalls and everything behind them should be protected via reaction mechanisms such as S/RTBH, flowspec, and/or IDMS.
  32. With SYN cookie support on the servers, the firewall WILL die of state table exhastion in a SYN flood attack.

    With (godawful) SYN cookie support on the firewall, the firewall MIGHT die trying to proxy the incoming TCP handshakes. But it won't fall over because of state table exhaustion. The only risk is in generating the cookie-ized ISN in the outgoing SYN/ACK.

    Do you say "more quickly" because state table exhaustion takes some time, while interrupt/CPU issues can be switched on and off instantly?
  33. The original article has as it's premise that firewalls were poor security devices whether stateful or stateless.
    This is correct as the security problem is not solved in the network. It is like debating which is better a sour egg or a rotten egg.
  34. The reason the firewall will fall over with SYN-cookies is still due to state-table exhaustion from the incoming traffic; and as the firewall is doing more (iatrogenic) things with syn-cookies, it'll fall over even faster. Apologies for being unclear.
  35. Implementing access policy in terms of ports/protocols in stateless ACLs on hardware-based routers/layer-3 switches does in fact provide useful security value, as long as it's followed up by implementing server OS/app/service hardening BCPs. And of course, a poor layer-7 architecture can render anything just about indefensible.

    The network also has a role to play in terms of mitigating DDoS attacks via reaction mechanisms such as S/RTBH and/or IDMS.

    Availability is an area in which the network can contribute, but layer-7 must be designed, implemented, and operated properly, or what's done in the network doesn't matter.
  36. Coming in late on this; blame my vacation.

    The environment I protect is an enterprise environment, with a mix of off-the-shelf and custom-written software: Ivan's dynamic hodgepodge.

    Stateful firewalls were the answer to a failure of application *and* operating system authors to properly secure their code. These machines, hence, could not be trusted to manage their own connections properly, and so a separate purpose-built appliance was needed.

    It really comes down to economics. The marketplace (or the bazaar) has largely proven that we are unwilling to spend more time & money on software to ensure it is well-audited, free of security holes. We collectively prefer new features in our software.

    Anyway, the stateful firewalls that I prefer give me everything I need: centralized management of a well-organized security policy. (Not to mention fulfillment of the auditors' requirement that we have such a firewall in place.)

    Moreover, I'm not convinced that the app & OS authors have gotten any better at writing their networking code, code that is perhaps not shielded by a stateless ACL.

    DDOS is also less of a concern for me. I don't have a high profile, and to date I've never been attacked in that manner. It may well happen someday, and if it does, I've got BGP and stateless ACL tools to deal with it.

    As Ivan wrote: different data centers, different solutions.

    -Brian
  37. Hello,

    Thank you for great post.

    I am interesting what are you thinking about statefull firewall inside enterprise data center, where all resources are owned by enterprise, statefull firewall between different tears of web application, and stetefull firewall between different application groups, lets say there is MS environment where every MS server has joined AD, and to segment these different application servers with stetefull firewall ?

    Thank you,
    Giorgi
    Replies
    1. In many cases you don't need stateful firewalls, and some enterprises have already realized that. I described some of the options in this case study:

      http://www.ipspace.net/Replacing_the_Central_Firewall
Add comment
Sidebar