I don’t need no stinking firewall ... or do I?

Brian Johnson started a lively “I don’t need no stinking firewall” discussion on NANOG mailing list in January. I wanted to write about the topic then, but somehow the post slipped through the cracks (thank you, Pavel, for a kindly reminder!) ... and I’m glad it did, as I’ve learned a few things in the meantime, including the (now obvious) fact that no two data centers are equal (the original debate had to do with protecting servers in large-scale data center).

First let’s rephrase the provocative headline from the discussion. The real question is: do I need a stateful firewall or is a stateless one enough?

Stateless firewalling is implemented quite easily with router access lists and thus works at line speed in most routers. Stateful firewalls implemented in routers are usually suitable for low-speed remote offices; you should use dedicated firewall devices in data centers.

Technology issues

Stateful firewall is the only option if you’re trying to tightly protect applications that use dynamic port numbers, including everything from peer-to-peer applications (including SIP) to RPC-based applications (let’s try not to call them broken ... how about unpredictable applications).

You can limit the dynamic port range for some of these applications and allow all ports in that range through the firewall ... while hoping that some other service on your server won’t grab one of those ports and expose itself unnecessarily.

If your applications use only well-known fixed port numbers (let’s call them fixed-port applications), you don’t have to inspect the application data stream and can match the applications with access lists; stateless solutions seem appropriate.

However, some stateful firewalls can add value even in fixed-port environments: they can delay the commitment of server resources to TCP sessions with TCP SYN cookies (also available in numerous server operating systems) and check the validity of TCP sessions.

You might think that there are no vulnerabilities left in TCP that could be exploited. A long while ago, everyone thought it was impossible to establish one-way spoofed TCP sessions even though there were known vulnerabilities in TCP sequence number generation ... until Kevin Mitnick proved them wrong.

Last but not least, stateful firewall in front of a server can block TCP fingerprinting attempts. Sometimes you simply don’t want the attacker to know too much about your infrastructure.

Design choices

There are two extremes you can be facing in a data center:

Unified large-scale infrastructure using fixed-port applications, including Google, Yahoo, Facebook, Twitter and a few others. You would expect to see from tens to thousands of almost-identical servers with standardized and identically-configured operating system in these environments. It’s quite manageable to harden and patch these servers and combined with fixed-port applications these environments usually offer, it makes perfect sense to be satisfied with router ACLs (and that was the case the proponents of “get rid of the firewall” line of thinking were promoting in the NANOG discussion).

Dynamic hodgepodge of servers, operating systems and application encountered in a usual enterprise data center. Server hardening is mission impossible (as you have so many different operating systems and/or versions of the same operating system), patching is the responsibility of individual server administrators (and they might not even be a unified team) and the applications are a nightmare from the security perspective. For example, early Outlook Web Access was easy to firewall but even then some people advocated putting OWA in the inside network (I wish I were blogging at that time, I would have had so much fun debunking that article). I was not able to find anything from Microsoft telling me how to configure my firewall to support OWA for Exchange Server 2010; the cynical response I got from a (non-Microsoft) security engineer a few days ago was “nobody can figure out which ports it uses”.

Now what?

Obviously you’re the only one who can decide where your Data Center environment is and which measures you’d like to implement, but as a generic rule, the more exposed (and the worse managed) a server is, the more you should lean toward a stateful firewall in front of it. Most enterprise data centers make heavy use of stateful firewalls and that’s also what we’re recommending to most of our customers.

To get insight into typical Data Center architectures (including load balancing and firewalling) and emerging DC technologies, watch my Data Center 3.0 for Networking Engineers webinar (buy a recording or yearly subscription). For an in-depth solution, you might want to consider the benefits of our Hypercenter architecture.

36 comments:

  1. Russell Heilling19 August, 2010 08:10

    Hi Ivan,

    A thought provoking post :)

    Server hardening may be "mission impossible" but it is still essential. Very rarely is every server provisioned in a separate broadcast domain, if you rely on a big-iron firewall for your security you are going to be left at risk from the zero-day that hits the machine next to you. Or even from the compromised administrator's laptop that connects without going through the firewall.

    I think there is an argument for defence in depth, putting a dedicated resource between you and your greatest attack vector to reduce your load is always going to be a good idea. If your risk is great it may even be worth adding an additional firewall level from a second vendor in series. If a stateless firewall can capture the largest threats without blocking valid content then it may be the right choice, either on its own, or in addition to the stateful firewall.

    Of course managing layered firewalls like this comes with its own configuration management issues that could put you at more risk of outage than running without the firewalls..

    Sure some apps are hard to put behind static ACLs, but equally there are plenty of examples of where stateful firewalls get their application code wrong and can break networks just as badly as a badly written ACL (an example that immediately springs to mind is the SMTP code in early PIXes. For years the first command I would enter in a default PIX config was "no fixup smtp". Memory leaks in SIP ALGs is a more contemporary and widespread issue.)

    Discounting endpoint security as "mission impossible" says to me that we need to find a new way to handle it because we sure as hell can't give up on it. Of course I don't have an answer for *how* we deal with it or I would be running away to the patent office right now ;)

    ReplyDelete
  2. Ronald Bartels19 August, 2010 08:57

    Of course no one needs a stinking firewall, all you need is NAT.

    ReplyDelete
  3. mokum von Amsterdam19 August, 2010 13:44

    Ofcourse we need statefull firewalls AND stateless router ACL's in front of everything. It's a great idea to implement an additional source of issues in front of an OS with issues carrying apps with issues. If not in obscurity & complexity, where would the fun in our jobs be?
    More proxies, more high availability, more active active is my motto, and I have yet to meet a vendor who does not agree 8-)

    ReplyDelete
  4. Roland Dobbins19 August, 2010 15:09

    Stateful inspection makes zero sense when every incoming connection is unsolicited - as is the case with Web servers, DNS servers, et. al.

    Furthermore, even the largest firewalls available have state-tables which are easily exhausted either by legitimate flash crowds or by layer-7 attack traffic crafted in such a way as to appear perfectly legitimate, passing all firewall rules and 'inspectors'. Stateful firewalls are DDoS chokepoints, they can withstand far fewer sessions than the hosts behind them, and they always fall over during DDoS attacks.

    And of course, the firewalls themselves are merely special-purpose computers running code - and there are bugs in that code, as the list of security flaws and patches for every commercial firewall vendor demonstrates. Add this to the fact that the code written for these devices receives much less scrutiny than OS/application/service code, and one can see that in addition to the availability threat they pose, firewalls also greatly broaden the attack surface and potentially provide obscure vectors for attackers to compromise these blackbox devices.

    None of the large-scale Internet properties with which readers are familiar put stateful firewalls (or so-called 'IPS') in front of their servers - at least, the ones which stay up and running.

    Stateful firewalls have no place whatsoever in front of servers, period. There's no state to inspect, and they greatly reduce the security posture of organizations which make the mistake of placing them in front of servers.

    Stateless ACLs in hardware-based routing/switching platforms which can handle millions of packets-per-second in hardware are the BCP for enforcing communications policy for servers.

    This NANOG presentation succinctly demolishes the myth that firewalls (and other stateful inspection devices) should be placed inline in front of servers:

    http://www.nanog.org/meetings/nanog48/presentations/Monday/Kaeo_FilterTrend_ISPSec_N48.pdf

    ReplyDelete
  5. Roland, you bring up many good points! It is often difficult to find a nice balance between strong protection measures and resource utilization. Strong gates do not a castle make.

    ReplyDelete
  6. Roland Dobbins19 August, 2010 15:48

    For an example of how firewalls fall over during DDoS attacks, as well as a list of best current practices (BCPs) for protecting servers, see my APNIC presentation here:

    http://meetings.apnic.net/__data/assets/pdf_file/0010/13987/rokusaddosjul09preso.pdf

    http://webcast.apnic.net/meetings/28/apops-10-roland-dobbins.mov

    ReplyDelete
  7. Ivan Pepelnjak19 August, 2010 16:41

    Thanks for the comment, Roland ... it's been a while since we last "spoke" and it's nice to hear from you.

    Obviously I can't possibly disagree with anything you wrote; your facts are absolutely correct. However, you're focusing at a particularly exposed and demanding market segment: large content providers. As I wrote in the post, they are one extreme of the whole spectrum, the other one being a mid-sized enterprise (don't want to go down to SMB) with very mixed server environment and application requiring dynamic ports.

    While the lessons learned in high-volume environment are definitely worth considering (and as I wrote, they are best served by router-based ACLs), not everyone has the same amount of knowledge, the team size that these content providers enjoy, the same requirements or the same exposure. These lessons are thus not directly applicable to other environments, as they can sometimes do more harm than good (see the other end of the spectrum).

    What you're talking about are F1 racing cars of the content business. That's where the technology is stressed, where we learn how to improve it and how to design it better (one can only hope the same knowledge transfer would work in the IT industry); but the methodologies used there cannot be applied directly to a commodity car (which will also not be exposed to most of the stresses an F1 racing car is).

    ReplyDelete
  8. Ivan Pepelnjak19 August, 2010 16:53

    Dear Guest,

    as you might have realized, the readers of my blog vary significantly in their exposure to particular technologies. While Roland questions the validity of my arguments, you're asking "what's the point". Obviously you need someone to explain the basics of stateless versus stateful firewalls. The articles in Wikipedia are not bad and look to be the first hit on Google when you're searching for "stateful firewall" (they refer to stateless firewalls as "packet filters") ...

    http://en.wikipedia.org/wiki/Stateful_firewall
    http://en.wikipedia.org/wiki/Packet_filter

    ... if you need more, just let me know; I might be able to help you.

    Now, Bro, how about some common decency ... like using your name like those that argue with the content of my post ... or politely telling me "hey, Ivan, I would appreciate if you could explain the difference between stateful and stateless firewalls, I was not able to find a decent document anywhere".

    ReplyDelete
  9. Your views intrigue me and I would like to subscribe to your newsletter.

    Although, what do you do to protect against TCP SYN DoS attacks?

    ReplyDelete
  10. Sweet - I'm going to tell my Cisco SE that we no longer want to purchase those ASA5580s, under the advisement of a CSCO solutions architect. :-P

    ReplyDelete
  11. Ivan Pepelnjak19 August, 2010 19:20

    Your network, your risk, your decision :-P

    ReplyDelete
  12. Pavel Skovajsa19 August, 2010 19:40

    Notice that Roland does not say that Cisco should stop selling/recommending firewalls.
    Roland says Cisco should stop selling/recommending stateful firewalls which are put infront of servers which receive unsolicited connections.

    In other situations, stateful firewalls are a perfect fit, f.e. permit inside -> outside traffic, while blocking the outside -> inside traffic.

    Put another way - what is the advantage of putting of stateful firewall in front of public facing DNS server?

    ReplyDelete
  13. Ivan Pepelnjak19 August, 2010 19:41

    If you have numerous servers in a multi-tenant environment, there's always PVLAN or layer-3 isolation with P2P interfaces (assuming your aggregation gear knows what's above L2).

    In an enterprise environment where the number of connections between various services grows exponentially, PVLANs could become a nightmare.

    As for server hardening: I'm far away from saying it can't be done; I'm just saying that not everyone knows how to do it ... unfortunately.

    ReplyDelete
  14. Pavel Skovajsa19 August, 2010 19:46

    ....nothing, Roland operates under assumtion that servers should protect themselves via implementing TCP cookies by default....moving the problem into the server space (which is actually where it belongs, but we spoiled them)

    ReplyDelete
  15. Ah, if only that was an acceptable answer for management.

    ReplyDelete
  16. My network, my risk, state auditor's decision :'(


    Being in higher ed, I really can't imagine NOT having a firewall. Now, NAC...whole different story.

    ReplyDelete
  17. Ivan,

    You mentioned TCP SYN cookie support in stateful firewalls. Frankly, I think this is an awful feature. It requires the firewall to intercept/terminate the TCP session, and can lead to lots of crazy failure scenarios. Imagine, for example, if the server (or an intermediate router) doesn't know the path back to the client. TCP 3 way handshake appears to succeed, but then the application never responds. What has really happened is that the client has never established 2-way communication with the server. ...and for what benefit? TCP SYN cookie support is easy to implement. A server that doesn't have it probably shouldn't be facing the wrath of the Internet.

    Also, about TCP fingerprinting... You said that a stateful firewall *can* prevent fingerprinting. I'm not sure that's true -- you'd need much more than just state awareness to prevent fingerprinting. You'd need to strip funny TCP options, rewrite sequence numbers... You're getting much closer to a full-blown TCP proxy than you are a stateful firewall. Not that it's a big deal (netcat can do it), it's just not something that I think of as a firewall function. Do firewall vendors market this as a feature?

    ReplyDelete
  18. Alberto Caldas20 August, 2010 05:52

    Talking about stateful firewalls, would SBCs like the Cisco UBE be considered specialized SIP firewalls? I assume they must be stateful even though SIP uses a well known udp port 5060 it is only used to set up audio or video sessions that can use any dynamically assigned udp ports.

    I haven't used the CUBE personally, but I was on a project where we were using a dedicated SBC appliance from a top SBC vendor but we replaced it with a ASA because we also wanted to control some other protocols like FTP, TFTP, etc ...

    ReplyDelete
  19. Roland Dobbins20 August, 2010 08:17

    Concur 100%. SYN-cookies should be handled on the hosts themselves.

    Fingerprinting is irrelevant, anyways. If one really wishes to obfuscate at that level, there are kernel patches which perform this functionality.

    ReplyDelete
  20. Roland Dobbins20 August, 2010 08:19

    Respectfully (and good to 'speak' with you, too, heh), these issues are even *more* important for the smaller fry. They get absolutely crushed with even legitimate flash crowd traffic, much less direct or collateral DDoS.

    Putting a stateful inspection device at a point in the topology *where there is no state to inspect in the first place* doesn't make sense at any scale. It is insane, IMHO.

    ReplyDelete
  21. Roland Dobbins20 August, 2010 08:21

    Actually, it's a bit more nuanced than that.

    Firstly, yes, TCP/IP stacks should be hardened.

    Secondly, S/RTBH is a great, low-cost mitigation mechanism for all kinds of packet-flooding attacks.

    Thirdly, intelligent DDoS mitigation systems (IDMS) can handle layer-3 through layer-7.

    [Full disclosure; I was heavily involved with the former Cisco (nee Riverhead) Guard IDMS when I was at Cisco, and now work for a vendor of IDMS systems.]

    ReplyDelete
  22. Roland Dobbins20 August, 2010 08:28

    To clarify, I'm no longer with Cisco - but I was 'infamous' in certain quarters within Cisco for espousing these very same views when I was there, so my strong recommendation against stateful firewalls in front of any server, anytime, anywhere has been consistent over the years.

    ;>

    And, Pavel, you're entirely correct - stateful firewalls have an important role to play in separating enterprise client access networks, home networks, and other endpoint networks from 'The Internet', in order to keep unsolicited connections off machines which are primarily workstations.

    But they shouldn't be wedged into broadband access networks due to scaling and near-infinite user application diversity, and they need to be removed from mobile wireless data networks, as they cause more problems than they solve (their presence on these networks is primarily a legacy of unfamiliarity with Internet-scale connectivity requirements at the time many of these networks were initially constructed).

    ReplyDelete
  23. Roland Dobbins20 August, 2010 08:33

    Due to the pessimal communications models of grotesqueries such as SIP, H.323, et. al., things like SBCs and IP PBXes and so forth should be shielded from a policy perspective as best they can via stateless ACLs in hardware, their inherent stateful inspection mechanisms must be enabled, and operators must be prepared to maintain their availability in the face of attack via mechanisms such as S/RTBH, flowspec, and/or IDMS.

    ReplyDelete
  24. Roland Dobbins20 August, 2010 08:36

    NAT in front of servers is even worse than stateful firewalls - guaranteed collapse in the event of even a minimal flash crowd, much less directed or collateral DDoS.

    ReplyDelete
  25. Roland Dobbins20 August, 2010 08:36

    Preach it, brother!

    ;>

    ReplyDelete
  26. Roland Dobbins20 August, 2010 08:38

    Thanks for the kind words, and I concur with your cited metaphor 100%!

    ;>

    ReplyDelete
  27. Roland Dobbins20 August, 2010 08:42

    Concur, the whole concept of trusting endpoint self-assessment reporting is inherently flawed - I argued heavily against it on this basis when it was first posited, and it's with no pleasure that I note that events have borne out this view.

    In terms of being forced to deploy stateful firewalls by various subspecies of auditors - the nonsensical PCI DSS 'Web application firewall' requirement can be met via mod_security. In other scenarios, education in order to help the auditors get past the 'firewalls = security' disinformation is the long-term solution.

    ReplyDelete
  28. Ivan Pepelnjak20 August, 2010 13:45

    So, is it correct to assume that your main point is: "stateful firewalls are not good DDoS mitigation devices"? I can easily agree with that.

    ReplyDelete
  29. I guess that if you're going to ignore Roland, and install a stateful firewall in front of servers, then putting the (godawful) TCP SYN cookie mechanism in the firewall is helpful. It helps the firewall survive in a place where it doesn't belong.

    ...And I didn't mean to suggest that fingerprinting prevention was a feature worth asking for!

    I just meant to explain that stateful inspection doesn't suggest anti-fingerprinting in my mind. There are so many fingerprinting vectors that it'd be tough for a man-in-the-middle to stop them all, and mere "stateful inspection" stops none of them.

    ReplyDelete
  30. Roland Dobbins20 August, 2010 20:32

    No - my main point is that it makes zero sense to insert a stateful inspection device in front of servers, where there's no state to inspect in the first place, as every incoming connection is unsolicited. Stateless ACLs in hardware-based routers/layer-3 switches are the correct way to enforce network access policies.

    As a matter of considerable amusement, one large commercial firewall vendor has implemented a 'stateful inspection bypass' feature - meaning that they expect you to spend $100K USD on their 10gb/sec stateful firewall, then *disable the stateful inspection which was the justification for buying it in the first place*, heh. This is a tacit admission that stateful inspection has no role in front of servers, as there's no state to inspect.

    The DDoS vector and increase in the overall compromise attack surface these devices bring with them are very real threats, and are additional reasons for not deploying firewalls in front of servers; but even if these weren't concerns (and they're HUGE concerns), the fundamental illogicality of inserting a stateful inspection device in the topology where there is in fact no state to inspect is the larger point.

    ReplyDelete
  31. Roland Dobbins20 August, 2010 20:34

    Actually, the SYN-cookie feature will end up causing the firewall to die more quickly under duress due to processing/interrupt overhead and memory exhaustion.

    If one is forced to install firewalls in front of one's servers despite the obvious disadvantages and security risks of doing so, the firewalls and everything behind them should be protected via reaction mechanisms such as S/RTBH, flowspec, and/or IDMS.

    ReplyDelete
  32. With SYN cookie support on the servers, the firewall WILL die of state table exhastion in a SYN flood attack.

    With (godawful) SYN cookie support on the firewall, the firewall MIGHT die trying to proxy the incoming TCP handshakes. But it won't fall over because of state table exhaustion. The only risk is in generating the cookie-ized ISN in the outgoing SYN/ACK.

    Do you say "more quickly" because state table exhaustion takes some time, while interrupt/CPU issues can be switched on and off instantly?

    ReplyDelete
  33. Ronald Bartels21 August, 2010 19:07

    The original article has as it's premise that firewalls were poor security devices whether stateful or stateless.
    This is correct as the security problem is not solved in the network. It is like debating which is better a sour egg or a rotten egg.

    ReplyDelete
  34. Roland Dobbins23 August, 2010 11:35

    The reason the firewall will fall over with SYN-cookies is still due to state-table exhaustion from the incoming traffic; and as the firewall is doing more (iatrogenic) things with syn-cookies, it'll fall over even faster. Apologies for being unclear.

    ReplyDelete
  35. Roland Dobbins23 August, 2010 11:38

    Implementing access policy in terms of ports/protocols in stateless ACLs on hardware-based routers/layer-3 switches does in fact provide useful security value, as long as it's followed up by implementing server OS/app/service hardening BCPs. And of course, a poor layer-7 architecture can render anything just about indefensible.

    The network also has a role to play in terms of mitigating DDoS attacks via reaction mechanisms such as S/RTBH and/or IDMS.

    Availability is an area in which the network can contribute, but layer-7 must be designed, implemented, and operated properly, or what's done in the network doesn't matter.

    ReplyDelete
  36. Coming in late on this; blame my vacation.

    The environment I protect is an enterprise environment, with a mix of off-the-shelf and custom-written software: Ivan's dynamic hodgepodge.

    Stateful firewalls were the answer to a failure of application *and* operating system authors to properly secure their code. These machines, hence, could not be trusted to manage their own connections properly, and so a separate purpose-built appliance was needed.

    It really comes down to economics. The marketplace (or the bazaar) has largely proven that we are unwilling to spend more time & money on software to ensure it is well-audited, free of security holes. We collectively prefer new features in our software.

    Anyway, the stateful firewalls that I prefer give me everything I need: centralized management of a well-organized security policy. (Not to mention fulfillment of the auditors' requirement that we have such a firewall in place.)

    Moreover, I'm not convinced that the app & OS authors have gotten any better at writing their networking code, code that is perhaps not shielded by a stateless ACL.

    DDOS is also less of a concern for me. I don't have a high profile, and to date I've never been attacked in that manner. It may well happen someday, and if it does, I've got BGP and stateless ACL tools to deal with it.

    As Ivan wrote: different data centers, different solutions.

    -Brian

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.