Do You Use SSL between Load Balancers and Servers?

One of my readers sent me this question:

Using SSL over the Internet is a must when dealing with sensitive data. What about SSL between data center components (frontend load-balancers and backend web servers for example)? Does it make sense to you? Can the question be summarized as "do I trust my Datacenter network team"? Or is there more at stake?

In the ideal world in which you’d have a totally reliable transport infrastructure the answer would be “There’s no need for SSL across that infrastructure”.

On planet Earth I wouldn’t worry that much about the networking team but more about servers from multiple applications sitting in the same L2 subnet (ideal scenario for an intruder trying to move laterally).

Also, I would ask the question "Is data encrypted at rest?" and when the answer is "NO" the next question should be "and why do you trust shared storage infrastructure more than shared network infrastructure?".

Finally, SSL within the data center usually protects the data exchanged between the load balancer and the web servers. While that data definitely deserves protection, there’s an even more important data stream: data exchanged between the web servers and the database servers. Are you encrypting the database connections? If not, why do you worry about encrypting web sessions?

The same reader also sent me a follow-up question:

If I understand you correctly, if servers are on the same unprotected L2 segment, encryption would then make sense. I suppose that if we go on that path, each server should have its own certificate (or at least each application). Otherwise, having same certificate on all wouldn’t give more protection as breaking in one server would give you access to the private key. Am I right?

If you’re serious about security, you should have a different certificate for each application. You should also use Ephemeral Diffie-Hellman key exchange.

Few Details

It’s really easy to decrypt a lot of TLS (formerly known as SSL) traffic if you have the server’s private key. Many TLS key exchange algorithms use a simplistic approach:

  • Client generates a random session key;
  • Client encrypts the session key with server’s public key;
  • Server decrypts the session key using server’s private key.

Once you have the server’s private key, you can decrypt the session keys, and subsequently all traffic exchanged with that server (that’s why you can decrypt HTTPS traffic with Wireshark).

Some TLS key exchange algorithms provide forward secrecy: it’s impossible to get the session key (and subsequently encrypted data) even if you have the private keys. These algorithms rely on Diffie-Hellman key exchange, more specifically on ephemeral DH where the session key is generated for every TLS session.

For more details, do spend some time reading the Wikipedia TLS article.

5 comments:

  1. Yes. I use SSH to manage my internal devices as well. What's the difference?
    Encrypt everything. Crypto is cheap and fast now, I don't see why you wouldn't.
  2. SSL doesn't make your internal network more reliable, it makes it less reliable because you are now susceptible to SSL issues in addition to TCP issues, IP issues, and network issues. HOWEVER, SSL makes your internal network more secure. If a Bad Guy gets into your data center somehow, but is not in your servers or load balancers yet, then SSL protects your data from the Bad Guy. Your Bad Guy could be a DC staff member.
    So it depends on if your DC is trustworthy or not. If not, then definitely use SSL
  3. The bad guys are already in your network. If you think that they aren't then you are most likely deluding yourself. Building things and then not being able to take measurements means that you are operating at least in part by faith. How many days so your systems wait to be patched? Do you use configuration management to catch configuration drift? Do you at least audit configuration drift? The things that actually cause breaches are slow patching tempo and administrative shortcuts. Yes the NSA can bust a buffer overflow in some device. Most large breaches cone from one of your admins taking a shortcut to meet a crazy arbitrary deadline and then going home before "fixing" the shortcut. (On a system that can only be patched once a quarter so that no one gets yelled at.)

    We need to stop pretending that we are doing this well.
  4. More and more apps do SSL by default and it's getting trickier to turn off redirects. So as a matter of practicality, we do SSL on the back end. The security is a nice bonus, but it's really so we can stick to standard/default app set up to get stuff done faster.

    It's also worth mentioning that we pay one (actually small) fee for unlimited certificates and can generate one in about 5 minutes. Our entire infrastructure team is pretty well practiced in getting real SSL certs generated and installed.
  5. In my experience attacks at app level can go hidden a lot easier with encryption turned on end-2-end. That's why, when I was still a network engineer we decrypted at the SLB (ADC if you entered networking around 2006) - sent traffic through a FW/IDS/AMP (directly or call-out) - sent it back to the SLB/ADC and re-encrypted with a weaker cipher if needed or left it in clear if somehow possible. It's amazing how many designs I've seen where TLS/SSL across a FW was deemed secure enough. Security is always a holistic affair and I'm no security guy by any stretch of the imagination.
Add comment
Sidebar