After I published the blog post describing how infrastructure cloud provides (example: AWS) might use smart Network Interface Cards (NICs) as the sweet spot to implement overlay virtual networking, my friend Christoph Jaggi sent me links to two interesting presentations:

• Can you still trust your network card
• The Jedi Packet Trick Takes over the Deathstar

Both presentations describe how you can take over a smart NIC with a properly crafted packet, and even bypass CPU on a firewall using smart NICs.

What Is a Smart NIC

If you’ve never heard about smart NICs, you might want to start with the chat we had with Or Gerlitz (Mellanox), Andy Gospodarek (Broadcom) and Jiri Pirko (Mellanox), and then look at the technical part of the Pensando presentation from Cloud Field Day 7 (you can safely skip the “we are so awesome” part).

Or in you’re short on time, here’s the crux of the story:

• Dumb NICs take packets from a circular buffer (TX ring) and send bits from those packets to the wire… or the other way round (in which case the circular buffer would be called RX ring).
• Smart NICs include everything a dumb NIC has (someone obviously has to transmit and receive the bits), but they can be programmed to munge the packets, including all sorts of crazy lookups and content manipulations. Most smart NICs are programmable, and include plenty of memory and a general-purpose CPU.

What Is the Problem?

We make mistakes. Sometimes the mistakes are burnt into hardware (see Pentium FDIV bug), most often they are hidden somewhere in software. History of IT security is littered with buffer overflows or code accepting inputs without any sanity check (see Heartbleed).

Then there are those nasty bugs that need a very specific sequence of events and very specific timing (see: Meltdown and Spectre… and researchers keep discovering all sorts of variants of exploits of speculative execution). Finally, even the management software that handles software upgrades and the like is vulnerable (see: Intel CSME vulnerability).

I think it’s fair to say that it’s only a question of time when we’ll see another catastrophic smart NIC exploit. Oops, we already did - see Broadpwn.

Amazon might be in pretty good shape with AWS Nitro. A cloud service provider has to care about security more than a typical enterprise vendor, and they claim to have provable security. As for smart NICs being used in enterprise servers, in particular those that can be programmed by the end-user, I have only one thing to say: expect some fun times.