A while ago I wrote a blog post about remote ND attacks, which included the idea of having /120 prefixes on server LANs. As it turns out, it was a bad idea, and as nosx pointed out in his comment: “there is quite a long list of caveats in all vendor camps regarding hardware in the last 6-8 years that has some potentially painful hardware issues regarding prefix length. Classic issues include ACL construction and TCAM specificity.”
One would hope that the newly-release data center switches fare better. Fat chance!
There are several interesting issues you might encounter with IPv6-enabled data center switches (other devices performing hardware layer-3 switching might exhibit similar behavior):
- IPv4 and IPv6 routing tables might share the same TCAM.
- IPv6 prefixes are four times longer than IPv4 prefixes, so you’d expect a switch with shared TCAM to handle four times as many IPv4 prefixes as IPv6 prefixes. If that’s not the case, tread carefully and dig deeper into the documentation.
A few real-life examples
ToR switches were never known for their huge table sizes (to be honest, they don’t need them if you have a good design), and some of them have dismal IPv6 tables. The worst I found: Juniper’s EX4500 with 1K IPv6 entries and Arista’s 7500 (the core switch) with 2K IPv6 entries. Good luck with your large-scale dual-stack deployment!
Then there’s Cisco’s Nexus 5500 with an interesting TCAM architecture: it can have up to 16K IPv4 or IPv6 routes, and 128 longest-prefix-match IPv6 entries. The fact that the number of IPv4 routes matches the number of IPv6 prefixes tells you a lot about the matching algorithm: either half the TCAM (or more) is empty with IPv4 routes or they’re doing an exact match on the top 64 bits of IPv6 addresses, which seems to be the case as there’s a separate entry for IPv6 LPM routes in the configuration limits document.
To rephrase: Nexus 5500 can have up to 16K /64 IPv6 prefixes and up to 128 non-/64 IPv6 prefixes. It does make perfect sense, assuming your data center uses /64 prefixes internally and a few summary routes (or default routing) toward the outside world or DC core.
Finally, there are loads of DC switches where the maximum number of IPv6 prefixes is half the maximum number of IPv4 prefixes, probably indicating that their TCAM matches only the top half of IPv6 addresses (and that installing /120 or /127 prefixes into these devices might be a Really Bad Idea). Unfortunately, many vendors are not as open and straightforward as Cisco is, and forget to mention these tiny little details in their documentation.
Have to mention that the Data Center Fabrics webinar contains IPv6, IPv4, MAC, ARP and ND table size information for data center switches from nine major vendors. You don’t want to know how many hours I’ve spent poring over datasheets, documentation and release notes.