The fundamental principle underlying OpenFlow and Software Defined Networking (as defined by Open Networking Foundation) is the decoupling of control- and data plane, with data (forwarding) plane running in a networking device (switch or router) and control plane being implemented in a central controller, which controls numerous dumb devices. Let’s start with the basics: what are data, control and management planes in a network device?

Management, Control, and Data Planes in Network Devices and Systems

Every single network device (or a distributed system like QFabric) has to perform at least three distinct activities:

  • Process the transit traffic (that’s why we buy them) in the data plane;
  • Figure out what’s going on around it with the control plane protocols;
  • Interact with its owner (or NMS) through the management plane.

Routers are used as a typical example in every text describing the three planes of operation, so let’s stick to this time-honored tradition:

  • Interfaces, IP subnets and routing protocols are configured through management plane protocols, ranging from CLI to NETCONF and the latest buzzword – northbound RESTful API;
  • Router runs control plane routing protocols (OSPF, EIGRP, BGP …) to discover adjacent devices and the overall network topology (or reachability information in case of distance/path vector protocols);
  • Router inserts the results of the control-plane protocols into Routing Information Base (RIB) and Forwarding Information Base (FIB). Data plane software or ASICs uses FIB structures to forward the transit traffic.
  • Management plane protocols like SNMP can be used to monitor the device operation, its performance, interface counters …
Management, Control, and Data Planes in a Router

Management, Control, and Data Planes in a Router

The management plane is pretty straightforward, so let’s focus on a few intricacies of the control and data planes.

We usually have routing protocols in mind when talking about Control plane protocols, but in reality the control plane protocols perform numerous other functions including:

  • Interface state management (PPP, LACP);
  • Connectivity management (BFD, CFM);
  • Adjacent device discovery (hello mechanisms present in most routing protocols, ES-IS, ARP, IPv6 ND, uPNP SSDP);
  • Topology or reachability information exchange (IP/IPv6 routing protocols, IS-IS in TRILL/SPB, STP);
  • Service provisioning (RSVP for IntServ or MPLS/TE, uPNP SOAP calls);

Data plane should be focused on forwarding packets but is commonly burdened by other activities:

Data plane forwarding is hopefully performed in dedicated hardware or in high-speed code (within the interrupt handler on low-end Cisco IOS routers), while the overhead activities usually happen on the device CPU, often in userspace processes – -the switch from high-speed forwarding to user-mode processing is commonly called punting.

Typical Architectures

The management and control planes are typically implemented in a CPU, while the data plane could be implemented in numerous ways:

  • Optimized code running on the same CPU;
  • Code running on a dedicated CPU core (typical for high-speed packet switching on Linux servers);
  • Code running on linecard CPUs (example: Cisco 7200);
  • Dedicated processors (sometimes called NPUs)
  • Switching hardware (ASICs);
  • Switching hardware on numerous linecards.

Based on the capabilities of the switching hardware, a device might run some simple time-critical control-plane protocols (example: BFD) in hardware or on NPUs.

Processing of Inbound Control-Plane Packets

In most router implementations, the data plane receives and processes all inbound packets and selective forwards packets destined for the router (for example, SSH traffic or routing protocol updates) or packets that need special processing (for example, IP datagrams with IP options or IP datagrams that have exceeded their TTL) to the control plane.

Management ports on some devices, including data center switches, are connected directly to the control-plane CPU and thus bypass the switching ASIC. For more details read Building Secure Layer-2 Fabrics)

The control plane might pass outbound packets to the data plane, or use its own forwarding mechanisms to determine the outgoing interface and the next-hop router (for example, when using the local policy routing).

Processing of Inbound and Outbound Control-Plane Packets

Processing of Inbound and Outbound Control-Plane Packets

Regardless of the implementation details, it’s obvious the device CPU represents a significant bottleneck (in some cases the switch to CPU-based forwarding causes several magnitudes lower performance) – the main reason one has to rate-limit ACL logging and protect the device CPU with Control Plane Protection features.

Latest blog posts in OpenFlow Basics series

6 comments:

  1. From security perspective, when we look at a core router, if the control plane is well protected, will it really be possible to trigger some specific back-door (let's assume the extreme situation that router has government|vendor placed back-door) through data plane (transit traffic, things like some craft packets shutting the router down or starting to record some specific traffic and send them over to some receiver)? would like to know your take on that.
    Replies
    1. If you believe there's a back door being buried deep in network device OS, it's always possible to trigger it with an innocuous-looking packet - something as simple as ping with a carefully crafted payload.

      Also, if I would be a device vendor, I wouldn't allow you to filter out my backdoor triggers with control plane protection, would I? ;)

      (* For everyone else: this is PURE UNFOUNDED SPECULATION *)
  2. Shouldn't ICMP be more on the control-plane area?
    Replies
    1. No, it's an inband response to a data-plane activity.
  3. Ok, but it's still a control-plane generated response.
    Also, what about ICMPv6?
    Replies
    1. I think you're confusing the ASIC/CPU split with data/control plane split.

      Even though punted traffic gets limited by what some vendors call CoPP, it doesn't transform punted traffic into control-plane protocols.
  4. I'm actually talking about router responses to punted traffic, not the punted traffic itself.

    In my mind i visualize the idea of control-plane as a layer in the network device, where various processes run and these processes handle everything destined to the router itself. Maybe routing protocols have the control-plane functionality built-in, but still there are cases where for intermediate devices they are simple data-plane traffic.

    After all C=Control in ICMP.

    Quoting from yourself (http://wiki.nil.com/Control_and_Data_plane):

    In most router implementations, the data plane receives and processes all inbound packets and selective forwards packets destined for the router (for example, routing protocol updates) or packets that need special processing (for example, IP datagrams with IP options or IP datagrams that have exceeded their TTL) to the control plane.
  5. Nice blog - I was asked to explain "SDN" to a DevOps meetup and ran them through this material as a reference so they could have context. The PPT for that is located here: http://www.slideshare.net/JeremySchulman/2014-0219-dev-ops-sdn-v1
    (disclaimer: I am by no means any expert on SDN)
  6. If separating the control plane and the data plane, what's the best OS for the control plane and what for data plane: Microsoft Windows, Ubuntu, Red Hat Enterprise Linux or variant, FreeBSD or variant, or NetBSD or variant? Ever heard of IX? That would make a great data plane OS while a Linux distribution be for the control plane. See https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-belay.pdf and https://courses.cs.washington.edu/courses/cse551/15sp/papers/ix-osdi14.pdf. Should Windows be for the management plane?

    Replies
    1. Most recently-developed solutions I've seen use one of the Linux distributions for the control plane.

      Never heard of IX (thanks for the links), so I cannot say whether it ever progressed beyond an interesting academic concept.

    2. Actually, Arrakis is based on Barrelfish and IX is based on Linux. They are counterparts.

Add comment
Sidebar