Build the Next-Generation Data Center
6 week online course starting in spring 2017

Packet- and Flow-Based Forwarding

I don’t like to correct my friends in public, but if someone says “I still believe that flow-based technologies will exceed the capabilities of packet-based technologies” (see Network Break 53), it’s time to revisit the networking fundamentals.

What is a flow?

According to Wikipedia (but what do they know…):

In packet switching networks, traffic flow, packet flow or network flow is a sequence of packets from a source computer to a destination, which may be another host, a multicast group, or a broadcast domain.

But maybe there’s something beyond packet-switching networks? Sure, you have three fundamental options:

There are no flows (apart from bit streams) in frequency- or time-division multiplexing, so we’re left with statistical multiplexing, and if we call whatever we’re switching “packet” every non-(F|T)DM system is a packet-switching system.

It’s all packet-based

Every single device operating at layers 2-4 (TDM and FDM work on layer 1) that I’ve seen in my life does the same thing:

  • Receive a packet;
  • Inspect the packet header and extract the relevant fields;
  • Perform forwarding table (or TCAM) lookup to find the output port(s);
  • Send the packet to output queue(s).

I still haven’t seen an OpenFlow-based or any other flow-based system that would do anything fundamentally different.

Flow-based technologies that are envisioned today thus cannot exceed the packet-based technologies because every flow-based system switches packets, and I sincerely hope the claim I quoted above wasn't meant to be along the lines of “virtual circuit switching will exceed the capabilities of connectionless switching”, because we’ve seen too many times how well that worked.

20 comments:

  1. And of course we all know why (virtual) circuit switching has been replaced with packet switching technology :-) To me, flows seem synonymous with virtual circuits.

    ReplyDelete
  2. First I haven't heard the podcast but from reading your post:
    It sounds a little to me like there's a confusion about the definition of a flow. Is your definition of a flow(sequence of similar packets from a to b) the same as your friend?

    Keep up the awesome work simplifying all these sometimes complicated topics. (should be posted in your Thank you-post)

    ReplyDelete
    Replies
    1. Hi Joe,

      Regardless of what definition one takes, flow-based forwarding is always packet-based. Writing a follow-up blog post as we speak...

      Best,
      Ivan

      Delete
    2. Yeah, usually when people say this, what they mean is "generalized packet-based forwarding (using e.g. OpenFlow)" is more powerful than "IEEE/IETF standard destination-based packet forwarding". Although whether this is actually true is debatable.

      Delete
  3. Hi Ivan - I thought the main difference is that flow-based forwarding can use more than just the destination IP address to make forwarding decisions, based on attributes such as source ip, source port, etc?

    ReplyDelete
    Replies
    1. The old grunts call that capability PBR, and it's been available on most traditional boxes for years. In any case, we're always switching packets, regardless of which header fields we use to make the forwarding decision.

      Delete
    2. What "exceed the capabilities" explained? Let's talk about details, not general ideas with no meaning. So it is interesting what does the "exceed the capabilities" statement mean.
      I have not heard this podcast...

      Delete
    3. Yeah, I would like to hear that explanation as well ;) Unfortunately, the whole thing was a lot of opinions and almost no data points.

      Delete
  4. Isn't a flow just an abstraction to think about or manipulate a series of related packets? Is OpenFlow a misleading name, isn't it a packet to flow mapper and configurator? I thought OF was a new standard method of configuring packet forwading rules instead of using mac learning or flooding or dynamic routing protocols.

    ReplyDelete
    Replies
    1. Yeah, OpenFlow is a totally misleading name, it's just a different way of downloading existing forwarding rules into forwarding hardware. There's nothing that could be called "flow-based forwarding" in OpenFlow, apart from the capability to match on fields other than destination L2/L3 addresses (aka PBR :D).

      Delete
  5. I submit that the real difference between packet and flows on the forwarding plane is that flows require the forwarder to be consistent amidst changes to the lookup table, whereas packets don't.

    A router is flow agnostic and doesn't need to respect the flow when deciding which next-hop to assign to a packet to adjust to, for example, changes in the lookup table that happen between packets, whereas a load-balancer does need to arrive at the same next-hop decision over and over again for each packet in the flow or it risks breaking the flow.

    And this is more than PBR because it creates a unique entry for each flow, all of which are a match for the same PBR rule, allowing flow A to go to next-hop 1 and flow B to go to next-hop 2.

    This is all still packet switching, but it is much more granular matches, which allows for much more granular actions and those actions can, because of the awareness of the flow context, do things that require the maintenance of state, particularly the state between a request and a response. When you do that at L4 you have a stateful firewall. When you do that at L7 or with the payload of L7, you have a load-balancer. When you do it because you really wish your packet switched network was a voice circuit or a virtual wire or a video channel you have a problem.

    That doesn't absolve the original quote of it's shame, and in fact, this kind of network state is both very complicated to manage well and applicable to a shrinking number of must-use scenarios if sane technological decisions about things like ipv6 and network identity are made; we shouldn't be artificially increasing them, because they should be naturally decreasing over time.

    (Of course, that quotes like that are not the consequence of sane technological decisions.)

    ReplyDelete
    Replies
    1. You got the gist of my upcoming blog post ;))

      However, there's an additional detail: while a flow-based forwarder treats all packets of the same flow in the same way (which might include always sending them toward the next hop), it still has to respond to changes in the forwarding table (for example, when the next hop becomes unreachable or is no longer a valid next-hop for the destination).

      Delete
    2. For example flowlet load balancing takes into account not only PBR rules (flow 'definition') bit also time to reach the 'merging' point before switching to another path. Today the devices are so complicated so the naming can be misleading and sparks only the academic discussion;).

      Details matter.

      Delete
    3. Flowlet implementations I've seen so far are way more simplistic and look at nothing more than whether a certain output bucket is empty, the only exception being Brocade VDX (which doesn't do flowlet load balancing but very smart per-packet load balancing).

      Pointer to a more complex implementation is highly appreciated.

      Delete
    4. Many flow-informed devices forgo next-hops/gateways entirely and instead set the outbound interface and destination MAC address based on incoming traffic. This is common practice for firewalls, load-balancers (F5 "Auto-LastHop"), and NAS (NetApp "FastPath" and EMC "IP Packet Reflect"). This ARP-be-damned forwarding can lead to very desirable steady-state forwarding behavior, but it's a liability to planned & unplanned changes since each device has its own idea of how to deal with topology changes. It can also affect capacity planning since link utilization doesn't follow what one would expect from RFC-compliant devices.

      Regardless, these fancy-pants devices are still shuffling packets to output queues.

      And to Ian's point that routers/switches are flow-agnostic: try turning on per-packet ECMP, or send fragmented traffic over a L4-hashed LAG, and see how that agnosticism plays out ... :)

      Delete
    5. Ivan pointed out that flow-based decision/forwarding is PBR to old-school network folks. I skim over some of the stuff from Open forums regarding Openflow-based controls. It seems to me that the main difference is where you update the forwarding tables from. With PBR, a network guy configures a layer-3 device to make layer-4-7 decisions, whereas with OF, someone makes a selection at the app layer (whether by a portal or by programming within an app) which propagates to the layer-3 device. If you read the cheerleader fluff of openflow proponents, the supposed advantage is that you can initiate a flow from one place and it will propagate across many L3 devices as long as they are all compliant. If you do this from the network side, obviously it would require touching many devices, and frequently. Again, I just skimmed and may not have it all straight. Regarding Ivan's original point, I don't see how flows could exceed packet-based forwarding unless they are referring to the outcome... I.E. a SD-WAN decision to switch traffic to alternate paths based on performance whereas packet-based forwarding would never do this?

      Delete
    6. @Autarch01: what SD-WAN does is just smarter load balancing based on link characteristics instead of routing protocol costs. No special sauce behind the curtain ;) See also http://blog.ipspace.net/2015/06/software-defined-wanwell-orchestrated.html

      Delete
  6. Thank you Ivan! You are just the right person to make that correction. That entire segment of 5 mins or so in the aforementioned was an exercise of pain. And it's sad because in the end this hurts more than benefit the dissemination of sdn/openflow, which is exactly the opposite of what their goal seem to be. But yeah, you know, BGP has bugs.... :-)

    ReplyDelete
  7. How about we call it "flow-based packet forwarding" and be done with it?

    ReplyDelete
  8. Hi Ivan,
    I am just a technical hands-on guy who doesnt undertand marketing so I would like to raise few basic points regarding openflow/sdn topic, which at this moment I dont have clarity on. Would you be able to comment on them:

    Forwarding-table size:
    At the moment, local CP (RIB) gets downloaded to FIB and we get packet forwarding on a router. If we start evaluating too many fields (PBR) and (assume) are able to push these policies to the FIB - what would become of the FIB table size ?? Wouldnt an open-flow based switch go through the same scaling issue on its forwarding level if each flow was to be treated differently on the device ??

    Out-of-band channel:
    If every device talking to a centralized control plane uses an out-of-band channel to talk to the openflow controller, isnt this a scaling concern?
    And if devices dont use out-of-band channel to talk to the openflow controller, what will happen if a core link goes down affecting device <> controller communication? is it expected that the FIB of the device would already have a pre-programmed "alternate" or "backup" path computed and downloaded into it to be able to reroute packets?
    Are we then talking about IP-FRR (which has topology restrictions) or LFA/RLFA ?..
    what do OF/SDN standards propose to mitigate this issue
    Also, the response time from centralized controller for network events would not be the same as a local CP on the device

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.