The Lack of Historic Knowledge Is so Frustrating

Every time I’m explaining the intricacies of new technologies to networking engineers, I try to use analogies with older well-known technologies, trying to make it simpler to grasp the architectural constraints of the shiny new stuff.

Unfortunately, most engineers younger than ~35 years have no idea what I’m talking about – all they know are Ethernet, IP and MPLS.

Just to give you an example – here’s a slide from my SDN workshop. Anyone familiar with the technologies I listed on the slide immediately grasps the limitations of various SDN approaches.

How many people still remember those technologies? They are dead for a good reason, but unless we know what the reason is, we keep reinventing them after a decade or two.

Want a few more examples? I was trying to explain the principles of SAN and differences between FC and FCoE to a networking engineer a while ago. I started with B2B credits in FC and told him Fibre Channel uses exactly the same approach as hop-by-hop windows we had in X.25, with the same results… resulting in confused blank stare. Further down the conversation I said FCoE uses lossless Ethernet, which uses PAUSE frames, which are exactly the same thing as Ctrl-S and Ctrl-Q on an async link. Same result.

In short, the lack of historical knowledge I see in younger engineers is depressing, and it’s obvious when looking at many SDN ideas or even products that “Those who fail to learn from history are doomed to repeat it”, but I simply don’t know how to fix it – it’s so hard to motivate young people to learn about seemingly irrelevant stuff. Any ideas are highly appreciated.

However, there is something you can do – always ask why do we do things the way we do them and understanding the fundamental principles instead of focusing on the intricacies of the configuration commands. Study old technologies and try to grasp why we no longer use them, because RFC 1925 section 2.11 remains as relevant as ever.

47 comments:

  1. This totally makes sense, however I can understand engineers who don't want to spend time studying technologies they will probably never use.. The networking world is so wide that you have to select some techs to focus on..
  2. For us coming from the CCIE world, everyone was raging about why Frame Relay was kept for so long. While not used any longer in most parts of the world, it did teach the concepts of NBMA. Something that is critical when working with DMVPN. Also, FR has some similar concepts to MPLS, making the gap to learn MPLS a bit lower.

    While I've never touched Decnet, Appletalk or IPX (except on home PC) I try to keep aware of what existed in the past. Even I can see that history is repeating itself and many of the SDN startups are repeating the most basic mistakes of the past...
    Replies
    1. I've come to the conclusion that most academics are fundamentally incapable of and/or are unwilling to acknowledge the existence of prior art.
  3. A good resource to take a look about how was networking in the late 80´s/90`s is the Connexions - Interoperability Report : http://www.cbi.umn.edu/hostedpublications/Connexions/index.html
  4. None of our NOC guys know what ATDT means, even the ones that did come through tech support.

    (PSTN is still more reliable than 3G for OOB, assuming you test it often - anybody got a good automated test script?).
    Replies
    1. ATDT - priceless ;)) Thanks for bringing this up!

      Now that I think about that, I started with ATDP :D... and I'm positive there's AT command set hidden within every 3G modem (in the good old days you could use it to send SMS messages).
    2. ATDT is still alive!. Just setup a simple GSM/SMS based solution and used AT commands to control GSM modem. It's really coll that this has not changed over the years.
    3. Ah!! the joy of ATDT :) I lost a lot of valuable time troubleshooting quirky modem problems, serial printing over TCP/IP, duplicate station IDs in arcnet connected over dumb arcnet hubs, thin Ethernet cable problems.....

  5. As a 27 year old network (and systems, unfortunately) engineer, I am very interested in older/obsolete technologies. I think they're really cool and obviously still relevant even though no one uses them anymore. I do, however, not have the time to study them. At work I barely have time for current technologies, and outside of work... well let's just say I do my best to stay away from enterprise IT stuff. I care too much about my mental health ;)
  6. My son is studying computer science and during his networking classes he is overwhelmed by the number of technologies used in the past. For me it's easy to describe them because I started playing with ARCNET. BUT I am not surprised that someone might find it difficult (impractical) to study the past just because it is huge amount of knowledge...
    Replies
    1. Well, nobody became a networking expert in a single university class. It took most of us a decade or so ;)
  7. "We learn from history that we learn nothing from history."
    George Bernard Shaw
  8. Do I understand correctly that you consider MPLS Traffic Engineering a failed technology ? If so, I'd love to see a post by you explaining why.

    I thought MPLS TE did the right things ? It aggregates small flows into very big flows. The tunnels are set up only once, not dynamically based on traffic. You configure tunnels only where you need them, not everywhere in the network. It got deployed. So why did it fail ? Did people stop using it ? Is it too complex to configure ? To troubleshoot ? Not worth the effort ? Curious.
    Replies
    1. Yeah, it wasn't fair - the slide doesn't explain the reason I put MPLS TE on it, but the subsequent slides do ;)

      The real problem of MPLS TE is that it's neigh impossible doing it right without central visibility. Here's a pretty good talk outlining the challenges: https://ripe64.ripe.net/archives/video/23/

      Also, see knapsack problem (which is NP-complete when faced with the right question even in a non-distributed form).
    2. Henk Smit - is that you?
    3. I really enjoyed working with Henk at Cisco - brilliant guy, I learned a lot from just listening to his comments and asides.
    4. This NP-complete problem can (and has been solved by bunch smart people) quite well. A minor dip into linear programming (another goodie from the grey beards ;-) can take you very far ...
    5. All NP-complete problems have been solved... but none of them in polynomial time unless you claim P = NP ;)

      What you (probably) wanted to say is that we have good-enough approximations that get results in reasonable time.
  9. Problem is there is so much to learn these days, that very little incentive to learn legacy 'dead' technologies for newcomers to the industry. Just trying to keep up with the pace of SDN, Fabrics, underlays, overlays, different vendor approaches, programmability, orchestration etc. makes the head spin.
    Replies
    1. If, however, you have sound fundamental knowledge, it's much easier to deal with "so much to learn" because it's mostly reinvented old stuff.

      You either invest in learning about the old failures, or learn the hard way why that old stuff failed (without ever realizing the significance of that failure) when the shiny new stuff breaks down ;)
    2. I was worried as anonymous by the huge amount of stuff to learn these days. However, after seeing and realizing that most engineers (including senior engineers with 10-15 years experience, sometimes called by Ivan, "expert beginners"), fail at the fundamental level, I took myself the opposite approach, going back for the fundamentals and ask yourself as much as you can how it works everything. I think is the best investment that any engineer can do in order to have a successful career.
    3. Well, the "expert beginners" term is not mine - this guy wrote a whole series of blog posts (and a book) on the topic:

      http://www.daedtech.com/tag/expert-beginner
  10. This is exactly why I've chosen to focus on the "security" side of SDN rather than the networking topology or transport side. I trust the network mind to get the packets to and fro, I just want the solutions to be inherently protected regardless of iteration.
    Replies
    1. If you don't understand the network/transport side, you are fundamentally underequipped to understand actually security, unless you mean Confused Information Systems Security Professional-type paper-shuffling.
  11. Well, you asked for suggestions... To me, this is a PERFECT opportunity for one of your wonderful webminars. "Fundamentals of Networking Technologies". You can include design choices, history, even math if you want to :-)
    Make it as a base for each of the roadmaps, and include it in the "free" tier as well.
    (This, of course, assumes it's not already there and I missed it :-/ )
    Replies
    1. Hi Fernando,

      Got the message ;)... and no, it's not there yet.

      Best,
      Ivan
  12. Maybe it's time to challenge your own belief on what a failure actually is. Is it truly a mistake or just an old link in a chain this is still evolving? I know there are ignorant people out there, but SDN is not just a big repeat of historical mistakes. This is something that has been progressing for awhile now, terminology and marketing lingo aside.
    Replies
    1. Dear Anonymous,

      I can't tell you how much I "appreciate" anonymous comments with vague unsubstantiated claims based on misunderstanding what I wrote or implying what I didn't write.

      In any case, should you decide to use your real name and focus on facts, we could have a nice civilized discussion.

      Kind regards,
      Ivan
  13. While I generally agree with you, as an armchair historian of technology I'd also point out that there many examples of "good" ideas that failed because they didn't work on their day's compute / memory / network that would likely work fine today. And that what "won" many times did because it was good enough not because it was better than it's competition but because of externalities like price or access to code or presence/absence of proprietary ecosystem. Things that certainly matter but don't really inform us about how to make good architectural decisions.

    Just because something failed commercially doesn't necessarily mean that what we should learn from that failure is that the technique itself was flawed. I could probably form cogent arguments with each of your lessons learned as to why that lesson should be unlearned if I were in the mood to be a cantankerous old man.
    Replies
    1. Hi Ian,

      We could easily start listing great ideas, from CLNS per-host addressing (I still think it was a good idea) and OSI session layer (see http://blog.ipspace.net/2009/08/what-went-wrong-tcpip-lacks-session.html) to VAX/VMS operating system or Digital's RDB database or Banyan Vines directory service... but that wasn't the point.

      I am positive that all things I listed on that slide deserve to be there, and I could add a few more (like X.25). Some of them were successful because they solved problems we don't have anymore (very high error rate) or problems people thought needed solving (ATM QoS).

      In any case, we should understand what their shortcomings were, so that when someone tries to reinvent the technologies we don't have to rediscover the shortcomings. Agreed?

      As for the "lessons learned", I would love to hear counter-arguments.

      Kind regards,
      Ivan
    2. Agreed. More of a blanket statement than any pointed issue with your list of failures.

      Lessons learned... I think that #1 and #3 would be essentially semantic arguments and, while they might make for a wonderful conversation at the bar, are not terribly interesting on the Internet ( define "scale" in flow-based systems, for example ). So let's talk about #2.

      There are semantics in play here as well, but I submit that the most evident reason hop-by-hop fails that I have encountered is the complexities involved in device administration rather and information security than in any inherent technical shortcoming in the _idea_ that all the information needed for forwarding is carried within the packet. Following instructions in a label or header is simple, making sure that the rules about what that label or header _mean_ to the follower are consistent and trustworthy is hard.

      So, while there is a long and not particularly distinguished list of failed attempts to do this, the ones that I can think of failed because we weren't ready with the security and administration apparatus, not because they didn't actually work well or scale well.

      And, on the flip side, by using path based forwarding to defer dealing with those security and administration shortfalls we just kicked the can down the road for a generation.

      Now we have examples of how to do ad-hoc tamper prevention and hyper-scale distributed node administration in real world situations, so it is perhaps reasonable that you could, for example, actually use IPv6+AH+hop-by-hop options and expect it would work. That is something of an ideal world argument, because it wouldn't work at all because we've all set our devices to ignore options because we can barely agree on how to propagate routes in a trustworthy and non-destructive manner, much less how to trust the rich end of the IP header.

      My point is that calling one better than the other is subjective when the "better" created a culture that is so afraid of a problem that they reject the solution.
    3. Time for a follow-up blog post with more details, which were totally beyond the point of this blog post, but seem to be interested to a few readers. Give me a few days ;)
    4. StreetTalk rocked, back in the day.
    5. I think there are a few examples of where technology failed to start purely because the time wasn't right.

      e.g. Apple's Newton -> Apple iPhone would be one
  14. circa 1994 had dual data centre IBM, each mainframe had lpars one of which was the network lpar one each with centralised control i.e vtam/ncp, one command and all the control moved from one site to the other non-disruptively. SDN is just old ideas recycled
  15. Networking education and various other areas of IT education have been co-opted via "partnership" between education institutions and vendors in many areas of the world.
    This produces swarms of "vendor symbiotes", which fit nicely into the vendors marketing strategy. In the end if this is mostly what is available for hire it is "fait accomplis".
    Having come from the days of running IPX, IP, DECNET, LAT, VINES, 3270 Terminals, Frame Relay, ATM, etc all on the same network it is entertaining to deal with the symbiotes.
  16. Agree completely. Great post. I have been in it since 84. I loved AT commands. I remember having a little BBS in 1989 using some BBS SW called pirate or black beard or something like that for my Novell and Token-Ring customers to dial in and obtain the latest desktop drivers for IPX and IBM NICs etc. I love the analogies on the older tech and SDN. Especially IntServ... You Nailed it Ivan.
  17. I would absolutely buy Ivan's book "Lessons Learned From Historical Networking Technologies".
    Replies
    1. And of course Douglas Comer's version of the book. Just to compare (http://packetpushers.net/podcast/weekly-show/show-255-future-networking-douglas-comer/)

      Not sure if there is a single version of the technology failure history;)
    2. That's funny, and true, to a certain extent - but the fundamental issue is that no matter how much you read and study up on this stuff, there's a huge chasm between abstract understanding and the hardcore, essential operational and architectural understanding gained from actually being forced to implement these technologies, watch them break in interesting and unexpected ways, and then be forced to fix them, on the fly.

      The situation with networking technologies today is highly analogous to the situation with automobile mechanics - it's only a tiny, shrinking percentage of aging practitioners who actually understand how the stuff works, the rest are just break/fix 'technicians' who aren't called to a vocation, but rather schlepping through a job.
    3. The business cannot afford hiring only experts. It's not only expensive. Imagine experts doing simple stuff;)
    4. The problem is that few, if any, future experts are in the pipeline.
  18. I am optimist in this matter. Not sure if we have single expert in every area of networking. Probably we have many.

    Some fundamental ideas are the same in many areas of Computer Science (QoS vs job handling batch processing during the mainframe days - just to cite Comer's podcast).
    I believe in smart people - every time I have difficult problem to solve, I simply approach very smart guy and ask for help. Many times the guy does not know the particular area I am working on. And it works - he/she can help just thinking differently (sees bits instead of adresses - citing the Comer again).

    Some people are simply bright. Some know a lot (also know history) and have some advantage over other. Not always they are the same people. We need both.
    You can learn the history (but the industry does not care if the previous attempt of using the idea was a failure) but it's difficult to fight brain limits.
  19. While I am a bit of a history buff myself, I would say that it is more useful to learn fundamental principles/algorithms of which the technology (networking in this case) is based upon on. So a solid foundation of computer science would help a lot.

    The tradeoffs are that it raises the learning complexity so it increases effort and time (more compute) :)
  20. 29 years in the telecom world here ... sadly enough :)

    I come from the world of x21, x25, PDH, SDH and WDM,- I kind of like this very rigid world of rules,- even after many years of Ip and routers.
    In old days we tested a STM-16 (eq 2.5G) system fully cascade wired for 2-3 week, kicking it, slamming the doors, jerking fibers/cables, connectors etc. Not a single bit error was accepted.
    Now we send 10 pings ... only 2 dropped? Perfect job!!

    What will the international network world look like in 20-30 years? I believe somehow in the upcoming SDN philosophy. Centralized control and configuration with end-to-end perspective.
    No need for each L3 device making it's own decisions.

    Could we imagine Tier1 providers having "major" SDN policies,- instructing tier2 SDN controllers ... to tier3 SDN controllers ... etc?

    Just thinking .... good old hierarchical and efficient thinking applied to ip traffic?
  21. Thanks Ivan; I love this post.

    & every now & again I feel a certain sense of pride when I'm called to implement or troubleshoot ATM or Frame Relay because no one else knows how to do it :-)
  22. IMHO it's the result of an industry living too much on ( bad ) marketing hype.
    If you believe that new is always better ( bad marketing hype ) than why bother remembering the past ? It's not only in protocols btw. In some firms it's also how they manage information about N-1 and N-2 products.
Add comment
Sidebar