Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

6 week online course

reserve a seat

Articles I wrote for TechTarget

Another year-end cleanup action: I wrote lots of articles for SearchTelecom in the last few years. You can find links to all of them (together with those I wrote for SearchNetworking and SearchEnterpriseWan) on this page. Enjoy!

You have to register with TechTarget to be able to view them, but they do respect your settings (you can decide not to subscribe to any of their mailing lists).

Add comment

Most commented posts written in 2010

These posts generated more than 20 comments (they are sorted by date, not by the number of comments):

Year after year, I’m amazed by the amount of information you contribute in your comments and the previously unexplored paths you’re showing me. Thank you all!

Add comment

Most popular posts written in 2010

Most of my 2010 top hits were written years ago. How did the posts written in 2010 fare? Google Analytics claims these were the most popular:

Only the first three made it into the top-30, the rest of them are at the head of a very long tail (which starts at approximately 3000 unique pageviews per article).

Add comment

2010 top-10 posts

According to Google Analytics, these posts got the most pageviews in 2010:

... and yet I’m stubbornly writing about stupid things nobody cares about like MLAG or CLNP.

see 1 comments

Another year (almost) gone by

It’s hard to believe it’s another Christmas Eve, but obviously there must be some truth in the old wisdom that time flies by as you get older. I know some of you have holiday maintenance windows in the next days; I can only wish you hiccup-free experience. But regardless of what’s waiting for you in the near future, if the days around Christmas are a holiday season for you, don’t forget that it all started with a loving family having pretty hard times, not with the Three Wise Men going to a shopping mall to buy gifts.

I’ll try to follow my own advice for a change; you won’t find me on Twitter during the rest of the year and the only blog posts I’ll publish will be the yearly statistics. As this is the last “real” post in 2010, I would like to wish you Merry Christmas and a fantastic 2011 (where fantastic includes happy, successful, healthy, and everything else you want it to be).

see 4 comments

Cleaning the Inbox: generic IT

Last gems found in dusty corners of my cluttered Inbox:

see 2 comments

Cleaning the Inbox: Internet-related links

Every Internet-related post is a great opportunity to increase comment count. I’ll pass this time, here are the articles I found interesting with little or no comments from my side. First the generic Internet:

And then my favorite controversy:

read more see 2 comments

Yearly webinar subscription – interesting questions

I received several questions about my yearly webinar subscription package. It started with a CCIE who was interested in buying it:

I just wanted to clarify: if I buy the yearly subscriptions I have access to 3 up and coming webinars PLUS all the archived material you have? That’s absolutely correct. Even more, if you decide you’d like to attend more than three live webinars in the 12 months after buying the subscription, you’ll get a 30% discount on all subsequent ones But even better (the terms have changed in the meantime), you get unlimited access to all live webinar sessions for the duration of your subscription.

Do you have a list of what is available in the archive material? Of course. You can view the list of all recordings (multiple recordings are available for some of the sessions) and the list of all PDF documents and router configuration sets.

read more see 1 comments

Cleaning the Inbox: data center, storage, virtualization

Links to great data center, storage and virtualization articles found in the depths of my bloated Inbox:

Technology short takes by Scott Lowe. A must-read.

Keys to Virtualization Success – this is how you do it right. Great job, Bob!

Virtualizing databases: too big to fail? – a common sense approach to “what should I virtualize” question.

read more Add comment

Cleaning the Inbox: networking links

I must have inherited some hamster (or pack rat if you're across the pond) genes: I’m collecting too many links to interesting blog posts and articles in my inbox, Delicious bookmarks and blog notes. Time to do some serious cleanup; let’s start with networking-related links (in no particular order)

Some Internet Architectural Guidelines and Philosophy – a must-read for people inventing crazy schemes like load balancing based on unicast flooding or MAC-over-MAC proprietary network virtualization (you know who you are but I doubt you read RFCs or my blog).

Real-Time Network Failure Detection – Terry Slattery describes how you can use BFD, UDLD, IP SLA and routing protocols to detect failures in your network. Read also my BFD IP Corner article for in-depth BFD details.

read more see 2 comments

Can you run OSPF over DMVPN?

Ian sent me a really good OSPF-over-DMVPN question after watching the recording of my DMVPN webinar (register here for a live session):

In the DMVPN webinar you discuss OSPF design and configuration. However, Cisco design guide says you should use a different routing protocol from what you use on your LAN but you seem to suggest it is okay to extend your OSPF network out to the DMVPN edge by continuing to use OSPF albeit in a different area.

The main issue you face when running OSPF over DMVPN is scalability: OSPF does not scale as well as other routing protocols when used over DMVPN.

read more see 3 comments

MLAG and load balancing

FullMesh added an excellent comment to my Multi-Chassis Link Aggregation (MLAG) and hot potato switching post. He wrote:

If there are two core routing switches and two access switches which are MLAGged together in both directions, and hosts that are dual-active LAGged to the pair of access switches, then the traffic would stay on whichever side the host places it.

He also opened another can of worms: load balancing in MLAG environment is dictated by the end hosts. It doesn’t pay to have fancy switches that support L3 or L4 load balancing; a stupid host implementing destination-MAC-address-based load balancing can easily ruin your day.

read more see 5 comments

MPLS VPN in enterprise networks

There are numerous reasons you might want to implement MPLS/VPN technology in your enterprise WAN; in most cases you have to provide local or end-to-end layer-3 isolation between different groups of users.

The “When should companies consider building MPLS networks into their WANs?” article I wrote for SearchEnterpriseWan lists a few of the scenarios (Himawan Nugroho’s blog describes another case study). If you’re looking for in-depth use cases, technology overviews and sample (working & tested) configurations, register for my Enterprise MPLS VPN Deployment webinar.

Add comment

Internet-related links (2010-12-19)

GigaOm published two interesting articles by Joe Weinman: in the first one, he describes why pay-per-use residential broadband Internet is probably inevitable, in the second one he predicts changes in user behavior if the service providers decide to implement it. I would also suggest you take time and read his in-depth Market for Melons article.

Obviously, collecting money costs money and the pay-per-use model is no exception (not to mention that most people would pay less), so the service providers prefer usage caps. There are numerous ways to implement usage caps, but implementing usage cap as an acceptable use policy and calling exceeding the cap policy violation is not the way to do it. Some people are truly trying to alienate the users.

read more see 4 comments

Yearly subscription to my webinars

A while ago I got an interesting challenge from one of my readers: “I would like to attend a few of your webinars, but the problem I have is that I’m interested in most of them. Is there something we can do?” After a few e-mails, we nailed down the concept I had been playing with for quite a while: yearly subscription package. It gives you three unlimited access to all live webinars and year-long access to all the materials and all the recordings I ever made for a fixed price. You can find a detailed description, list of all recordings and list of all available materials on my web site.

Buying the yearly subscription is easy: select the first webinar you’re interested in (the list of upcoming webinars is also on my web site) and buy the Yearly subscription ticket when registering; you can also buy directly from my web site. You’ll get access to the recordings and PDF materials a few minutes after the registration.

see 4 comments

Where would you need GRE?

In a recent tweet from @Neelixx following my duct tape of networking joke I became a GRE lover. Jokes aside, let’s see where it makes sense to use GRE.

Whenever you want to transport your data over a third-party IP infrastructure without exposing your addressing and routing structure (example: building a VPN across a public IP infrastructure), you need a mechanism that allows you to encapsulate your IP packets (which are not routable by the third-party IP infrastructure) into routable IP envelopes.

read more see 12 comments

HP Virtual Connect: every vendor has its own dinosaurs

I was listening to the HP Virtual Connect (VC) PPP podcast recently and got the impression that HP VC is a weirdly convoluted product. I started wondering what exactly they were thinking when they were designing it ... and had the epiphany when Ken Henault took a step back and explained the history leading to the current complexity (listen to the Packet Pushers podcast to get the whole story)

read more see 6 comments

DHCPv6 IA_PD relaying works with 12.2SRE2

Last week I ran numerous lab tests while preparing router configurations for the Building IPv6 Service Provider Core webinar (register here or buy a recording). One of the fantastic test results: DHCPv6 relaying works correctly on a 7200 running 12.2(33)SRE2, even when the client requests IA_PD option.

read more see 6 comments

How much IPv6 address space should a residential customer get?

A while ago I wrote about IPv6 addressing challenges some ISPs face and recommended what I thought was agreed-upon practice of giving residential customers a /64 or a /56. Not long after, I received an e-mail from an IPv6 guru saying:

[Worse] is when people start claiming to have expertise in IPv6 and promulgate this idea of residential /56s and /64s as immutable fact. The reality is that it is becoming more and more apparent that /56s and especially /64s to residential customers are going to be harmful to future innovation in IPv6.
read more see 35 comments

Remote access section added to the IPv6 service provider webinar

Due to extreme student interest, I’ve added a whole new remote access section to my Building IPv6 Service Provider Core webinar (register here or buy a recording). It covers PPPoE and Carrier Ethernet access methods (PPPoE configuration can be used in any dial-up environment; Carrier Ethernet configuration is probably applicable to cable as well) and describes the following topics:

  • SLAAC on access networks for hosts connecting to the IPv6 Internet;
  • DHCPv6 prefix delegation required by IPv6-enabled CPE routers;
  • Prefix allocation (SLAAC and DHCPv6) from local pools;
  • DHCPv6 relays and SLAAC/DHCPv6-RADIUS integration.

As always, attendees of past webinars can download the updated materials immediately and will get access to the new recording after the next week’s session.

Add comment

Multi-Chassis Link Aggregation (MLAG) and hot potato switching

There are two reasons one would bundle parallel Ethernet links into a port channel (official term is Link Aggregation Group):

  • Transforming parallel links into a single logical link bypasses Spanning Tree Protocol loop avoidance logic; all links belonging to the port channel can be active at the same time (see also: Multi-Chassis Link Aggregation basics).
  • Load sharing across parallel links in a port channel increases the total bandwidth available between adjacent L2 switches or between routers/hosts and switches.

Ethan Banks wrote an excellent explanation of traditional port channel caveats (proving that 1+1 sometimes does not equal 2); things get way worse when you start using Multi-Chassis Link Aggregation due to hot potato switching (the switch tries to forward packets toward destination MAC address as soon as possible) used by all MLAG implementations I’m familiar with.

read more see 13 comments

CLNP and the multihoming myths

When IESG decided to adopt SIP, not TUBA (TCP/UDP over CLNP) as IPv6, a lot of people were mightily disappointed and some of them still propagate the myths how CLNP with its per-node addresses would fare better than IPv6 with its per-interface addresses (you might find the writings of John Day on this topic interesting and Petr Lapukhov is also advocating this view in his comments).

These views are correct when considering small-scale (intra-network) multihoming, but unfortunately wrong when it comes to Internet-scale multihoming, where CLNP with TCP on top of it would be as bad as IPv4 or IPv6 is (routing table explosion due to multihoming is also one of the topics of my Upcoming Internet Challenges webinar).

read more see 4 comments

Can we go back to CLNP?

Paulie, a frustrated enterprise IPv6 early adopter summarized his pains in a comment to my “Small-site multihoming in IPv6: mission impossible?” post saying “[IPv6/IPv6 support] is a mess and depressing” and asked “Is it too late to go to CLNS?”

Quite a few old-timers (I’m definitely one of them) lament the glory days of VMS, DECnet Phase V and CLNP, but while CLNP was a viable alternative for the next-generation IP in 1993, it would fare worse than IPv6 today.

read more see 3 comments

Another security product killed

We all knew MARS is becoming a dead end (Cisco first removed third-party support and then stopped developing the product), now it’s official. MARS is dead.

Just in case you haven’t noticed, this is the third security product (after WAF and XML Gateway) Cisco has killed this year. Are they implementing borderless networks or trimming down to core competences while preparing for onslaught of market adjacencies?

see 8 comments

Interesting links (2010-12-04)

A medley of technology links harvested from my inbox:

Add comment

Chinese BGP incident: was it a traffic hijack?

You’re probably familiar with the April fat fingers incident in which Chinanet (AS 23724) originated ~37.000 prefixes for about 15 minutes. The incident made it into the annual report of US Congress’ U.S.-China Economic and Security Review Commission (page 243 of this PDF) and the media was more than happy to pick it up (Andree Toonk has a whole list of links in his blog post). We might never know whether the misleading statements in the report were intentional or just a result of clueless technical advisors, but the facts are far away from what they claim:

read more see 2 comments

Small-site multihoming in IPv6: mission impossible?

Summary: I can’t figure out how to make small-site multihoming (without BGP or PI address space) work reliably and decently fast (failover in seconds, not hours) with IPv6. I’m probably not alone.

Problem: There are cases where a small site needs (or wants) to have Internet connectivity from two ISPs without going through the hassle of getting a BGP AS number and provider-independent address space, and running BGP with both upstream ISPs.

read more see 10 comments

Internet peering disputes: follow the money

You’ve probably heard about the recent peering dispute between Level-3 and Comcast ... and might have enjoyed the frenzy with which the blogging pundits have followed the false net neutrality scent left by Level-3 spin doctors.

Facts first: Level-3 is trying to dump huge amount of data into Comcast’s network for free.

read more see 26 comments

Requirements for IPv6 in ICT equipment

Greg Ferro reached an interesting conclusion after going through my Content over IPv6 presentation: we won’t see IPv6 for a few years, so why bother. Although I disagree with his approach, he may be right ... but if you decide to ignore IPv6, you might be forced to implement it in a hurry, at which point you’ll be stuck if your equipment won’t support IPv6. The very minimum you need to do today is to buy IPv6-ready gear (and yell at the vendors if they try to charge extra for IPv6 support).

read more see 4 comments

FCoE between data centers? Forget it!

Was anyone trying to sell you the “wonderful” idea of running FCoE between Data Centers instead of FC-over-DWDM or FCIP? Sounds great ... until you figure out it won’t work. Ever ... or at least until switch vendors drastically increase interface buffers on the 10GE ports.

FCoE requires lossless Ethernet between its “routers” (Fiber Channel Forwarders – see Multihop FCoE 101 for more details), which can only be provided with Data Center Bridging (DCB) standards, specifically Priority Flow Control (PFC). However, if you want to have lossless Ethernet between two points, every layer-2 (or higher) device in the path has to support DCB, which probably rules out any existing layer-2+ solution (including Carrier Ethernet, pseudowires, VPLS or OTV). The only option is thus bridging over dark fiber or a DWDM wavelength.

read more see 7 comments

VMware Virtual Switch: no need for STP

During the Data Center 3.0 webinar (register here) I always mention that you can connect a VMware ESX server (with embedded virtual switch) to the network through multiple active uplinks without link aggregation. The response is very predictable: I get a few “how does that work” questions in the next seconds.

VMware did a great job with the virtual switch embedded in the VMware hypervisor (vNetwork Standard Switch – vSS – or vNetwork Distributed Switch – vDS): it uses special forwarding rules (I call them split horizon switching, Cisco UCS documentation uses the term End Host Mode) that prevent forwarding loops without resorting to STP or port blocking.

read more see 7 comments

Cisco IOS Login Enhancements are not IPv6-aware

One of the comments to my “IPv6 in Data Center: after a year, Cisco is still not ready” post included the following facts:

Up through at least 15.0(1)M and 12.2(53)SE2 the IPv6 support for management protocols is spotty; syslog is there, SNMP traps and the RADIUS/TACACS control plane aren't.

Another bug along the same lines was discovered by Jónatan Jónasson: When the Cisco IOS Login Enhancements feature logs successful or failed login attempt, it reports the top 32 bits of the remote IPv6 address in IPv4 address format. Here’s a sample printout taken from a router running IOS release 15.0(1)M.

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: test] 
[Source:] [localport: 23] at ... P#who Line User Host(s) Idle Location * 0 con 0 idle 00:00:00 2 vty 0 test idle 00:00:06 FEC0::CCCC:1

It looks like the recommendation we’ve been making two years ago is still valid: use IPv4 for network management.

see 7 comments

Content over IPv6: No Excuses!

Yesterday I spent the whole day at another fantastic IPv6 Summit organized by Jan Žorž of the go6 institute. He managed to get two networking legends: Patrik Fältström (he was, among numerous other things, a member of Internet Architecture Board) had the keynote speech (starts @ 11:40) and Daniel Karrenberg (of the RIPE fame) was chairing the technical panel discussion. My small contribution was a half-hour talk on the importance of IPv6-enabled content (starts @ 37:00).

read more see 4 comments

IPv6 in Data Center: after a year, Cisco is still not ready

Today I’m delivering another IPv6 presentation, this time at the 4th Slovenian IPv6 Summit organized by tireless Jan Žorž from the go6 Slovenian IPv6 initiative. It’s thus just the right time to review the post I wrote a bit more than a year ago about lack of IPv6 readiness in Cisco’s Data Center products. Let’s see what has changed in a year:

read more see 20 comments

Upcoming Internet Challenges webinar

Last spring I prepared an “Upcoming Internet Challenges” presentation for Slovenian CCIE club based on my blog posts about not-so-well-known challenges Internet is facing in the next few years (the presentation is also available on SlideShare).

A few weeks ago one of the regular attendees of my webinars sent me an e-mail saying “Which webinar covers this topic? It seems extremely interesting and I would like to hear from you directly.” So here it is: the Upcoming Internet Challenges webinar. It will be a one-time event, so make sure you register for it if you’re interested in this topic (the recording will also be available as part of the yearly subscription package).

Add comment

Time-based static routes

Before someone accuses me of being totally FCoE/DCB-focused, here’s an interesting EEM trick. Damian wanted to have time-dependent static routes (you could use them to ensure expensive backup path is only established during the working hours). I told him to use cron with EEM to modify router configuration (and obviously lost him in the acronym forest)... but there’s an even better solution: use reliable static routing and modify just the track object’s state with EEM.

read more see 5 comments

FCoE, QCN and Frame Relay analogies

Just when I hoped we were finally getting somewhere with the FCoE/QCN discussion, Brocade managed to muddy the waters with its we-still-don’t-know-what-it-is announcement. Not surprisingly, networking consultants like my friend Greg Ferro of the Etherealmind fame responded to the shenanigan with statements like “FCoE ... is a technology so mindboggingly complicated that marketing people can argue over competing claims and all be correct.” Not true, the whole thing is exceedingly simple once you understand the architecture (and the marketing people always had competing claims).

Pretend for a minute that FC ≈ IP and LAN bridging ≈ Frame Relay, teleport into this parallel universe and allow me to tell you the whole story once again in more familiar terms.

read more Add comment

Nexus 1000V: another IPv6 #FAIL

Just stumbled across this unbelievable fact in the Nexus 1000V release notes:

IPV6 ACL rules are not supported.

My first reaction: “You must be kidding, right? Are we still in 20th century?” ... and then it dawned on me: Nexus 1000V is using the NX-OS control plane and it’s still stuck in 4.0 release which did not support IPv6 ACLs (IPv6 support was added to NX-OS in release 4.1(2)).

read more see 3 comments

Does FCoE need QCN (802.1Qau)?

One of the recurring religious FCoE-related debates of the last months is undoubtedly “do you need QCN to run FCoE” with Cisco adamantly claiming you don’t (hint: Nexus doesn’t support it) and HP claiming you do (hint: their switch software lacks FC stack) ... and then there’s this recent announcement from Brocade (more about it in a future post). As is usually the case, Cisco and HP are both right ... depending on how you design your multi-hop FCoE network.

read more see 1 comments

Data Center Bridging (DCB) Congestion Notification (802.1Qau)

The last (and the least popular) Data Center Bridging (DCB) standard tries to solve the problem of congestion in large bridged domains (PFC enables lossless transport and ETS standardizes DWRR queuing). To illustrate the need for congestion control, consider a simple example shown in the following diagram:

It came to my attention that a vendor might be using this blog post to justify the need for QCN in FCoE environments. Should that be the case, please make sure you also read about the difference between dense and sparse FCoE, the (lack of) need for QCN in FCoE and whether it makes sense to run FCoE over TRILL. Finally, consider how you’ll troubleshoot FCoE environments.

read more see 4 comments

vCloud disruptiveness: nothing new

The vCloud Director: hand the network over to server admins post received several fantastic well-reasoned comments (thank you all!) that you should read in their entirety. Jónatan Natti correctly pointed out (among other things) that we’ve often heard “And now a networking vendor is trying to persuade people with limited exposure to [...] issues to rebuild [...]" where [...] could stand for Voice/PBX, SNA or storage.

Unfortunately, in a retrospective, although a lot of that noise was FUD (or resulted from excessive complexity of legacy technology), the core of those claims was often spot-on. Ronan McGurn underestimated voice (he was part of a very large crowd, including a certain five-letter vendor) and I also have a few personal Voice/SNA campfire stories to share.

read more Add comment

DHCP the Microsoft way: almost standard

Srinivas sent me the following printout a few days ago and asked me whether I could explain the weird DHCP bindings (I removed the lease expiration column from the printout):

Switch#sh ip dhcp binding
Bindings from all pools not associated with VRF:
IP address          Client-ID/              Type
                    Hardware address/
                    User name     0152.4153.2000.188b.    Automatic
                    00     0152.4153.2000.188b.    Automatic
read more see 3 comments

Enterprise MPLS/VPN webinar

We were implementing MPLS/VPN building block in our customers’ enterprise networks for almost a decade, often using Multi-VRF (the solution formerly known as VRF-Lite) and sometimes implementing full-blown private MPLS/VPN networks (including private MPLS/VPN with IPsec over two MPLS/VPN service provider networks).

I always considered enterprise MPLS/VPN to be an equivalent of enterprise BGP: rare, but not so hard to grasp once you decide you need it. The MPLS/VPN Packet Pushers podcast was an eye opener: MPLS/VPN seems to be more complex than I thought. Let’s fix that: if you think your enterprise network could use MPLS/VPN, you’ll get a solid foundation in my Enterprise MPLS/VPN Deployment webinar (register here).

read more Add comment

vCloud Director: hand the network over to server admins

A few months ago VMware decided to kick away one of the more stubborn obstacles in their way to Data Center domination: the networking team. Their vCloud architecture implements VLANs, NAT, firewalls and a bit of IP routing within the VMware hypervisor and add-on modules ... and just to make sure the networking team has no chance of interfering, they implemented MAC-in-MAC encapsulation, making their cloudy dreamworld totally invisible to the lowly net admins.

read more see 7 comments

VPLS is a technology, not just a service provider offering

The Internet Exchange and Peering Points Packet Pushers Podcast is as good as the rest of them (listen to it first and then continue reading), but also strangely relevant to the data center engineers. When you look beyond the peering policies, route servers and BGP tidbits, an internet exchange is a high-performance large-scale layer-2 network that some data center switching vendors are dreaming about ... the only difference being that the internet exchanges have to perform extremely well using existing products and technologies, not the shortest-path-bridging futures promised by the vendors.

read more see 26 comments

IPv6 addressing: how wrong can you get it?

Mike was wondering whether his ISP is giving him what he needs to start an IPv6 pilot within his enterprise network. He wrote:

So I got an IPv6 assignment with a /120 mask (basically our IPv4/24 network mapped to IPv6) and two smaller networks to use for links between our external router and the ISP.

Believe it or not, I’m not making this up. I was as amazed as you probably are.

Dear Mike’s ISP: where were you when the rest of the world was preparing to deploy IPv6? Did you read IPv6 Unicast Address Assignment Considerations (RFC 5375) or IPv6 Address Allocation and Assignment Policy from RIPE or your regional registry?

read more see 26 comments

Solving the MPLS/VPN QoS challenge

Two weeks ago I wrote about the challenges you’ll encounter when trying to implement end-to-end QoS in an enterprise network that uses MPLS/VPN service as one of its transport components. Most of the issues you’ll encounter are caused by the position of the user-SP demarcation point. The Service Providers smartly “assume” the demarcation point is the PE-router interface ... and everything up to that point (including their access network) is your problem.

read more see 6 comments

What is MPLS-TP and is it relevant?

At the time when I was writing my MPLS books and developing MPLS courses for Cisco, everyone was ecstatically promoting GMPLS (Generalized MPLS) as the next unifying technology of everything, making someone so fed up with the fad that he wrote the Electricity over IP RFC.

GMPLS got implemented in high-end routers, but never really took off (at least I’ve never seen or even heard about it). Obviously the transport teams found the idea of routers requesting on-demand lambdas with IP-based protocols too hard to swallow.

read more see 19 comments

Early access to webinar recordings

I get a question along these lines at least once a week: “I would love to register for your webinar, but it’s too late for me. I need the information sooner than that.

My usual response was: “Don’t worry, just register and let me know you’d like to get access to the materials & recordings.” and I would then send a URL to a prior Webex recording and a PDF file to the attendee.

Recently I rolled out webinar management system to build the infrastructure for another exciting project (more about that in a week or so) and one of the nice side effects is that you get access to past materials (automatically, without the need to ask me) right after the registration: half an hour after placing the order with Eventbrite, you can access the webinar materials and view Webex recordings of all past sessions.

read more see 6 comments

Which MPLS-based VPN service should I choose?

A while ago, Packet Pushers did a Q&A podcast (or was it this one ... they’re all great) and one of the questions was “should I buy MPLS/VPN or VPLS service?” Greg’s response was along the lines of “Ivan would be the right one to answer this question” and as my regular readers and attendees of my webinars know, you can get a very comprehensive version of the answer in my Choose the optimal VPN service webinar (register here or buy a recording).

You’ll find a somewhat shorter answer in the Find the best MPLS/VPN service for your WAN tutorial I wrote for

Add comment

Multi-chassis Link Aggregation (MLAG): external brains

This is my third MLAG post. You might want to read the Multi-chassis Link Aggregation Basics and Multi-chassis Link Aggregation: Stacking on Steroids posts before continuing.

Juniper has introduced an interesting twist to the Stacking on Steroids architecture: the brains of the box (control plane) are outsourced. When you want to build a virtual chassis (Juniper’s marketing term for stack of core switches) out of EX8200 switches, you offload all the control-plane functionality (Spanning Tree Protocol, Link Aggregation Control Protocol, first-hop redundancy protocol, routing protocols) to an external box (XRE200).

read more see 12 comments

Interesting links (2010-10-31)

This set of links is somewhat different: they all deal with IT, but mostly with non-technical part of it.

Kevin Bovis @ Etherealmind continues a series of great insightful articles with Network Design – Creativity and Compromise. Must-read for anyone striving to be involved in network design.

The IT Disconnect by Tom Carpenter tackles one of the fundamental IT issues: we’re too busy dealing with our problems to focus on the problems of those paying us to deliver the service.

Chuck Hollis is describing stupid approaches customer use to structure their RFPs. If you were ever working for a vendor or system integrator, you’ll find it hilarious.

Mike Workman is dealing with the same problem from the customer perspective: “Why is it so hard to shop?

Add comment

DHCPv6 relaying: another trouble spot?

My DHCPv6+PPPoE post received a very comprehensive comment from Ole Troan (thank you!) in which he explains the context in which DHCPv6 was developed (a mechanism to give a static IPv6 prefix to a customer) and its intended usage (as the prefix is static, it should have a very long lifetime).

However, when you deploy DHCPv6 in some modern access networks (it’s not just PPPoE, Carrier Ethernet fares no better), you might experience subtle problems. Let’s start with a step-by-step description of how DHCPv6 works:

read more see 14 comments

DMVPN scalability

Alexander sent me a very valid question: “Do you cover scalability problems in your DMVPN webinar” (register here). Of course I do, more than half of the webinar is devoted to them.

As you know, DMVPN is a combination of multiple technologies, including ISAKMP (key exchange), IPsec (encryption), GRE (tunneling), NHRP (tunnel endpoint resolution) and a routing protocol. Any one of these can be a limiting factor:

read more see 1 comments

DHCPv6 over PPPoE: Total disaster

Every time someone throws me an IPv6 curveball, I’m surprised when I discover another huge can of worms (I guess I should have learned by now). This time it started pretty innocently with a seemingly simple PPPoE question:

What happens if an ISP decides to assign dynamic IPv6 subnets? With static assignment, the whole stuff is pretty straight-forward due to ND, RA & DHCPv6, but if dynamic addresses are used, what happens if the subnet changes - how will the change be propagated to the end-user devices? The whole thing is no problem today due to the usage of NAT / PAT...

LAN address allocation with changing DHCPv6 prefix is definitely a major problem, but didn’t seem insurmountable. After all, you can tweak RA timers on the LAN interface, so even though the prefix delegated through DHCPv6 would change, the LAN clients would pick up the change pretty quickly. WRONG ... at least if you use Cisco IOS.

read more see 9 comments

Solution: EIGRP summarization breaks Phase 2 DMVPN

Last week I posted an interesting challenge: what happens if you configure route summarization in a Phase 2 DMVPN network? The only response came from an anonymous contributor strongly suspected to be a routing/DMVPN expert working for a CCIE-related training company. Everyone else obviously found the question too trivial ... or too obscure, in which case it could be a good idea to join us at the next DMVPN: From Basics to Scalable Networks webinar.

The anonymous responder was somewhat cryptic, so let’s do a step-by-step explanation. We’ll use a simple 3-router network; C1 is hub, R2 and R3 are spokes.

read more see 9 comments

QoS over MPLS/VPN networks

A while ago John McManus wrote a great DSCP QoS Over MPLS Thoughts article at Etherealmind blog explaining how 6-bit IP DSCP value gets mapped into 3-bit MPLS EXP bits (now renamed to Traffic Class field). The most important lesson from his post should be “there is no direct DSCP-to-EXP mapping and you have to coordinate your ideas with the SP”. Let’s dig deeper into the SP architecture to truly understand the complexities of this topic.

We’ll start with a reference diagram: user traffic is flowing from Site-A to Site-B and the Service Provider is offering MPLS/VPN service between PE-A and PE-B. Traffic from multiple customer sites (including Site-A) is concentrated at SW-A and passed in individual VLANs to PE-A.

read more see 1 comments

Yes, it's still the same blog

Don't worry, you're still reading the same blog ... finally I found some time to implement the new design that has been waiting in my Inbox for over a month. Numerous glitches still have to be fixed, but at least the major changes are done.

How do you like the new design? Please share your opinion in the comments. Thank you!

see 27 comments

Coping with long-distance vMotion requests

During the last Data Center webinar (register here or buy a recording) I got an interesting question when describing the inherent problems of long-distance vMotion: “OK, I understand all the implications, but how do I persuade my server admins?”

The best answer I’ve heard so far came from an old battle-hardened networking guru: “Well, let them try”.

read more see 5 comments

EIGRP summarization in DMVPN Phase 2 networks

Imagine the following scenario: you’ve configured a Phase 2 DMVPN network with a hub and a few spokes. DMVPN is configured properly, IPSec and NHRP are working, you can ping all around the DMVPN cloud.

Next step: configuring EIGRP. You know you have to disable EIGRP split horizon and EIGRP next-hop processing. You even remember to configure interface bandwidth.

Someone told you to minimize the EIGRP routing traffic, so you use EIGRP stub routers on the spokes and route summarization on the hub router. The final EIGRP configuration is shown in the following diagram (click to enlarge).

read more see 3 comments

Data Center Interconnect (DCI) encryption

Brad sent me an interesting DCI encryption question a while ago. Our discussion started with:

We have a pair of 10GbE links between our data centers. We talked to a hardware encryption vendor who told us our L3 EIGRP DCI could not be used and we would have to convert it to a pure Layer 2 link. This doesn't make sense to me as our hand-off into the carrier network is 10GbE; couldn't we just insert the Ethernet encryptor as a "transparent" device connected to our routed port ?

The whole thing obviously started as a layering confusion. Brad is routing traffic between his data centers (the long-distance vMotion demon hasn’t visited his server admins yet), so he’s talking about L3 DCI.

The encryptor vendor has a different perspective and sent him the following requirements:

read more see 5 comments

DMVPN webinar: From B to S

Short summary: The DMVPN webinar (register here) now includes detailed EIGRP and BGP configurations guidelines and sample configurations as well as in-depth description of Phase 2 / Phase 3 behavior and scalability guidelines. I also made sure you don’t need any prior DMVPN knowledge.

Tangential thought: B stands for Basics, S for Scalability. Any other interpretation is strictly prohibited.

And here’s the long story ...

read more Add comment

Multi-chassis Link Aggregation: Stacking on Steroids

In the Multi-chassis Link Aggregation (MLAG) Basics post I’ve described how you can use (vendor-proprietary) technologies to bundle links connected to two upstream switches into a single logical channel, bypassing the Spanning Tree Protocol (STP) port blocking. While every vendor takes a different approach to MLAG, there are only a few architectures that you’ll see. Let’s start with the most obvious one: stacking on steroids.

read more see 9 comments

IPv6 links (2010-10-17)

IPv6 is becoming a mainstream topic ... here are a few interesting articles from the last weeks:

There is no Plan B: why the IPv4-to-IPv6 transition will be ugly – explains why we are forced to go to IPv6, whether we like it or not.

Akamai: Why our IPv6 upgrade is harder than Google's – this is what happens when you’re caught asleep at the wheel. All you can do is yammer how your service is more complex than that of your competitors ... if I would be a customer, I would be worried and starting to look elsewhere.

Untunneling IPv6 – looks like we’re finally getting rid of IPv6 tunnels. We’re down to 10% ... just a bit more and it will become safe to have A and AAAA records for the same web hosts.

It’s bad enough when people in blogosphere claim IPv6 is safer than IPv4 and that it will enable better DRM because hosts are not hidden behind NAT. When the US federal government gets on the same bandwagon, it’s worrying.

On a more positive note, when you get past the myths and accept that IPv6 is not more secure than IPv4, there’s a great NIST publication to help you get started.

Last but definitely not least, for all those that think SHIM, HIP, ILNP, LISP ... will save the Internet, Job Snijders has published a great get-back-to-earth reality-checklist.

Oh, and I simply have to mention that the next session of my Building IPv6 Service Provider Core webinar is in mid-December.

Add comment

PFC/ETS and storage traffic: the real story

Data Center Ethernet (or DCB or CEE, depending on who you are) is a hot story these days and it’s no wonder that misconceptions galore. However, when I hear several CCIEs I highly respect talk about “Priority Flow Control can be used to stop all the other traffic when storage needs more bandwidth”, I get worried. Exactly the opposite is true: you use PFC to stop the overzealous storage traffic (primarily FCoE, but also iSCSI) to make sure you don’t drop it.

read more Add comment

PPPoE testbed, part 2

During my last Building IPv6 Service Provider Core webinar (register here) I got a lot of questions about IPv6 over PPPoE (obviously we’re close to widespread IPv6 implementation; I never got PPPoE questions before). I wanted to test various scenarios in my IPv6 lab and thus enabled PPPoE on an Ethernet link between CE and PE routers using the configurations I published last year.

This time I wanted to test multiple configurations in parallel ... no problem thanks versatile PPPoE implementation in Cisco.

read more see 10 comments

Building a Private Could

SearchNetworking has just published my Building a Private Cloud article in which I describe why you have to consider a private cloud solution if you operate a large data center, and an overview of the path to get there.

Read more @ SearchNetworking

More information:

Add comment

Which webinar should I choose if I’m building a DC interconnect?

Brad asked me about the availability of a DCI webinar (short answer: early next year) and continued “As an enterprise engineer, I know very little of how the service providers engineer their networks. I'm aware you do have a service provider webinar as well but I am wondering which one of those would be most beneficial for me to attend.

As we’ve already discussed some of his DC issues, I knew that he’s in the “sane” part of the DC universe (pure layer-3 interconnect, no vMotion or bridging between data centers) and thus has to solve a traditional routing design challenge.

For him, it makes no sense to wait for the DCI webinar; that one will cover the designs and technologies you need when everyone else is pushing you to implement long-distance bridging. It might also include LISP in DC and the load balancing tricks F5 does ... and I’ll try to give you as much ammunition as I can to help you persuade your boss that there are other things beyond bridge-everywhere craze.

read more Add comment

DMVPN Phase 1 overview (webinar recording)

One of the many VPN technologies covered in the Choose the Optimal VPN Service webinar (register here) is DMVPN Phase 1. Phase 1 with its hub-and-spoke topology is the easiest DMVPN implementation and gives you a simple baseline that can be later expanded with Phase 2 and Phase 3 features (for in-depth DMVPN information check the DMVPN: Advanced and Crazy Scenarios webinar).

The recording from the webinar describing technologies underlying DMVPN and DMVPN Phase 1 principles is now available on YouTube and Vimeo.

see 1 comments

Ethernet inter-frame gap: Another living fossil?

Recently I’ve stumbled across a year-old post by James Ventre describing the reasons output rate on an Ethernet-type interface (as reported by the router) never reaches the actual interface speed. One of them: inter-frame/packet gap (IPG).

I was stunned ... I remember very well the early days of thick/thin coax Ethernet when the IPG was needed for proper carrier sense/collision avoidance detection (probability of a collision decreases drastically as you introduce IPG), but on a high-speed point-to-point full duplex link? You must be kidding.

read more see 8 comments

DMVPN: from concept to pilot in 36 hours

Participants of my latest webinars might remember the concept of on-site workshops that I’m usually mentioning after the first break (it’s also mentioned in every webinar description). The networking team from a large multinational company has decided to test it in practice and invited me for a 3-day DMVPN workshop.

The agenda of these workshops is usually pretty simple:

  • Day 1: technology overview and review of the existing network design/challenges.
  • Day 2: work on proposed new network design.
  • Day 3: tying up loose ends and preparations for pilot/migration.

We agreed on a tentative agenda along these lines and I prepared the material for the technology overview using parts of my Choose the Optimal VPN Service webinar (to compare DMVPN with other VPN solutions) and the DMVPN webinar. Oh boy, was I in for a surprise.

read more see 2 comments

Licensing requirements for various VPN solutions

During the Choose the Optimal VPN Service webinar (register here) I got this interesting question: “What are the (customer network) licensing requirements for various VPN solutions?” (the webinar covers MPLS/VPN, VPLS, pseudowires, GRE, GRE-over-IPSec, IPSec VTI, GETVPN, DMVPN and hybrid designs).

All MPLS-based solutions require no special license on the customer side; they are implemented by the Service Provider, the customer requires basic IP routing functionality. Furthermore, BGP is now included in the IP Base image that you get with every ISR router and it’s part of the base 6500/7600 image for quite a while.

GRE and DMVPN (w/o IPSEC) are (according to Feature Navigator) available in IP Base image. I’m positive the GRE part is true; I would check the DMVPN functionality on an actual box before placing the order (or order Advanced IP Services image). For IPSec you need Advanced IP Services or Advanced Security image.

According to an ISR G2 licensing document, you need SECK9 license for both DMVPN and IPSec.

Add comment

New topics added to the Data Center webinar

I’ve added several new topics to the Data Center 3.0 for Networking Engineers webinar (register here or buy a recording), making it a 100+ slides behemoth:

  • Anycast in the load-balancing section (which also covers DNS-based load balancing, dedicated load balancers, Microsoft NLB and application-specific solutions).
  • More details on multi-chassis link aggregation and VSS/vPC architecture in the LAN reference architecture section (other topics in the same section: port extenders and large-scale bridging, including TRILL, 802.1aq and FabricPath).
  • Multihop FCoE and FIP snooping in the storage protocols section (other protocols described in this section: SCSI, FC, FCoE, iSCSI and DCB).

As always, the attendees of the previous Data Center webinars will get access to the new recording and the new PDF materials.

see 6 comments

Multiple EIGRP autonomous systems in a VRF

A while ago Ron sent me an intriguing question: “Is it possible to have two EIGRP AS numbers in the same VRF?” Obviously he’s working on a network with multiple EIGRP processes (not an uncommon pre-MPLS/VPN solution; I did a network design along the same lines almost 20 years ago).

It’s easy to run multiple EIGRP autonomous systems in the global IP routing table; just create more than one EIGRP process. They can even run over the same set of interfaces. EIGRP-in-a-VRF implementation is slightly different; you configure an address family within another EIGRP process and (optionally) specify an AS number that does not have to match the AS number of the EIGRP process.

read more see 1 comments

Interesting links (2010-10-03)

Ethan Banks is continuing his deliberations on going independent (or not). He’s definitely collected some very interesting feedback.

Stephen Foskett shared a link to an interesting blog post: “How to Pitch A Tech Blogger”. A must read for vendors attending Tech Field Day (have you noticed there is no mention of Gartner or IDC? ;)

Another great read from Stephen: a large dose of common sense hidden under the “Four Fundamental Best Practices for Enterprise IT” title. I wonder if the vendors touting TRILL and inter-DC bridging with long-distance vMotion ever got to the “Minimize Complexity” part, let alone “Align Expectations with Reality”.

The “Hack The Stack Or Go On a Bender With a Vendor” tackles the age-old dilemma: build or buy, as it applies to the cloudy environments.

Brad Hedlund continues his series of great UCS posts with the explanation of UCS Fabric Failover feature.

Jeremy Gaddis documented his impressions of the Net Field Day 2010 event: day 1 and day 2. If you’ve missed NFD-related posts, Stephen Foskett makes sure you’re able to read them all via his Net Field Day 2010 Links page.

Add comment

Multi-chassis link aggregation (MLAG) basics

If you ask any Data Center networking engineer about his worst pains, I’m positive Spanning Tree Protocol (STP) will be very high on the shortlist. In a well-designed fully redundant hierarchical network where every device connects to at least two devices higher in the hierarchy, you lose half the bandwidth to STP loop prevention whims.

Of course you can try to dance around the problem:

read more see 27 comments

Introduction to 802.1Qaz (Enhanced Transmission Selection – ETS)

Enhanced Transmission Selection (ETS) is the second part of the Data Center Bridging puzzle (I’ve already described Priority Flow Control). It specifies two different technologies:

  • Queuing mechanisms in bridges
  • Data Center Bridging eXchange protocol: a Control/Negotiation protocol that allows bridges and hosts to negotiate QoS parameters in a bridged network.

Although some bridges from some vendors supported numerous QoS mechanisms in the past, 802.1Qaz is the first attempt to standardize a richer set of QoS behaviors than the strict priority queuing defined in 802.1p.

read more Add comment

FCoE Quote-of-the-Day

“Use FC where you feel you need to, use Ethernet (NFS, CIFS, iSCSI) everywhere else -- and save money and effort in the process”

Etherealmind and myself have been singing this song for quite some time (probably upsetting a few people working for my favorite vendor), but this time it comes straight from EMC’s CTO. And he didn’t even mention FCoE in his list of storage protocols.

Add comment

What exactly is a Nexus 4000?

Someone mentioned a while ago in a comment to one of my blog posts that the Nexus 4000 switch already supports multihop FCoE. Now that we know what multihop FCoE really is, let’s see how Nexus 4000 fits into the picture.

The Cisco Nexus 4000 Series Design Guide starts with a confusing set of claims:

  • The Cisco Nexus 4000 Series Switches provide the Fibre Channel Forwarder (FCF) function.
  • Nexus 4000 is a FCoE Initialization Protocol (FIP) snooping bridge.
read more see 4 comments

ATAoE: response from Coraid

A few days after writing my ATAoE post I got a very nice e-mail from Sam Hopkins from Coraid responding to every single point I’ve raised in my post. I have to admit I’ve missed the tag field in the ATAoE packets which does allow parallel requests between a server and a storage array, solving some of the sequencing/fragmentation issues. I’m still not convinced, but here is the whole e-mail (I did just some slight formatting) with no further comments from my side.

read more see 8 comments

DMVPN: Non-Unique NHRP Registrations

During my last DMVPN: Advanced And Crazy Scenarios webinar (register here), one of the students mentioned the need for non-unique NHRP registrations in environments where the public IP address of a DMVPN spoke site changes due to DHCP lease expiration or PPPoE session termination. Finally I found some time to recreate the scenario in my DMVPN lab; here are the results.

read more see 8 comments

Hiding documentation ... will they never learn?

One of the best presentations we had last week during the Net Field Day 2010 was given by Doug Gourlay from Arista. Their products have numerous highly interesting features; Terry liked their use of TDR and I was particularly delighted by the VM Tracer and decided to write about it as soon as I find some time (read: today).

2012-09-29: To keep the record straight: a few months after I wrote this blog post, Arista made most of the EOS documentation available online (as of today, it's latest version only, with no release notes).

read more see 17 comments

Setting access lists with RADIUS

Chris sent me an interesting challenge a few days ago: he wanted to set inbound access lists on virtual access interfaces with RADIUS but somehow couldn’t get this feature to work.

Uncle Google quickly provided two documents on an older one (explaining the IETF attributes, vendor-specific attributes and AV-pairs) and the most recent one (with more attributes and less useful information) covering every Cisco IOS software release up to 12.2 (yeah, it looks like the RADIUS attributes haven’t been touched in a long time). According to the documentation, attribute #11 as well as AV-pairs ip:inacl/ip:outacl and lcp:interface-config should work, but the access list did not appear in the interface configuration.

read more see 5 comments

Advanced DMVPN webinar: router configurations

If you register for my Advanced DMVPN webinar (register here), you’ll get 12 sets of complete router configurations covering every single design scenario described in the webinar. The seven router lab topology emulates an enterprise DMVPN deployment with a redundant central site, a redundant remote site (with two routers) and two non-redundant remote sites (using two uplinks in a few scenarios). The seventh router emulates the Internet. The configurations can be used on any hardware (real or otherwise) supporting recent Cisco IOS software, allowing you to test and modify the design scenarios discussed in the webinar.

read more see 2 comments

Multihop FCoE 102: VN_port proxy and FIP snooping

A few weeks ago I wrote about the multihop FCoE basics and the two fundamentally different ways an FCoE network could be designed: FCoE on every switch or FCoE on the edges with DCB-extended bridging in the middle.

There are two other configurations you’ll likely see in access parts of an FCoE network: FCoE VN_port proxying and FIP snooping.

read more Add comment

Virtualization links (2010-09-19)

A medley of virtualization/access network links:

Access layer virtualization: VN-Tag and VEPA. A nice summary of what Cisco and HP are trying to sell us as VN-Tag and VEPA. The truth might be a bit different (definitely not Joe’s fault).

UCS Network Adapter Options Overview. A nice summary of three NIC architectures with a reasonable answer to the question: “why would I need virtual NICs in my server?” applicable to a generic Data Center environment.

Great Minds Think Alike – Cisco and VMware Agree On Sharing vs. Limiting. Another fantastic introductory QoS post. It looks like we have to repeat the “rate-limiting is bad, queuing is good” mantra ad nauseam hoping everyone eventually gets it. Bonus: people addicted to GUI might finally get it due to the illustrative highway analogies.

Speaking of illustrative, Brad Hedlund published a huge VMware 10GE QoS Design Deep Dive with Cisco UCS, Nexus article (no, I haven’t read it all yet ... my flight home from #TechFieldDay took only 11 hours).

Add comment

Net Field Day 2010 – first impressions

I just spent frantic three days in San Jose with a dozen of fellow bloggers attending the Net Field Day 2010 event masterfully organized by Stephen Foskett and Claire Chaplais (thank you both for a truly outstanding experience!). I can’t tell you how delighted I was when they selected me as one of the participants, more so as this event finally allowed me to get in touch with a number of people I was regularly meeting in vSpace. However, the whole point of the Net Field Day is to talk with the vendors and figure out what they’re doing, so let’s start with my first impressions.

The sorry state of the industry. My first impression: real networking innovation is gone.

read more see 2 comments

ATAoE for converged Data Center networks? No way

When I started writing about storage industry and its attempts to tweak Ethernet to its needs, someone mentioned ATAoE. I read the ATAoE Wikipedia article and concluded that this dinky technology probably makes sense in a small home office ... and then I’ve stumbled across an article in The Register that claimed you could run a 9000-user Exchange server on ATAoE storage. It was time to deep-dive into this “interesting” L2+7 protocol. As expected, there are numerous good reasons you won’t hear about ATAoE in my Data Center 3.0 for Networking Engineers webinar and I described a few of them in a blog post I wrote to SearchNetworking’s Fast Packet blog.

Read more @ SearchNetworking

Add comment

Advanced DMVPN webinar: new content description

I have arrived to San Jose for the Net Field Day 2010 and I’m totally jet-lagged. Nine hour time difference and being awake for 24 hours after spending 15 hours in airplanes totally fried my brains, so instead of trying to write another technical blog post, let me just tell you about the changes I made to the DMVPN: Advanced And Crazy Scenarios webinar description (register here).

As I started developing the webinar, I had a pretty good idea about the topics I wanted to cover and created what seemed to be a good content outline. However, as I started to develop individual topics and dived deep into the DMVPN problems, the material somehow started to self-reorganize to make the flow better and all of a sudden the description I wrote was out-of-sync with the reality. That’s fixed now ... and if you’re familiar with the old outline, you’ll see that the new one makes more sense as it closely ties the DMVPN phases with the scalability problems you have to solve.

see 2 comments

Storage networking is like SNA

I’m writing this post while travelling to the Net Field Day 2010, the successor to the awesome Tech Field Day 2010 during which the FCoTR technology was launched. It’s thus only fair to extend that fantastic merger of two technologies we all love, look at the bigger picture and compare storage networking with SNA.


  • If you’re too young to understand what I’m talking about, don’t worry. Yes, you’ve missed all the beauties of RSRB/DLSw, CIP, APPN/APPI and the likes, but major technology shifts happen every other decade or so, so you’ll be able to use FC/FCoE/iSCSI analogies the next time (and look like a dinosaur to the rookies). Make sure, though, that you read the summary.
  • I’ll use present tense throughout the post when comparing both environments although SNA should be mostly history by now.
read more Add comment

Long-distance vMotion and the traffic trombone

Few days ago I wrote about the impact of vMotion on a Data Center network and the traffic flow issues. Now let’s walk through what happens when you move a running virtual machine (VM) between two data centers (long-distance vMotion). Imagine we’re moving a web server that is:

  • Serving a few Internet clients (with firewall/NAT and/or load balancing somewhere in the path);
  • Getting most of its data from a database server sitting nearby;
  • Reading and writing to a local disk.

The traffic flows are shown in the following diagram:

read more see 8 comments

IPv6 SP Core webinar: router configurations

The attendees of my Building IPv6 Service Provider Core webinar (register here) get several sets of complete router configurations for a six router lab that emulates a typical Service Provider network with a residential customer and an enterprise BGP customer. The configurations can be used on any hardware (real or otherwise) supporting recent Cisco IOS software, allowing you to test and modify the design scenarios discussed in the webinar.

read more Add comment

vMotion: an elephant in the Data Center room

A while ago I had a chat with a fellow CCIE (working in a large enterprise network with reasonably-sized Data Center) and briefly described vMotion to him. His response: “Interesting, I didn’t know that.” ... and “Ouch” a few seconds later as he realized what vMotion means from bandwidth consumption and routing perspectives. Before going into the painful details, let’s cover the basics.

read more see 12 comments

Introduction to LISP

I’ve been mentioning LISP several times during the last months. It seems to be the only viable solution to the global IP routing table explosion. All other proposals require modifying layers above IP and while that’s where the problem should have been solved, expecting those layers to change any time soon is like waiting for Godot.

If you’re interested in LISP, start with the introduction to LISP I wrote for Search Telecom, continue with the LISP tutorial from NANOG 45 and (for the grand finale) listen to three Google Talks from Dino (almost four hours).

Read my article @

see 8 comments

Tunnel Route Selection (recording from the webinar)

A while ago I wrote about the Tunnel Route Selection feature of Cisco IOS and how it could be used to solve the redundantly-connected spoke site issue. Here’s the basic design: you have two uplinks to two ISPs, two DMVPN tunnel interfaces, each one sourced from one of the uplinks and two default routes. Everything works great until one of the ISPs enables RPF checks ... and then the all hell breaks loose. More in a short clip made from a recording of my DMVPN – From Basics To Scalable Networks webinar.

see 5 comments

Introduction to 802.1Qbb (Priority-based Flow Control — PFC)

Yesterday I wrote that you don’t need DCB technologies to implement FCoE in your network. The FC-BB-5 standard is quite explicit (it also says that 802.1Qbb is the other option):

Lossless Ethernet may be implemented through the use of some Ethernet extensions. A possible Ethernet extension to implement Lossless Ethernet is the PAUSE mechanism defined in IEEE 802.3-2008.

The PAUSE mechanism (802.3x) gives you lossless behavior, but results in undesired side effects when you run LAN and SAN traffic across a converged Ethernet infrastructure.

read more see 16 comments

Another week, another #fail

A few weeks ago, my “friendly” registrar completely messed up my zone file, last week Websense decided my blog was spam and this week yours truly managed to change his domain password without realizing the change would break all scheduled jobs, including RSS feed update (I have to do some extra mangling due to Blogger’s lack of features). Takeaway: try to minimize the number of systems and services used in your solution (this conclusion should be very familiar to those that attended my Next-generation IP Services webinar) ... and prefer Linux over Windows ;)

Unfortunately the three failures had a common pattern: they affected at least 10% of my readers (as judged by the traffic statistics), but in all three cases a single user (thank you, Igor, Dan and @windexh8er) reported the problem ... hundreds of others decided they don’t care enough to fire off a simple e-mail saying “Hey, Ivan, you have a problem.”

If you spot a problem with my blog that doesn’t go away in a few hours, could you please send me a short e-mail? It’s simply impossible to test all potential environments; for example, the CNAME problems affected only a few (the most compliant) DNS servers and we use Ironport, not Websense.

Add comment

DMVPN: Advanced and Crazy Scenarios (2010-07-07) survey results

More overdue paperwork: the results of the July DMVPN: Advanced and Crazy Scenarios webinar (register for the next session). The audience was (as expected) very senior, with more than half of the respondents having 7+ years of internetworking experience and most of them working as system engineers:

read more Add comment

Virtual aggregation: a quick fix for FIB/TCAM overflow

Quick summary for the differently-attentive: virtual aggregation solves TCAM overflow problems (high-level description of how it works).

During the Big Hot and Heavy Switches podcast, Dan Hughes complained that the Nexus 7000 switch cannot take the full BGP table. The reason is simple: it’s TCAM (FIB) has only 56.000 entries and the BGP table has almost 350.000 routes.

Nexus 7000 is a Data Center switch, so the TCAM size is not really a limitation (it would usually have a default route toward the WAN core), but the same problem is experienced by Service Providers all over the world – the TCAM/FIB size of their high-speed routers is limited.

read more see 5 comments

RIBs and FIBs (aka IP routing table and CEF table)

Every now and then, I’m asked about the difference between Routing Information Base (RIB), also known as IP Routing Table and Forwarding Information Base (FIB), also known as CEF table or IP forwarding table.

We’ve discussed this topic during the Enterprise MPLS Packet Pushers Podcast, so you might want to listen to that one first before going into details.

Let’s start with an overview picture (which does tell you more than the next thousand words I’ll write):

read more see 16 comments

The IPv6 “experts” strike again

IT World Canada has recently published an interesting “Disband the ITU's IPv6 Group, says expert” article. I can’t agree more with the title or the first message of the article: there is no reason for the IPv6 ITU group to exist. However, as my long-time readers know, that’s old news ... and the article is unfortunately so full of technical misinformation and myths and that I hardly know where to begin. Trying to be constructive, let’s start with the points I agree with.

IPv6 was designed to meet the operational needs that existed 20 years ago. Absolutely true. See my IPv6 myths for more details.

ITU-T has spun up two groups that are needlessly consuming international institutional resources. Absolutely in agreement (but still old news). I also deeply agree with all the subsequent remarks about ITU-T and needless politics (not to mention the dire need of most of ITU-T to find some reason to continue existing). That part of the article should become a required reading for any standardization body.

And now for (some of) the blunders:

read more Add comment

BGP: time to grow up

If you’re in the Service Provider business, this is (hopefully) old news: on Friday, RIPE decided to experiment with the Internet causing routers running IOS-XR to hiccup. They stopped the experiment in less than half an hour and only 2% of the Internet was affected according to Renesys analysis (a nice side effect: Tassos had great fun decoding the offending BGP attribute from hex dumps).

My first gut reaction was “something’s doesn’t feel right”. A BGP bug in IOS-XR affects only 2% of the Internet? Here are some possible conclusions:

read more see 3 comments

Multihop FCoE 101

The FCoE confusion spread by networking vendors has reached new heights with contradictory claims that you need TRILL to run multihop FCoE (or maybe you don’t) and that you don’t need congestion control specified in 802.1Qau standard (or maybe you do). Allow me to add to your confusion: they are all correct ... depending on how you implement FCoE.

read more see 7 comments

Interesting links (2010-08-29)

In his HSRP, vPC and the vPC peer-gateway command post Jeremy Filliben documents how the storage vendors ignore RFCs and implement what they think is proper ARP handling, causing havoc in a redundant network.

Andrew Vonnagy writes about another extreme stupidity customer convenience Microsoft managed to implement: you can turn any Windows 7 into a rogue Access Point. Like we didn’t have enough problems already.

And then there’s Charles Stross, taking the “where we went wrongrants observations to a completely new level. While I’m complaining about lack of session layer in TCP/IP and broken socket API, he’s taking on Von Neumann architecture.

Add comment

Storage networking is different

The storage industry has a very specific view of the networking protocols – they expect the network to be extremely reliable, either by making it lossless or by using a transport protocol (TCP + embedded iSCSI checksums) that was only recently made decently fast.

Some of their behavior can be easily attributed to network-blindness and attempts to support legacy protocols that were designed for a completely different environment 25 years ago, but we also have to admit that the server-to-storage sessions are way more critical than the user-to-server application sessions.

read more see 1 comments