Category: BGP

Numerous articles describing the widespread routing instabilities caused by sloppy parser of a small router vendor (including posts at BGPmon, Renesys, Arbor Security and my blog) hinted that the unusual BGP update caused so many problems because the ISPs were using outdated Cisco IOS releases. This is definitely not the case; all classic IOS releases were affected.

Rodney Dunn from Cisco and myself were quickly able to reproduce so far unknown bug in Cisco IOS that occurs only when the inbound AS-path contains close to 255 AS numbers and the router does inbound or outbound AS-path prepending. The new bug is tracked as CSCsx73770 and affects downstream EBGP or IBGP sessions as follows:

The “BGP experiment” a small European ISP performed in February 2009 has generated quite a splash: Cisco has discovered a new BGP bug that can be triggered only if you have a long enough AS-path and do outbound AS-path prepending (and a few of us learned more BGP intricacies we never wanted to know), lots of people have (hopefully) discovered the importance of the bgp maxas-limit configuration command and at least some ISPs have implemented inbound prepending filters that I wrote about almost a year ago. However, most of us thought that the original problem arose due to inexperienced operators of a leaf AS.

Mikael Abrahamsson was the first to notice that the number of prepends matches the low-order 8 bits of the offending AS number. Further contributors to NANOG mailing list confirmed that two autonomous systems with very long prepends are using BGP routers from Mikrotik. You configure those boxes with commands that have syntax deceptively close to Cisco's, but expect the number of AS numbers to prepend, not the AS-path. Obviously no range checking is done on the configuration parameter and the high-order 8 bits are ignored.

So it looks like the incident started with a box that accepts invalid configuration parameter used in an AS with very high value in low-order 8 bits (quite improbable, but obviously not impossible). Numerous ISPs that did not limit the BGP updates they were propagating and an IOS bug did the rest.

In February 2009, a greenhorn ISP (they joined RIPE less than four months before the incident) in central Europe managed to generate a BGP update with too many AS numbers in the AS path, confusing older routers. You can find the details in an old Renesys blog post; at the peak of the instability, they were receiving over 100.000 BGP updates per second.

It’s very easy to protect yourself (and your downstream neighbors) from an operational error like this one. Cisco has implemented the AS-path length limiting code in IOS release 12.2. One would hope that the major ISPs would have started using this feature years ago; obviously that’s not the case, so here’s how you do it… just to make sure everyone understands what the bgp maxas-limit command does and hopefully implements it in this millennium.

BGP allows numerous attributes (including AS-path, metrics, local preference and communities) to be attached to every advertised IP prefix. The total length of BGP attributes attached to a single IP prefix can be very large (up to 64K bytes). IP prefixes with excessive amount of attribute data residing in the BGP table can results in significant memory utilization and trigger software bugs.

Cisco IOS can limit the maximum length of the AS-path attribute with the bgp maxas-limit length router configuration command. It’s highly advisable that you use this command together with other BGP security measures to reduce the potential impact of oversized AS-path attributes on the operation of your network.

The maximum sensible length of the AS-path attribute depends on your position within the Internet. Core operators observe lower AS-path lengths than the edge points. Due to CSCdr54230, accepting AS-paths having more than approximately 100 AS numbers was best avoided; reasonable values are usually much lower… and as always, Geoff Huston published a measurement giving you the answer to that question.

Configuring the bgp maxas-limit command does not impact the regular BGP operation. The maxas-limit is checked during the inbound update processing. Prefixes with oversized AS-path length are simply ignored; BGP sessions are not disrupted.

Test bed description

The bgp maxas-limit functionality can be easily demonstrated in a test bed consisting of only two routers.

hostname R1
!
ip cef
!
interface Loopback0
 ip address 10.0.1.1 255.255.255.255
!
interface Serial1/0
 description Link to R2 s1/0
 ip address 10.0.7.13 255.255.255.252
 encapsulation ppp
!
router bgp 65000
 no synchronization
 bgp log-neighbor-changes
 network 10.1.1.0 mask 255.255.255.0
 network 10.1.2.0 mask 255.255.255.0
 neighbor 10.0.7.14 remote-as 65100
 neighbor 10.0.7.14 route-map prepend out
 no auto-summary
!
ip classless
!
ip route 10.1.1.0 255.255.255.0 Null0
ip route 10.1.2.0 255.255.255.0 Null0
!
ip prefix-list prepend seq 5 permit 10.1.2.0/24
!
route-map prepend permit 10
 match ip address prefix-list prepend
 set as-path prepend 65000 65000 65000 65000
!
route-map prepend permit 20
!
line con 0
 exec-timeout 0 0
 privilege level 15
 logging synchronous
 transport preferred none
 stopbits 1
!
ntp logging
end 

hostname R2
!
ip cef
!
interface Loopback0
 ip address 10.0.1.2 255.255.255.255
!
interface Serial1/0
 description Link to R1 s1/0
 ip address 10.0.7.14 255.255.255.252
 encapsulation ppp
!
router bgp 65100
 no synchronization
 bgp log-neighbor-changes
 bgp maxas-limit 3
 neighbor 10.0.7.13 remote-as 65000
 no auto-summary
!
line con 0
 exec-timeout 0 0
 privilege level 15
 logging synchronous
 transport preferred none
 stopbits 1
!
ntp logging
end 

Exception logging

The bgp maxas-limit functionality does not impact the regular BGP operation. Whenever an inbound BGP update is received with an oversized AS-path attribute, the router logs a warning message and ignores the update.

%BGP-6-ASPATH: Long AS path 65000 65000 65000 65000 65000
 received from 10.0.7.13: More than configured MAXAS-LIMIT 

The AS-path length limiting functionality can also be observed with any of the debug ip bgp update commands. A sample printout is included below:

BGP(0): 10.0.7.13 rcv UPDATE w/ attr: nexthop 10.0.7.13, origin i,
  metric 0, originator 0.0.0.0, path 65000 65000 65000 65000 65000,
  community , extended community , SSA attribute
BGPSSA ssacount is 0
BGP(0): 10.0.7.13 rcv UPDATE about 10.1.2.0/24 -- DENIED due to:
  AS-PATH length over 4072;
BGP(0): 10.0.7.13 rcvd UPDATE w/ attr: nexthop 10.0.7.13, origin i,
  metric 0, path 65000
BGP(0): 10.0.7.13 rcvd 10.1.1.0/24...duplicate ignored 

When I was discussing the details of the BGP troubleshooting video with one of my readers, he pointed out that I should mention the need for CEF switching in EBGP multipath scenario. My initial response was “Why would you need CEF? EBGP multipath is older than CEF” and his answer told me I should turn on my gray cells before responding to emails: “Your video as well as Cisco’s web site recommends CEF for EBGP multipath design… but interestingly, it does work without CEF”.

The real reason we need CEF in EBGP load sharing designs is the efficacy of load distribution. Without CEF, the router will send all traffic toward a single BGP prefix over one of the links (fast switching performs per-destination-prefix load sharing). With CEF, the load is distributed based on the source-destination IP address pair combinations. Even if multiple clients send the traffic toward the same server, the load is spread across available links.

Just stumbled across this: Amazon is offering the MPLS and VPN Architectures, MPLS and VPN Architectures, Volume II and Internet Routing Architectures (2nd Edition) (Networking Technology) (from Sam Halabi) for a total of $160.

read more see 3 comments

A week ago I've asked whether anyone has a good use for BGP aggregation. Out of 1300 daily visitors of my blog and 2200 subscribers to the RSS feed,  four (4) people found the time and energy to reply. Thank you for your thoughts!

Let me conclude that BGP aggregation is not widely used (so I will spend much energy covering it in my future posts) ... all the other potential conclusions are too pessimistic.

One of the presentations at the recent Defcon 16 event demonstrated how you can use the very common laziness of the Internet Service Providers to hijack any prefix you want (just ask YouTube). Nothing new so far, but the part where they fake the AS path in the hijacked announcement to create a safe (hijack-free) conduit back to the destination is brilliant ... and the TTL manipulation is the icing on the cake.

Of course this is a well-known BGP vulnerability (actually, implementation sloppiness on the part of ISPs) that we've been writing about for a long time, but the Defcon presentation is probably the first documented step-by-step recipe for a realistic MITM attack.

Hat tip to Jeremy Stretch; I found the link to the Defcon presentation on his blog.

Establishing parallel EBGP sessions across parallel links between two edge routers (EBGP peers) – as displayed in the diagram below – is the most versatile form of EBGP load balancing. It does not require static routing or extra routing protocol (like the design running EBGP between routers’ loopback interfaces), device-specific tricks like configuring the same IP address on multiple interfaces) or specific layer-2 encapsulation (like Ethernet LAG or Multilink PPP).

It even allows proportional load-balancing across unequal-bandwidth links and combinations of various layer-2 technologies (for example, load-balancing between a serial line and an Ethernet interface). The only drawback of this design is the increased size of the BGP table, as every BGP prefix is received from the EBGP neighbor twice.

For the last 10 years, I've been preaching that you should use static BGP prefix advertising (with the network mask router configuration command) to advertise your IP address space into the public Internet, not the BGP aggregation. I might see some use for BGP aggregation in enterprise networks (or MPLS VPN networks) using BGP as the core routing protocol with other routing protocols serving the edge, but I cannot find a good scenario where BGP aggregation in public Internet would be a good solution. Do you use BGP aggregation in your network? Do you have a good scenario that you'd like to share with us? Write a comment.

BGP route reflectors have been supported in Cisco IOS well before I started to develop the first BGP course for Cisco in mid 1990s. It’s a very simple feature, so I was pleasantly surprised when I started digging into it and discovered a few rarely known details.

The Basics

Route reflector is an IBGP feature that allows you to build scalable IBGP networks. The original BGP protocol (RFC 1771) contained no intra-AS loop prevention mechanism; routers were therefore prohibited from sending routes received from an IBGP peer to another IBGP peer, requiring a full-mesh of IBGP sessions between all BGP routers within an AS.

EIGRP was always described as the only routing protocol that can do unequal-cost load sharing. As it turns out, BGP is another one (although it's way more limited than EIGRP). For example, if you have two links into a neighbor AS, you can load-share across them proportionally to their bandwidth.

EBGP load balancing was introduced with the BGP 4 Multipath Support feature in IOS release 11.2. Initially, EBGP supported up to six maximum paths; IOS release 12.0(S) increased that value to 8, IOS release 12.3T to 16 and 12.2S (including 12.2SRC) to 16.

If you want to display any IPV6-related BGP objects (neighbors, routes …) you can use the familiar BGP commands, but have to prefix them with show ip bgp ipv6 unicast. For example, to display the BGP neighbors active in the IPv6 address family, you would use show ip bgp ipv6 unicast summary command. I doubt you like so much typing (I don't, just entering the IPv6 addresses is enough for me); luckily Cisco IOS has aliases - just configure alias exec bgpv6 show ip bgp ipv6 unicast and (for consistency) alias exec bgpv4 show ip bgp ipv4 unicast.

Update 2010-03-12: Cisco IOS also supports show bgp ipv6 unicast command, which (at least) makes BGP ipv4-agnostic.

Last week I've described how you can extend the exec-mode CLI commands with almost no knowledge of Tcl. A bit more work is required if your commands include Tcl special characters (quotes, braces or backslashes).

For example, to display all routes advertised by customers of AS X, you'd use the following show command: show ip bgp regexp _X_([0-9]+)(_\1)*$ (the regular expression is explained in the AS-path based filter of customer BGP routes post). This command cannot be entered as a Tcl string with variable substitution; Tcl would interpret the [ and \ characters. You could enter the whole command in curly braces, but then there would be no variable substitution that we need to insert command line parameters. To make Tcl happy, use the following Tcl commands:

1. set cmd {first-part-of-command} stores the command prefix into the cmd variable;
2. append cmd $argv appends the command line arguments to the command;
3. append cmd {rest-of-command} appends the rest of the IOS exec command;
4. puts [exec $cmd] executes the command and prints the results.

For example, the following code will display the customers of a BGP AS specified in the command line (after being stored in a flash file and defined in an alias, of course):

set cmd {show ip bgp regexp _}
append cmd $argv
append cmd {_([0-9]+)(_\1)*$}
puts [exec $cmd]