Category: BGP

Petr Lapukhov describes an interesting scenarion in his post BGP Time-Based Policy Routing: a multi-homed customer that uses one upstream link (for example, more reliable but slower one) during the work hours, switching to the other upstream link (faster, less reliable) after that.

He uses BGP communities to achieve the switch (perfect solution if your ISP supports them) and time-based ACL in a route-map to set the community based on time-of-day. As Cisco changed the way BGP imports local routes in IOS release 12.3T, he then devises an ingenious solution based on reliable static routing to trigger a change in the IP routing table.

The optimum solution is way simpler: you just configure two EEM applets to perform clear ip route network command at appropriate times.

BGP implementation on Cisco IOS gives you a number of filtering options, including prefix filters, AS path filters and route maps. While it might be tempting to learn just the most versatile tool available (route maps) and discard all the others, judicious use of all available tools can simplify your router configurations.

For example, an Internet Service Provider might want to filter incoming updates received from the customers to ensure they’re not advertising transit routes and advertise only IP prefixes they actually own. Inbound route maps might also be needed to attach BGP communities to inbound routes or set BGP attributes (for example, local preference) based on communities attached to incoming routing updates.

You can perform all these tasks with route-maps, but then you’d probably have to create a dedicated route-map for each customer (as the inbound prefix filter has to be customer-specific). Changing your routing policies or community definitions would require changing a lot of route maps.

Update 21-feb-08@15:39: distribute lists and prefix lists can't coexist (they cannot be configured in the same direction on the same neighbor)

On the other hand, if you use all the filters available in the BGP routing process, you could:

• Use neighbor filter-list in to check the customer-specific AS path requirements;
• Use neighbor prefix-list in to filter customer prefixes, reject too long prefixes or prefixes not belonging to the customer;
• Use neighbor route-map in to drop prefixes belonging to your own address space, implement various routing policies and set BGP communities.

The solution scales even better if you configure common filters (route-map in and filter-list in in our scenario) in a BGP peer session template.

Any serious (or at least security-aware) ISP should not blindly accept BGP routes from its customers but at the very minimum do some sanity checks on them. For example, if a multi-homed customer is clumsy enough to advertise BGP routes between service providers, it’s nice if you still stop him from turning into a transit AS. The required filter is conceptually quite simple: all the BGP routes from the customer should contain only his AS number in the AS-path.

The initial non-scalable approach is obvious: accept only the AS paths that have exactly the customer’s AS number in the AS path. For example, if your customer’s AS number is 65001, you could use this filter: ip as-path access-list 100 permit ^65001$.

Enterprise networks primarily use BGP with their Internet Service Providers if they want to be multi-homed (connected to more than one ISP). A very common requirement in a multi-homed design is the primary/backup setup where the lower speed (or sometimes lower quality) link should only be used when the primary link fails.

Competent ISPs help their customers reach this goal by using BGP local preference within their network and giving the customers the ability to indicate the desired value of BGP local preference through BGP communities: if the route received directly from the customer has low local preference, all other routes are preferred, resulting in the desired traffic flow that avoids the backup link if at all possible as shown in the next diagram:

BGP communities are extra attributes you can attach to an IP route carried by BGP. You can use communities to indicate which routes should be propagated or filtered (for example, the well-known NO_EXPORT community signifies that the route it’s attached to shall not be sent outside of the local AS), to influence route selection on remote routers or to trigger other BGP-dependent IOS features (for example, quality-of-service marking based on BGP).

Each BGP community is a 32-bit value. The best practice dictates that the top 16 bits should be the AS number of the network defining the community meaning and the bottom 16 bits are defined by the network administrator.

I'm often promoting the idea of separating customer routing from core routing in the design articles I write. The only viable solution (unless you want to implement MPLS VPN and migrate customer routing into VPNv4) is to carry customer routes in BGP, redistributing them into BGP from other routing sources. On the other hand, I’m telling you that you should advertise only static IP prefixes into the public Internet. Obviously there’s a seeming disconnect between the two advices.

However, the dilemma is easily solved with the no-export BGP community that prevents an IP prefix from being advertised over EBGP sessions. Whenever you redistribute customer routes into BGP, you should attach the no-export community to them, ensuring that only the statically advertised IP prefixes will be propagated outside of your AS boundaries.

Configuring a large number of similar BGP peers on a router and ensuring that the changes in your routing policy or BGP design are applied to all of them can be a management nightmare. BGP peer groups were the only scalability tool available on Cisco IOS until the IOS release 12.3T and they had significant limitations as they were also used as a performance improvement tool.

IOS releases 12.0S and 12.3T introduced peer templates, a scalable hierarchical way of configuring BGP session parameters and inbound/outbound policies. For example, to configure the session parameters for all your IBGP sessions, use the following session template:

router bgp 65001
 template peer-session IBGP
  remote-as 65001
  description IBGP peers
  password s3cr3t
  update-source Loopback0

After the session template has been configured, adding a new IBGP peer takes just a single configuration command (two if you want to add neighbor description):

router bgp 65001
 neighbor 10.0.1.2 inherit peer-session IBGP
 neighbor 10.0.1.2 description R2

Policy templates are similar to session templates, and contain neighbor parameters that influence processing of prefixes of an individual BGP address family (example: filtering of inbound updates).

Continuing the IBGP example, you might want to group route reflector clients in a policy template, and ensure the route reflector propagating all BGP communities to them:

router bgp 65001
 template peer-policy Internal
  route-reflector-client
  send-community both
 exit-peer-policy

After defining a policy template, you can apply it to multiple address families, for example:

router bgp 65001
 neighbor 10.0.1.2 inherit peer-session IBGP
 neighbor 10.0.1.2 description R2
!
 address-family ipv4
  neighbor 10.0.1.2 activate
  neighbor 10.0.1.2 inherit peer-policy Internal
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.0.1.2 activate
  neighbor 10.0.1.2 inherit peer-policy Internal

The routing information you source into the public Internet with BGP should be as accurate and stable as possible. The best way to achieve this goal is to statically configure the IP prefixes you’ve been allocated on your core routers and advertise them into BGP:

• BGP will only advertise an IP prefix if a matching entry is found in the IP routing table. To ensure the IP prefix you want to advertise is always present, configure an IP static route to null interface, unless you're advertising a connected interface (example: Internet edge router on a DMZ segment).
• Most public IP prefixes advertised today do not fall on the classful network boundary. To advertise a classless prefix, you have to configure the prefix and the mask in the BGP routing process.

Internal BGP (IBGP) sessions (BGP sessions within your autonomous system) are identified by the neighbor’s AS number being identical to your AS number. While the external BGP (EBGP) sessions are usually established between directly connected routers, IBGP sessions are expected to be configured across the network.

The current best practice is to configure IBGP sessions between the loopback interfaces of the BGP neighbors, ensuring that the TCP session between them (and the BGP adjacency using the TCP session) will not be disrupted after a physical link failure as long as there is an alternate path toward the adjacent router.

Sometimes you end up having wrong BGP AS number throughout your network. It could be a result of an unexpected merger or split or you could have started using a private BGP AS number and realized you have to connect to the Internet using a real AS number. The proper solution would be a total reconfiguration of the whole network, but of course not many engineers have the time and courage to do it ;), so it's time to introduce another kludge: the neighbor local-as configuration command.

You've probably already heard the phrase "When the only tool you have is a hammer, everything looks like a nail" (and seen people acting according to it). Likewise, if you have an IOS release with EEM support, a lot of things that would require smart design could be solved in a brute-force way with a few EEM applets. For example, the problem of the BGP default route could be solved “easily” with a few applets that track syslog messages reporting when the BGP neighbors go up/down.

Sometimes you'll see a weird route status (RIB-failure) in your BGP table, for example:

GW#show ip bgp ¦ include r>
r> 10.2.0.0/16      10.0.1.2                 0             0 65001 i

A more thorough investigation of the BGP entry does not give you a lot of additional information:

GW#show ip bgp 10.2.0.0
BGP routing table entry for 10.2.0.0/16, version 7
Paths: (1 available, best #1, table Default-IP-Routing-Table, RIB-failure(17))
Flag: 0x820
  Advertised to update-groups:
        1    2
  65001
    10.0.1.2 from 10.0.1.2 (10.0.1.2)
      Origin IGP, metric 0, localpref 100, valid, external, best

The “mistery” is solved when you inspect the entry in the IP routing table:

GW#show ip route 10.2.0.0
Routing entry for 10.2.0.0/16
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1

The GW router has a static route that collides with the EBGP route and thus the BGP route cannot be inserted in the IP routing table (as the static route has administrative distance 1).

Let's conclude with a few interesting facts about the RIB failures:

• The RIB failure feature was introduced in IOS release 12.2T; prior to that, the BGP routes with higher administrative distance than other route sources were silently ignored (similar to all other routing protocols).
• You can display BGP routes that are not inserted in the IP routing table with the show ip bgp rib-failure command, which also explains why the BGP route was not inserted in the IP routing table.
• The BGP routes that are not used due to higher administrative distance are still advertised to all BGP peers (contrary to what most other distance-vector routing protocols do), unless you configure bgp suppress-inactive (introducted in 12.2T and 12.0(26)S).

You might have been there before: the BGP neighbor becomes reachable after you fix a fault in the network, but the BGP session takes “forever” to be established (actually, the hold off is less than a minute, but time is running slower when you are waiting for the network to recover). However, when testing the BGP fast peering session deactivation, I made an interesting discovery: the restart time is improved as well; as soon as the path to the BGP neighbor appears in the IP routing table, the BGP session is established. The debugging printouts from my router are included below (I've used neighbor 10.0.3.3 fall-over configuration command):

03:28:42: RT: add 10.0.3.3/32 via 10.2.0.2, ospf metric [110/75]
03:28:42: RT: NET-RED 10.0.3.3/32
03:28:44: RT: Try lookup less specific 10.0.3.3/32, default 1
03:28:44: RT: Found subnet on less specific 10.0.3.3/32
03:28:44: %BGP-5-ADJCHANGE: neighbor 10.0.3.3 Up

Designing and operating large BGP networks has always been a challenge, as you have to deploy BGP on all core routers and design a hierarchy of internal BGP routers to get around the full-mesh limitation. When MPLS was introduced, it gave us means of deploying BGP only on the network edges, with the core routers carrying just the information about the BGP next hops.

As I know some of you run large networks, could you help me understand what you're using (without giving away too much information, of course):

• Are you running a BGP network without MPLS or are you using BGP on the edges and MPLS transport in the core?
• If you have a large number of BGP routers, do you have a nice hierarchy of BGP route reflectors (or confederations) or ad-hoc implementation where every router has all neighbors as RR-clients?

Full disclosure: I might use the information you give me in an upcoming article.

We all know that BGP is meant to converge slowly… well, the MPLS/VPN service providers tend to disagree, as their users are not used to minute-long convergence times. One of the major components of slow BGP convergence is the time it takes a router to discover that a neighbor has disappeared. Traditionally, the BGP keepalive packets were sent every minute and it took up to three minutes to discover that a neighbor is down.

Of course you could fine-tune those times with the neighbor timers configuration command, but the reduced timers resulted in increased TCP traffic and consequently increased CPU load, which could reach tens of percents if the timers were set to a few seconds and the router had lots of BGP neighbors.