Blog Posts in May 2008
- I was wrong.
- I'm really impressed.
The rootkit presentation prompted Cisco to generate an excellent document describing how to detect patched IOS images and the precautions you can take to ensure an intruder does not get access to your devices.
On the other hand, I was bitterly disappointed by the lack of coverage from the "industry press". There was speculation that Cisco released three patches in anticipation of the presentation (anyone who looked into what those patches were would easily find out that two of them were not IOS related) and a few notable exceptions correctly describing the situation, but some publications that were very loud before the presentation forgot to tell their readers that the threat was "slightly" over-rated. Of course, the lack of interest in non-sensational news has already started conspiracy theories.
If you want to have more details, read a down-to-earth description of the presented rootkit by Nicolas Fischbach.
If you'd like to review the e-lesson and write/blog about it and the concept of short e-learning modules covering very specific technologies or design scenarios, send me a message describing what you'd like to do and I'll organize the access to the e-lesson for you.
What really surprised me was the very large percentage of users using BGP. Either BGP is becoming very popular in the enterprise networks or most of the readers are coming from Service Provider environments.And last but not least, it looks like I'm focusing on the topics (BGP and OSPF) that you actually use in your networks (or maybe it's a self-fulfilling prophecy and you read my blog because I write about topics that you're interested in :).
If anyone has discovered a reliable technique that detects a keypress event (= character available on stdin) in the Tcl loop, please let me know. The Ctrl/C solution is a kludge.
Can you please recommend me literature explaining the architecture of Cisco routers and switches (buffers, control-plane, forwarding-plane, process switching …)The best one I've found so far is the Inside Cisco IOS Software Architecture, but it's a bit old, so if you've found something better, please comment.
Full disclosure: if you click on the link above and buy the book, I might eventually get $1.76 from Amazon.
The presentation part of this e-lesson describes NTP basics and IOS implementation details. The hands-on remote exercise enables you to configure multi-tier NTP implementation on Cisco routers.
I have seen the post on running OSPF across a PIX firewall. Since I did not have a PIX, I tested the solution by replacing PIX with a router.
I had configured the neighbor statements on both routers, but the OSPF was failing to come up. The debug indicated that the router emulating PIX was sending time exceeded ICMP to both OSPF-speaking routers.
The OSPF hello by default has a TTL of 1 which I think is an issue with this scenario. Is there anything special thats done on PIX to get OSPF working?
The answer is quite simple: PIX is not behaving like a router, but rather like a bridge with additional IP features (NAT and traffic filters). It does not decrement the TTL of a transit packet (which could lead to interesting loops if you badly mess up a redundant topology) … and I have to congratulate Sharath for an excellent diagnosis of the problem.
If a customer already owns provider-independent addresses but does not have a registered AS number, it can use a private AS number to establish BGP peering with the ISP to which it’s multi-hoped.
I obviously had some very bad experiences in the past and my subconsciousness has expressed itself :)
- The router and the cable modem are power-cycled.
- The router starts to bridge between all LAN interfaces, effectively connecting inside workstations directly to the cable modem.
- One of the workstations could detect a LAN failure (due to router reload) and restart the DHCP process (a Windows XP host would definitely do that).
- The DHCP requests from the workstation are bridged straight to the cable modem which caches the workstation's MAC address and forwards the DHCP request.
- The workstation is assigned a public IP address (at this time, the workstation is connected directly to Internet and thus vulnerable).
- The router loads Cisco IOS and reinitializes the Ethernet interfaces. Bridging between internal and external interfaces is stopped.
- The router sends DHCP request on the outside interface, but the modem ignores it, as the MAC address of the DHCP request differs from the previously cached one.
In most cases, the cable modem has to be power-cycled to lose the cached MAC address.
This behavior can be observed only if the router and the cable modem are reset at the same time and the cable provider doesn't care much about MAC security and allows the modem to learn the MAC address. If you reset only the cable modem, the router is not bridging (no problem); if you reset just the router, the cable modem still caches the router's MAC address and ignores the DHCP request from the inside workstation(s).
Would it be possible to explain in detail a scenario where dual-as and as-overide is being used and another scenario where dual-as (using no-prepend / replace-as), as-overide and remove-private-as are used?I decided to start from the last item. The neighbor remove-private-as option is used in scenarios where you run BGP between public and private AS numbers to collect IP prefixes and advertise these prefixes to the rest of the world as belonging to the public AS. The most common design in this category is multihoming to a single ISP.
However, the Cisco's response to this announcement (which was basically saying "we haven't seen anything new yet") included a nice gem: a link to the Cisco Guide to Harden Cisco IOS Devices document.
I was writing the June IP Corner article and needed to set up DNS servers within the lab. I used example.com as the domain name and decided to check what would happen if you'd visit the actual www.example.com web site (try it out). It politely referenced me to RFC 2606, which documents the reserved domain names you can use.
As a rule, you should use private IP addresses, AS numbers and domain names in all technical documentation you're producing (unless, of course, you're describing an actual network). If you're forced to use public addresses or AS numbers (for example, to illustrate how the neighbor remote-private-as command works), you should clearly state that the AS numbers are imaginary.
Use the latest 12.4T software (at least 12.4(15)T5) if you want reliable CPPr operation.
- control-plane aggregate service-policy disables any control-plane subinterface service policies.
- If you want to use the per-subinterface (host, transit and cef-exception) policies, you have to remove the inbound service policy from the control-plane aggregate path.
- Routed packets that cannot be CEF-switched (have to be punted to another switching mechanism) are classified as transit packets.
- Local multicast packets with destination IP addresses within IP prefix 22.214.171.124/24 and packets with TTL <= 1 are classified as transit packets in 12.4(15)T5. These packets will be classified as cef-exception packets in the future (see the Understanding CPPr document).
- Unicast packets without IP options addressed to the router and having TTL > 1 are classified as host packets.
- Non-IP traffic (ARP, Frame Relay keepalives, CDP ...) is classified as cef-exception.
The TTL-related rules explain why the router classifies IBGP packets as host packets and EBGP packets as transit packets. As soon as you configure neighbor ebgp-multihop on the router router, inbound EBGP packets become host packets.
The situation has probably changed over the last years, I would (sadly) expect EIGRP to decline and (happily) BGP to grow. Let's figure it out; please respond to this week's readers' poll. Of course you can choose more than one routing protocol.
Our security experts have replicated the behavior and reported it to Cisco PSIRT. Fortunately it's a known vulnerability, documented as CSCsd60259 (release note is available on CCO to registered users) and fixed with a ROMMON upgrade.
New routers are shipped with new ROMMON version, so you shouldn't be seeing this behavior on brand new boxes … but one cannot help but wonder why such a nasty behavior was not documented as a field notice/security advisory.
The search results produced a few very interesting links, among them a well-structured presentation on RTBH that refers to a paper describing how you can detect remote DoS attacks with the backscatter analysis (assuming the attackers are randomly spoofing source IP addresses).
By now you should be asking yourself “Well, so what's the latest and greatest X.25 feature?” Brace yourselves: 12.4(15)T has introduced support for X.25 accounting and the Call Detail Records (CDRs) are generated as syslog messages.
- You're creating a multi-AS BGP test lab on Sunday evening;
- The core AS is running 12.2SRC code;
- You insert a P-router in the core network ... because every large network has P-routers;
- You create BGP session templates instead of configuring two parameters of a few IBGP neighbors;
- You configure MPLS in the core network instead of using BGP on all routers ... because it saves you a few BGP sessions ... and that's the way things should be done anyway;
- When configuring OSPF, you define inter-AS links as passive interfaces ... not because you're running OSPF in the other AS but for security reasons :)
- ... add your comment here ...
We've always been trying to minimize asymmetric routing, in both design and implementation phase, as it impacts a number of IP services/features, including:
- Network Address Translation;
- Content-based Access Control (CBAC);
- Reflexive access lists;
- Redundant firewalls (at least until recently);
- IP Multicast;
In some scenarios, asymmetric routing can impact delay/jitter and consequently the perceived quality of service.
However, asymmetric routing is a reality within the Internet (it's close to impossible to guarantee symmetric routing even for multi-homed end users) and it might even help in some scenarios (low-speed/low-delay upstream link with high-speed/high-delay downstream link).
What's your opinion? Is asymmetric routing harmful? Should we strive to avoid it ... or do you just accept it as one of facts of life?
If “policy routing” reminds you of nightmarish spaghetti solutions of hop-by-hop route-maps, read my Scalable Policy Routing article. It describes how you can use a distance vector routing protocol (BGP is used in the article, but you could do the same with EIGRP or even RIP) to solve common policy routing problems in a highly scalable way.
I wanted to implement a mechanism that would automatically (using EEM) block unstable OSPF neighbors. Once you identify the neighbors to block (this should be the hard part), blocking them is easy if you're running point-to-point interfaces (you just make the interface passive), but blocking a single neighbor on a multi-access interface is a royal pain. I didn't want to use the access lists, as it would be very hard to integrate OSPF-specific filters with existing incoming access-lists configured on the interfaces. Control Plane Protection looked like the ideal tool to use; if I could drop certain inbound IP packets (OSPF hello packets) based on their source IP address (= unstable neighbor) and IP protocol (= OSPF), they would never get to the OSPF process and the adjacency would not form, resulting in a more stable network.
Update: After the comment from William Chu, I've tested 12.4 mainstream release. OSPF is blocked as configured. Next I've re-read the documentation … and found that one of the documented restrictions is that the host subinterface only filters UDP and TCP traffic. Configuring the service policy on the aggregate path (the control-plane keyword with no options) worked.
Before trying to figure out the integration between the SYSLOG messages and router configuration changes, I performed an easy test: I tried blocking all OSPF traffic in the host control-plane (the one controlling packets received by the IOS processes) with the following configuration:
class-map match-all BlockOSPFHowever, according to the show commands, the service policy did not identify any packets as belonging to the BlockOSPF class and the OSPF adjacencies were not affected:
match access-group name BlockOSPF
ip access-list extended BlockOSPF
permit ospf any any
service-policy input ControlPlane
C1#show policy-map control-plane hostAfter a few more tests, I had to conclude that the Control Plane Protection using host subinterface does not work on OSPF packets (and it
Control Plane Host
Service-policy input: ControlPlane
Class-map: BlockOSPF (match-all)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: access-group name BlockOSPF
Class-map: class-default (match-any)
5 packets, 400 bytes
5 minute offered rate 0 bps, drop rate 0 bps
C1#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
10.0.0.12 1 FULL/DR 00:00:30 10.0.1.2 FastEthernet0/0
10.0.0.2 0 FULL/ - 00:00:33 10.0.0.2 Serial1/0.101
10.0.2.2 0 FULL/ - 00:00:33 10.0.0.1 Serial1/0.100
I'm reading your book MPLS and VPN Architectures and I've found the ip vrf forwarding name fallback global command in the “Additional Lookup in the Global Routing Table” section. I can only find this command in Junos, but not in IOS.
… and he was right. When we were writing the book, we described several features that were still in development as it looked like they would be in the production code by the time the book was published. Many of them made it into the public IOS releases (for example, the Carrier's Carrier architecture), but some of them (like this command) simply vanished from the surface.
However, it looks like the engineers that switched from Cisco to Juniper took the concept with them and implemented it in JunOS, so JunOS has this feature but IOS doesn't.