linux.gen.nz

Writing this up because it was so hard to find information about on Google

I’ve had a problem recently where a Cisco router would refuse to pass fragmented IP packets. The initial packet with ‘more fragments’ set would make it through ok, but the subsequent fragment(s) making up the remainder of the original packet would get eaten. Nothing in the log on the router showed the traffic being dropped, but removing the ACL from the interface would cause the traffic to start flowing as normal. It turns out that the release of IOS we were using had a bug which would cause non-initial fragments to silently be dropped if the ACL parser hit an ‘evaluate’ statement in an access list. Because the router in question had reflexive ACLs configured, the evaluate statement was used in a bunch of places.

The solution appears to be upgrading the router to IOS 12.4(21), but a workaround is to move all the evaluate statements to the very bottom of the ACL, above the final ‘deny ip any any’.

Here’s a simple config that reproduces the bug:

!
interface FastEthernet0/0
ip address 172.31.255.230 255.255.255.224
ip access-group e-in in
duplex auto
speed auto
!
interface FastEthernet0/1
ip address 172.31.255.194 255.255.255.224
duplex auto
speed auto
!
ip route 0.0.0.0 0.0.0.0 172.31.255.209
!
ip access-list extended e-in
permit ip 172.31.255.224 0.0.0.31 any log-input
deny ip any any log-input
ip access-list extended e-in-r
evaluate rfx-in
permit ip 172.31.255.224 0.0.0.31 any log-input reflect rfx-out
deny ip any any
ip access-list extended e-in-r2
permit ip 172.31.255.224 0.0.0.31 any log-input reflect rfx-out
evaluate rfx-in
deny ip any any
!

When the ‘e-in’ ACL is being used, traffic from FastEthernet0/0 to FastEthernet0/1 is passed fine. When I send in a 4000-byte UDP packet with hping3 (hping3 172.31.254.1 -2 -p 1000 -d 4000 -c 1), tcpdump shows the 1514-byte initial fragment, followed by a 1514-byte and 1082-byte followup fragment coming out the other side of the router. These tests were performed on a router running IOS 12.4(17):

root@kohr-ah:~# tcpdump -eni eth0 host 172.31.255.226 and udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
22:44:11.302836 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1514: 172.31.255.226.2343 > 172.31.254.1.1000: UDP, length 4000
22:44:11.304585 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1514: 172.31.255.226 > 172.31.254.1: udp
22:44:11.305746 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1082: 172.31.255.226 > 172.31.254.1: udp

3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@kohr-ah:~#

When the e-in-r ACL is used, however, only the initial fragment makes it through:

root@kohr-ah:~# tcpdump -eni eth0 host 172.31.255.226 and udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
22:45:26.410104 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1514: 172.31.255.226.1581 > 172.31.254.1.1000: UDP, length 4000

1 packets captured
1 packets received by filter
0 packets dropped by kernel
root@kohr-ah:~#

Upgrading the router to IOS 12.4(21) causes the e-in-r ACL to pass traffic as expected. As described above, moving the ‘evaluate’ line to the bottom of the ACL is also an effective workaround. The e-in-r2 ACL also passes all the fragments as it should:

root@kohr-ah:~# tcpdump -eni eth0 host 172.31.255.226 and udp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
22:46:26.830418 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1514: 172.31.255.226.1657 > 172.31.254.1.1000: UDP, length 4000
22:46:26.830431 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1514: 172.31.255.226 > 172.31.254.1: udp
22:46:26.830435 c8:00:83:10:00:01 > 00:0c:29:3e:b0:4a, ethertype IPv4 (0×0800), length 1082: 172.31.255.226 > 172.31.254.1: udp

3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@kohr-ah:~#