I was recently watching BRKDCT-2333 - Data Center Network Failure Detection and, after going through the usual suspects - L1 (carrier loss, link signaling), L2 (LACP, UDLD, CFM/Link-OAM), L3 (protocol keepalives, BFD) - the presenter talks about BFD over EtherChannel. But not only for node protection (see below), but for link protection as well, running micro-BFD sessions on each individual EtherChannel member. After understanding how it is done, I started wondering what's the point (tangible benefit) of using this feature?
Running BFD over an EtherChannel bundle
A single BFD session, like any other IP flow, will be polarized and go through only one link out of the bundle. This essentially masks the failure of any other member of the EtherChannel until LACP takes the faulty link out of the bundle. BFD could be tuned to detect a failure much faster than LACP, but in this case L3 forwarding would continue via this faulty link that's not protected by BFD because from its point of view the node at the other end is still alive (responding to BFD packets on the healthy link).
This sounds like a problem that needs to be solved right? Well, I don't know about that, but let me tell you about the solution anyhow.
Should be no surprise that the solution is to run individual BFD sessions on each EtherChannel member link, but aggregate them at the supervisor level into one master session that communicates with the routing protocol. Cisco did some proprietary magic and then the IETF came up with RFC7130 which standardizes this whole thing so vendor interop can be a thing.
But why?
Let's get to the main question of this article... why do I actually need this? If you already have a couple examples in mind, please pause reading and write them down (in the comments below), as I'd really love to know what they are!
My initial reaction was: why do you need BFD on an EtherChannel bundle which should consist of point-to-point links that get taken down very quickly by L1 failure detection mechanisms?
Let's look at the RFC7130 introduction, which states:
Currently, the Link Aggregation Control Protocol (LACP) is used to detect failures on a per-physical-member link. However, the use of BFD for failure detection would (1) provide a faster detection, (2) provide detection in the absence of LACP, and (3) would be able to verify the ability for each member link to be able to forward L3 packets.
and
the solution defined in this document enables operators who have already deployed BFD over different technologies (e.g., IP, MPLS) to use a common failure detection mechanism.
(1) OK, so faster detection than LACP, yes that's something, but only when L1 can't provide it.
(2) Absence of LACP - I'll leave this for those that really hate their networks and operations teams.
(3) Hmmm, a fault that allows Ethernet frames to flow, but not IP packets? Leave a comment if you have an example, I'm quite curious.
I wasn't very convinced, so I went and asked a bunch of smart people about it... and these are their ideas:
- It could be that there's a "bump in the wire" type device, breaking the point-to-point link assumption. This could fail closed on one side only, taking the link down for one device, but leaving it up for the other, which would indeed wait for LACP to time out.
- I should've thought of this as I've seen devices like this doing in-line DPI - never seen them fail, so it's anybody's guess if the scenario above applies or not.
- DWDM without end-to-end link loss signaling - maybe I've been lucky, but I've always had this feature and would be wary of provisioning links without it.
- A device crash where links stay up - essentially control-plane stops responding, but I've argued that in this case the EtherChannel is affected as a whole and a single BFD session is sufficient (node protection).
- Some platforms do BFD in hardware while LACP is handled by the CPU - so basically this would alleviate some control-plane scaling concerns.
- Coolness factor and job security ;)
In the end, this feature/RFC exists because multiple networks out there had a real need for it. Like with any feature that affects a design, it's important to understand its functionality, limitations and benefits before throwing it in because it's just cool.
And, as always, thanks for reading
For more detailed information on network failure detection watch BRKDCT-2333 or read Daniel's excellent notes.