Overload functionality in overlay networks using fault detection protocols

This application claims the benefit of U.S. Provisional Application No. 62/128,880, filed Mar. 5, 2015, the entire contents of which is hereby incorporated by reference as if set forth herein in its entirety.

This disclosure relates to computer networks and, more particularly, to virtual private local area networks.

Networks that primarily utilize data link layer devices are often referred to as layer two (L2) networks. A data link layer device is a device that operates within the second layer of the Open Systems Interconnection (OSI) reference model, i.e., the data link layer. One example of a common L2 network is an Ethernet network in which end point devices (e.g., servers, printers, computers, and the like) are connected by one or more Ethernet switches. The Ethernet switches forward Ethernet frames, also referred to as L2 communications or L2 packets to devices within the network. As the Ethernet switches forward the Ethernet frames, the Ethernet switches learn L2 state information for the L2 network, including media access control (MAC) addressing information for the devices within the network and the physical ports through which the devices are reachable. The Ethernet switches typically store the MAC addressing information in MAC tables. When forwarding an individual Ethernet frame, an ingress port of an Ethernet switch typically broadcasts the Ethernet frame to all of the other physical ports of the switch unless the Ethernet switch has learned the specific physical port through which the destination MAC address devices is reachable. In this case, the Ethernet switch forwards a single copy of the Ethernet frame out the associated physical port.

A virtual private local area network service (VPLS) is one example of an L2 virtual private network (VPN) service that may be used to extend two or more remote customer networks, i.e., VPLS sites, through a layer three (L3) intermediate network (usually referred to as the VPLS core) in a transparent manner, i.e., as if the intermediate network does not exist and the remote customer networks are instead directly connected to one another. In particular, the VPLS transports L2 communications, such as Ethernet packets, between customer networks via the intermediate network. In a typical configuration, provider edge (PE) routers coupled to the customer networks operate as ingress and egress for label switched paths (LSPs) or other tunnels that may be used as pseudowires within the provider network to carry encapsulated L2 communications as if the customer networks were directly attached to the same local area network (LAN). These PE routers may be referred to as “members of the VPLS domain” in that they run a VPLS instance for the VPLS domain and maintain L2 state information for the VPLS service. The PE routers may use either Border Gateway Protocol (BGP) or Label Distribution Protocol (LDP) as the control plane protocol for signaling the VPLS service. While VPLS is an example of a multipoint-to-multipoint service, an L2 virtual circuit or pseudowire is an example of a point-to-point service that may be used to connect two remote customer networks.

In some instances, an Interior Gateway Protocol (IGP) may be run over a pseudowire to provide a seamless private network for the customer. Running an IGP over a pseudowire may establish an overlay network in which two customer networks appear to be connected by a single logical link, where the single logical link is comprised of multiple physical links and PE routers in the service provider network. In some instances, when a PE device is taken offline for maintenance, a service provider may wish to preemptively notify other devices that the PE device is being taken offline to prevent packets from being dropped. However, informing other devices in a network that a PE device is being taken offline may be difficult to achieve in overlay networks.

The techniques described herein are generally directed to reducing or preventing transient black-holing of network traffic in an overlay network. Various aspects of the techniques described in this disclosure may extend network protocols used for fault detection to signal when a PE network device is going to become nonoperational. For example, a PE network device residing in an overlay network may execute a bidirectional forwarding detection (BFD) network protocol extended to signal when the PE network device is going to become nonoperational. The PE network device may generate and transmit a BFD message that includes a diagnostic bit indicating whether or not the PE network device is going to become nonoperational.

A CE network device positioned in a first customer network that utilizes the overlay network may participate in a BFD session with the PE network device and receive the BFD message. The CE network device may execute another instance of the BFD network protocol to detect faults over the logical link between another CE network device in a second customer network. The CE network device may generate and send a BFD message having the diagnostic bit so as to inform the CE network device in the second customer network as to whether or not the PE network device is going to become nonoperational. In this manner, the techniques may enable the CE network devices to become aware of nonoperational and thereby redirect traffic via a different logical link in the overlay network so as to prevent transient black-holing of network traffic in the overlay network.

In one aspect, a method comprises executing, by a first CE network device positioned in a first customer network, an instance of a network protocol to detect faults between the first CE network device and a second CE network device positioned in a second customer network. The method further comprising transmitting, by the first CE network device, a message to the second CE network device via the instance of the network protocol signaling that a provider edge (PE) network device is going to become nonoperational. The PE network device is positioned in an intermediate network providing interconnectivity between the first customer network and the second customer network.

In another aspect, a first customer edge (CE) network device positioned in a first customer network comprises a control unit configured to execute an instance of a network protocol to detect faults between the first CE network device and a second CE network device positioned in a second customer network. The first CE network device may also include an interface configured to transmit a message to the second CE network device via the instance of the network protocol signaling that a provider edge (PE) network device is going to become nonoperational. The PE network device is positioned in an intermediate network providing interconnectivity between the first customer network and the second customer network.

In another aspect, a method comprises executing, by a first CE network device positioned in a first customer network, a first instance of a network protocol to detect faults between the first CE network device and a second CE network device positioned in a second customer network. The method also comprises receiving, by the first CE network device, a message from the second CE network device via the instance of the network protocol signaling that a provider edge (PE) network device is going to become nonoperational. The PE network device is positioned in an intermediate network providing interconnectivity between the first custom network and the second customer network.

In another aspect, a first customer edge (CE) network device positioned in a first customer network comprises a control unit configured to execute a first instance of a network protocol to detect faults between the first CE network device and a second CE network device positioned in a second customer network. The first CE network device may also include an interface configured to receive a message from the second CE network device via the instance of the network protocol signaling that a provider edge (PE) network device is going to become nonoperational. The PE network device is positioned in an intermediate network providing interconnectivity between the first custom network and the second customer network.

In another aspect, a network system comprises a first customer network including a first customer edge (CE) network device, a second customer network including a second CE network device, and an intermediate network providing interconnectivity between the first customer network and the second customer network, the intermediate network including a provider edge (PE) device. The PE network device includes a control unit configured to execute an first instance of a network protocol to detect faults between the PE network device and the first CE network device and an interface configured to transmit a message via the first instance of the network protocol signaling that the PE network device is going to become nonoperational. The first CE network device includes a control unit configured to execute a second instance of the network protocol to detect faults between the first CE network device and a second CE network device positioned in a second customer network, and an interface configured to transmit a message to the second CE network device via the instance of the network protocol signaling that a provider edge (PE) network device is going to become nonoperational.

The details of one or more aspects of the techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques will be apparent from the description and drawings, and from the claims.

FIG. 1 is a block diagram of an example system that includes one or more network devices configured to prevent transient black-holing of traffic in an overlay network in accordance with the techniques described herein.

FIG. 2 is a block diagram illustrating an example CE router 18 A configured to prevent transient black-holing of traffic in an overlay network in accordance with the techniques described herein.

FIGS. 3A and 3B illustrate example link-overload TLVs that may be used to prevent transient black-holing of traffic in an overlay network in accordance with the techniques described in this disclosure.

FIG. 4 illustrates an example BFD message that may be used to prevent transient black-holing of traffic in an overlay network in accordance with the techniques described in this disclosure.

FIG. 5 is flowchart illustrating example operations implemented by multiple network devices to prevent transient black-holing of traffic in an overlay network in accordance with the techniques described herein.

FIG. 6 is flowchart illustrating example operations of a network device that may prevent transient black-holing of traffic in an overlay network in accordance with the techniques described herein.

FIG. 7 is block diagram of multiple network devices that may implement operations prevent transient black-holing of traffic in a broadcast network in accordance with the techniques described herein.

FIG. 1 is a block diagram of an example system that includes one or more network devices configured to prevent transient black-holing of traffic in an overlay network, in accordance with the techniques described herein. FIG. 1 illustrates an example system 8 in which a routed layer three (L3) service provider network 10 is a VPLS core to provide layer two (L2) connectivity between hosts 12 A and 12 B of VPLS sites 14 A and 14 B (“VPLS sites 14 ”). That is, virtual private local area network service (VPLS) may be used to extend L2 connectivity for two or more remote networks, e.g., VPLS sites 14 , through L3 SP network 10 in a transparent manner, as if intermediate service provider (SP) network 10 does not exist. In particular, the VPLS transports layer two (L2) communications, such as Ethernet packets, between one or more host devices 12 A and 12 B (“host devices 12 ”) within VPLS sites 14 via SP network 10 . In a typical configuration, provider edge (PE) routers 16 A- 16 D (“PE routers 16 ”) exchange L2 frames (e.g., Ethernet frames) with customer edge (CE) routers 18 A and 18 B (“CE routers 18 ”). Although shown for purposes of example as CE routers, PE routers 16 may in some examples be coupled to VPLS sites 14 by other devices, such as network switches.

PE routers 16 are interconnected by a set of label switched paths (LSPs) that may be used as VPLS pseudowires within L3 SP network 10 to carry encapsulated L2 communications as if the customer networks were directly attached to the same local area network (LAN). For instance, a VPLS pseudowire 21 A may be configured between PE router 16 B and PE router 16 A. VPLS pseudowire 21 A may be implemented with one or more physical links (and/or other PE routers) that couple PE router 16 B and PE router 16 A. Using VPLS pseudowire 21 A, VPLS sites 14 A and 14 B may appear to be directly attached to the same local area network (LAN). As such, VPLS pseudowire 21 A is shown as a tube or tunnel in the example of FIG. 1 that extends through each of PE routers 16 A and 16 B to indicate that the VPLS extends to the edges of VPLS core 10 .

In BGP-based VPLS networks, BGP is used as the control plane protocol for signaling the VPLS service, but it should be understood that other appropriate protocols may also be used. PE routers 16 that participate in the BGP-based VPLS signaling and maintain L2 state information for the customer sites may be referred to as “members of the VPLS domain.” In the example of FIG. 1 , VPLS pseudowire 21 A is established to carry communications between PE router 16 A and 16 B, and VPLS pseudowire 21 B is established to carry communications between PE router 16 C and PE router 16 D. VPLS pseudowires 21 may be bi-directional pseudowires. Additional details related to VPLS can be found in K. Kompella, “Virtual Private LAN Service (VPLS) Using BGP for Auto-discovery and Signaling,” Request for Comments: 4761, The IETF Trust, January 2007, the contents of which are hereby incorporated by reference in their entirety.

In the example of FIG. 1 , VPLS site 14 A is connected to SP network 10 by a bridged L2 access network that provides redundant physical L2 connectivity to SP network 10 through multiple PE routers 16 A, 16 C via links 26 A and 26 B, a technique which is referred to as “multi-homing.” Specifically, VPLS site 14 A, via CE router 18 A, is multi-homed to SP network 10 through PE routers 16 A and 16 C. Additional details related to multi-homing in BGP-based VPLS can be found in K. Kompella, “Multi-homing in BGP-based Virtual Private LAN Service,” draft-kompella-l2vpn-vpls-multihoming-02.txt, November 2008, which is hereby incorporated by reference in its entirety.

In some examples, multi-homing PE routers 16 A, 16 C may be configured to form a multi-chassis link aggregation group (LAG) for physical access links 26 A, 26 B within a bridged L2 access network that provide L2 connectivity for multi-homed VPLS site 14 A. In one example, each of CE router 18 A and PE routers 16 A, 16 C optionally execute the Link Aggregation Control Protocol (LACP) to bundle physical interfaces associated with access links 26 A, 26 B and treat the bundle as a single logical interface in terms of L2 forwarding. That is, CE router 18 A associates links 26 A, 26 B with a single logical interface for purposes of L2 forwarding to utilize the links in parallel to increase the link speed beyond the limits of any one single cable or port, and to increase the redundancy for higher availability. Moreover, PE routers 16 A, 16 C may form a LAG as a multi-chassis LAG in that physical links 26 A, 26 B do not solely interconnect two network devices but instead provide CE 18 A with connectivity to different network devices, i.e., PE routers 16 A and 16 C. CE router 18 B, PE router 16 B and PE router 16 D may be configured with physical links 26 C and 26 D in a similar manner as described with respect to CE router 18 A, PE router 16 A and PE router 16 C.

In some examples, PE routers 16 and/or CE routers 18 may run an Interior Gateway Protocol (IGP). IGPs use flooding-based distribution mechanisms to announce topology information to routers within the network. These routing protocols may rely on routing algorithms that require each of the routers to have synchronized routing topology information. Examples of IGPs may include Open Shortest Path First (OSPF) and Intermediate system to intermediate system (IS-IS) routing protocols. OSPF and IS-IS are link state protocols that use link state messages to ensure their routing topology is synchronized with respect to available interfaces, metrics and other variables associated with network links. For example, OSPF utilizes Link State Advertisements (LSAs) as link-state messages, while IS-IS uses Link State protocol data units (LSPs) as link-state messages to exchange information.

A router generating a link state message typically floods the link state message throughout the network such that every other router receives the link state message. In network topologies where routers are connected by point-to-point connections, each router floods link state messages to adjacent routers reachable on each interface to ensure synchronization. In networks using multi-access media, such as an Ethernet network, the routers within the network flood the link state messages to all other routers. In either case, the receiving routers construct and maintain their own network topologies using the link information received via the link state messages. IS-IS is specified in “Intermediate system to Intermediate system routing information exchange protocol for use in conjunction with the Protocol for providing the Connectionless-mode Network Service (ISO 8473)”, ISO, ISO/IEC 10589:2002, the entire contents of which is incorporated herein by reference. Further details of OSPF can be found in RFC 2328, Internet Engineering Task Force (IETF), herein incorporated by reference.

Generally, PE routers 16 and CE routers 14 use flooding-based routing protocols to announce topology information to each other and synchronize link-state databases maintained by each of the routers. Link state messages defined by the IGP may include one or more Type, Length, Values (TLVs). A TLV may include one or more fields that each includes respective data, where the data may be processed by a router during a lookup operation. During the lookup operation, the router may perform one or more operations based on the data included in the fields of the TLV.

TLVs may be carried by a variety of different types of link state messages. For example, PE routers 16 and CE routers 14 typically exchange initial link state messages to establish the adjacency relationship. For example, PE routers 16 and CE routers 14 may exchange IS-IS HELLO protocol data units (PDUs) to establish adjacencies with other routers. PE routers 16 and CE routers 14 may include one or more TLVs described herein within such PDUs. Each link state message may be refreshed periodically on the network and is acknowledged by the receiving routers. For example, PE routers 16 and CE routers 14 may utilize periodic IS-IS link state PDUs for synchronizing their link-state databases and utilize partial sequence number PDUs (PSNPs) and complete sequence number PDUs (CSNPs) to acknowledge receipt of the information. PE routers 16 and CE routers 14 may embed the TLV within the link state PDUs, or within the acknowledgement PDUs.

In the example of FIG. 1 , CE routers 18 may run IGPs, such as OSPF or IS-IS, over VPLS pseudowires. Accordingly, in the example of FIG. 1 , CE routers 18 A and 18 B each are each included in the same IGP domain 29 . An IGP domain may include an identifiable set of one or more network devices that each executes an IGP and exchanges link-state messages with other network devices in the same identifiable set. By running IGP over VPLS pseudowires implemented by PE routers 16 , one or more of CE routers 18 may establish an overlay network in which the IGP is “overlaid” or otherwise running on VPLS pseudowires. In such overlay networks, PE routers 16 and physical links between PE routers 16 that are used to implement VPLS pseudowires 21 A and 21 B may not be visible to CE routers 18 that are running IGP. However, VPLS pseudowires 21 A and 21 B may appear as respective links to CE routers 18 A and 18 B within the IGP domain.

In some examples, an administrator that operates service provider network 10 may perform maintenance on one or more of PE routers 16 . To perform maintenance on a PE router, the PE router may be powered down, rebooted, or otherwise taken offline, such that the PE router cannot send and/or receive packets. Accordingly, any packets sent to the PE router, while maintenance is being performed on the PE router, may be dropped, rather than processed and/or forwarded by the PE router. For instance, if PE router 16 A is taken offline for maintenance, any packets sent by CE router 18 to PE router 16 A or PE router 16 B to PE router 16 A may be dropped (which may be referred to as “transient traffic black-holing”).

IGPs, such as OSPF and IS-IS may implement “node-overload” functionality to avoid transient traffic black-holing. For instance, RFC 3137 in OSPF and RFC 3787 in IS-IS, may specify node overload functionality, rather than link-overload functionality as described in this disclosure. Taking RFC 3137 in OSPF as an example, a PE router that is being taken offline for maintenance may announce its router-LSA to its neighboring routers in its IGP domain and indicate that all of the PE router's links are unavailable to send or receive network traffic. However, such techniques only provide for an “overloaded node” (e.g., the entire PE router being taken down for maintenance), rather than allowing the PE router to specify a particular link of the PE router to be overloaded, such that the particular link is made unavailable for sending or receiving network traffic. Moreover, a PE router, using the techniques of RFC 3137 in OSPF and RFC 3787 in IS-IS, must be in the same IGP domain as the neighboring routers or otherwise visible to the neighboring routers in order for the neighboring routers to receive the notifications from the PE router that it is going down for maintenance.

In the case of overlay networks, such as FIG. 1 , where CE router 18 A and CE router 18 B appear to be directly connected when using VPLS pseduowire 21 A, PE router 16 A and therefore PE router 16 B are not visible within the same IGP domain 29 that includes CE router 18 B and CE router 18 A. Because PE routers 16 A and 16 B are not visible within IGP domain 29 that includes CE router 18 A and CE router 18 B, it may not be possible for PE router 16 A to use RFC 3137 in OSPF or RFC 3787 in IS-IS to notify CE router 18 B when PE router 16 A is going down for maintenance. As such, under the RFC 3137 in OSPF and RFC 3787 specifications, PE router 16 A may not have a way to notify CE router 18 B to stop forwarding network traffic on the logical link 30 defined in the IGP domain between CE routers 18 A and 18 B that uses VPLS pseudowire 21 A. In some examples, a logical link may refer to a link that is not defined or identifiable by its physical structure, by rather by a path between two endpoints (e.g., two network devices), where the path may be implemented by one or more physical links and/or network devices.

Although PE router 16 A may be able to notify CE router 18 A that PE router 18 A is going offline for maintenance using a protocol such as Bidirectional Forwarding Detection (BFD) because PE router 16 A and CE router 18 A are directly coupled by a physical link, PE router 16 A may not be able to notify CE router 18 B that PE router 16 A is being taken offline for maintenance. The BFD protocol is defined in RFC 5880, which is herein incorporated by reference in its entirety. While PE router 18 A and CE router 18 A may establish a BFD session that enables PE router 18 A to notify CE router 18 A that PE router 18 A is going offline for maintenance, PE router 16 A may not be able to notify CE router 18 B that PE router 16 A is going offline for maintenance using a BFD session because PE router 16 A and CE router 18 B are not directly coupled by a physical link. As such, under the RFC 3137 in OSPF and RFC 3787 specifications, CE router 16 B would continue to send network traffic to CE router 18 A using a link in the IGP domain that causes traffic to flow on VPLS pseudowire 21 A, although PE router 16 A has been taken offline for maintenance. Consequently, such network traffic may be dropped or black-holed. Techniques are described in this disclosure to prevent transient traffic black-holing in an overlay network when a router, such as PE router 16 A is taken offline for maintenance.

As further described with respect to FIG. 1 , techniques of the disclosure allow CE router 18 B to divert network traffic to alternate PE router 16 D before primary PE router 16 A is taken offline for maintenance, such that rerouting of network traffic may prevent transient traffic black-holing at PE router 16 A. For example, PE router 16 A may initially receive one or more instructions that cause PE router 16 A to commence with maintenance. PE router 16 A may receive the one or more instructions in response to user input from an administrator or automatically as a result of a scheduled or asynchronous event.

In response to receiving the one or more instructions to commence with maintenance at PE router 16 A, PE router 16 A may send a BFD packet to CE router 18 A that includes data usable by CE router 18 A to determine that PE router 16 A is being taken offline for maintenance. The data usable by CE router 18 A to determine that PE router 16 A is being taken offline for maintenance may include a diagnostic bit set in a header of the BFD packet (which may also be referred to as a “BFD message”).

In other words, each of CE router 18 A and PE router 16 A may execute a first instance of BFD, which are shown as BFDs 33 A, to detect faults between CE 18 A and PE router 16 A. As shown in the example of FIG. 1 , BFD session 33 A is executed within a virtual local area network (VLAN) 35 configured between PE router 16 A and CE router 18 A. BFDs session 33 A configured within PE router 16 A and CE router 18 A may periodically send BFD messages (e.g., every 100 milliseconds) to indicate that each of PE router 16 A and CE router 18 A and link 26 A between PE router 16 A and CE router 18 A are operational. When PE router 16 A is to undergo maintenance, BFD session 33 A of PE router 16 A may transmit a BFD message including a diagnostic bit set to indicate that the PE router 16 A will be nonoperational. In this respect, CE router 18 A may receive the BFD message and, upon processing the BFD message, be notified that PE router 18 A is going to become nonoperational.

In other examples, CE router 18 A may determine that PE router 16 A has not sent a BFD packet within a specified time interval, and therefore PE router 16 A is being taken offline for maintenance. Although described with respect to a particular type of network protocol used to detect faults, i.e., BFD in this example, the techniques may be performed with respect to any network protocol capable of detecting network faults.

In any case, CE router 18 A, upon being notified that PE router 16 A is being taken offline for maintenance, may stop forwarding network traffic to PE router 16 A. In particular, CE router 18 A may determine information identifying logical link 30 in its forwarding information and may set the link state of logical link 30 to a “link overload” state. To identify logical link 30 , CE router 18 A may determine which of its interfaces is coupled to PE router 16 A via physical link 26 A, on which CE router 18 A determined that PE router 16 A is being taken offline. CE router 18 A may then determine the identified interface is used to send and receive network traffic for logical link 30 .

CE router 18 A may set the link state of logical link 30 to a “link overload” state by assigning a metric to the logical link 30 that indicates the link is not usable to send or receive any network traffic. Accordingly, in some examples, overloading a link may refer to configuring a router to not use the link to forward and/or receive network traffic. In some examples, the metric may be a value in range of values, where a maximum value in the range of values indicates that the link is not usable to send or receive any network traffic. Therefore, CE router 18 A may set the link state of logical link 30 to a maximum metric to stop sending traffic to PE router 16 A. To continue sending and receiving network traffic with CE router 18 B, CE router 18 A may re-route any network traffic to CE router 18 B using logical link 31 that is present in IGP domain 29 . For purposes of FIG. 1 , network traffic sent and received using logical link 31 is carried by pseudowire 21 B, and network traffic sent and received using logical link 30 is carried by pseudowire 21 A.

CE router 18 A may also advertises the link overload state to other nodes in IGP domain 29 by flooding the information that indicates the link overload state via link-state messages, such as LSAs and LSPs. As further described in this disclosure (e.g., with respect to the examples of FIGS. 3A-3B ), CE router 18 A may include a “link-overload TLV” within the link-state message that specifies the link overload state to inform other nodes in the IGP domain of the link overload state of logical link 30 . In some examples, the link-overload TLV may be a sub-TLV of another TLV that is included within the link state message. The link-overload TLV may include an address that identifies CE router 18 B (e.g., an endpoint of logical link 30 in IGP domain 29 ).

In the example of FIG. 1 , CE router 18 B may receive a link-state message that includes the link-overload TLV from CE router 18 A. Based on the information included in the link-overload TLV, CE router 18 B may set or otherwise assign a metric to logical link 30 , such that CE router 18 B stops sending network traffic to CE router 18 A using logical link 30 . For instance, CE router 18 B may set the metric for logical link 30 to a maximum metric. In some examples, the traffic reroute performed by CE router 18 B may occur due to the metric increase on link 30 . Accordingly, the solution proposed by techniques of this disclosure may be backward compatible and therefore only CE router 18 B may need to implement the extensions that include the link-overload information, as described in this disclosure. CE router 18 B may also advertise the link overload state to other nodes in IGP domain 29 by flooding the information via link-state messages, such as LSAs and LSPs. By advertising the link overload state to other nodes in IGP domain 29 , CE router 18 B may cause traffic from other nodes to be diverted using links other than logical link 30 .

As an alternative to or in conjunction with the above link-overload TLV aspects of the techniques, CE router 18 A and CE router 18 B may execute a second instance of BFD, which is shown as “BFD 33 B” in the example of FIG. 1 and utilize BFD session 33 B to signal that PE router 16 A adjacent to CE router 18 A is going to become nonoperational. In some examples, CE router 18 A generates a BFD message including a diagnostic bit set to indicate that a PE router adjacent to CE router 16 A along logical link 30 , i.e., PE router 16 A in the example of FIG. 1 , is going to become nonoperational. CE router 18 A may transmit the generated BFD message via BFD session 33 B to CE router 18 B, thereby informing CE router 18 B that PE router 16 A is going to become nonoperational.

CE router 18 B may then determine via the routing information that PE router 16 A is adjacent from CE router 18 A along the logical link over which BFD session 33 B is executed. CE router 18 B may then update the routing information (i.e., the IGP topology in this example) to indicate that PE router 16 A is going to become nonoperational. CE router 18 B may set or otherwise assign a metric to logical link 30 in a manner similar to that described above based on the diagnostic bit in the BFD message, such that CE router 18 B stops sending network traffic to CE router 18 A using logical link 30 . For instance, CE router 18 B may set the metric for logical link 30 to a maximum metric. In some examples, the traffic reroute performed by CE router 18 B may occur due to the metric increase on link 30 .

To continue sending and receiving network traffic with CE router 18 A, CE router 18 B may determine an alternate path to CE router 18 A. For instance, CE router 18 B may perform a Shortest Path First (SPF) calculation based on the topology of the network in example system 8 . In the example of FIG. 1 , CE router 18 B may determine that logical link 31 is available to send and receive network traffic with CE router 18 A. In particular, CE router 18 B may determine that logical link 31 is a part of the shortest path between CE router 18 A and 18 B. In some examples, a “shortest path,” may be a path with the fewest number of nodes or hops between CE router 18 A and 18 B. In some examples, a “shortest path,” may be a path that will carry network packets in the shortest amount of time between CE router 18 A and 18 B. In some examples, a “shortest path,” may be a path that satisfies one or more criteria when carrying packets between CE router 18 A and 18 B.

In response to determining that logical link 31 is available to send and receive network traffic with CE router 18 A, CE router 18 B may configure one or more of its packet forwarding engines (or “forwarding units”) to forward network traffic to CE router 18 A using logical link 31 . In this way, CE router 18 B may re-route network traffic from CE router 18 B to CE router 18 A to bypass PE router 16 A that is going offline for maintenance.

In the techniques of the disclosure described in FIG. 1 , PE router 16 A may continue forwarding packets during the time period from when PE router 16 A initially notifies CE router 18 A is being taken offline for maintenance until PE router 16 A no longer receives network packets from PE router 16 B or CE router 18 A. In some examples, PE router 16 A may continue forwarding packets during the time period from when PE router 16 A initially notifies CE router 18 A is being taken offline for maintenance until a timer of a defined time duration expires. The defined time duration may be set by an administrator or may be a hardcoded value in PE router 16 A. In any case, PE router 16 A may start the timer when PE router 16 A notifies CE router 18 A that PE router 16 A is being taken offline for maintenance, and may continue forwarding network traffic until the timer expires.

As described above, techniques of the disclosure may prevent transient black-holing of traffic in an overlay network when PE router 16 A is taken offline for maintenance. By incorporating a link-overload TLV into link-state messages and/or updating a BFD message to include an indication of whether PE router 16 A is going to become nonoperational, techniques of the disclosure enable CE router 18 B in IGP domain 29 , which is not directly coupled to PE router 16 A, to identify the particular link (e.g., logical link 30 ) that shall be bypassed. Accordingly, CE router 18 B in IGP domain 29 may re-route network traffic along an alternate path (e.g., logical link 31 ) that does not include PE router 16 A being taken offline for maintenance. In this way, techniques of the disclosure may avoid transient black-holing of traffic at PE router 16 A. In some examples, the described overload functionality in the techniques of this disclosure may be achieved in overlay networks without requiring any configuration overheads and/or requiring only minimal configuration overheads. The techniques of the disclosure may also be used during link migrations to achieve traffic diversion.

FIG. 2 is a block diagram illustrating an example CE router 18 A configured to prevent transient black-holing of traffic in an overlay network, in accordance with the techniques described herein. In this example, CE router 18 A includes control unit 42 that provides control plane functionality for the network device. CE router 18 A also includes switch fabric 48 interconnecting a set of line cards (“LCs”) 50 A- 50 N, each of which includes one or more of packet-forwarding engines (“PFEs”) 53 (which may also be referred to as “forwarding units”) that send and receive traffic by a set of interface cards 51 (“IFCs 51 ”) that typically have one or more physical network interfaces (ports). LCs 50 , components thereof, and switch fabric 48 collectively provide a data plane (which may also be referred to as a “forwarding plane”) for forwarding transient network traffic, such as the L2 packets described herein. Although not shown in FIG. 2 , PFEs 53 may each comprise a central processing unit (CPU), memory and one or more programmable packet-forwarding application-specific integrated circuits (ASICs). Switch fabric 48 provides a high-speed interconnect for forwarding incoming data packets between PFEs 53 for transmission over a network.

Control unit 42 provides an operating environment for various protocols that perform control plane functions for CE router 18 A. For example, control unit 42 includes BGP 66 as the control plane protocol for signaling the VPLS service 67 , such as signaling and establishing the individual pseudowires to transport the VPLS packet through the VPLS core. VPLS service 67 implements the VPLS protocol, such as including flooding and layer two (L2) learning, e.g., learning of customer device MAC addresses, from inbound pseudowires and association of those customer MAC addresses with corresponding outbound pseudowires and output interfaces. VPLS module 67 may maintain MAC tables for each VPLS instance established by router 40 . Learning and flooding may alternatively reside within PFEs 53 . Example details of MAC learning by a router within a VPLS domain are further described in U.S. patent application Ser. No. 12/246,810, “INTER-AUTONOMOUS SYSTEM (AS) VIRTUAL PRIVATE LOCAL AREA NETWORK SERVICE (VPLS),” filed on Oct. 7, 2008, the entire contents of which are incorporated herein by reference. Control unit 42 also includes IGP 75 . In some examples, IGP 75 may include OSPF and/or IS-IS. In some examples, IGP 75 may implement a link-overload TLV as described in this disclosure.

Control unit 42 may also include BFD 73 , which may be an implementation of the BFD protocol. BFD 73 may perform various aspects of the techniques described above related to signaling that PE router 16 A is going to become nonoperational. BFD 73 may program one or more of interface cards 51 , one or more of PFEs 53 and/or one or more of line cards 50 to establish first and second BFD sessions 33 A and 33 B. In some examples, BFD 73 may configure BFD sessions 33 A and 33 B within the same one of the interface cards 51 , PFES 53 and/or line cards 50 as a result of BFD session 33 A being established over link 26 A and link 26 A being utilized by logical link 30 . As such, the logical port for the logical link is configured within the same one of line cards 50 , PFEs 53 and/or interface cards 51 as the physical port for link 26 A. BFD 73 may configure BFD sessions 33 A and 33 B in the same layer three (L3) interface, as BFD session may require L3 rather than layer two (L2) interfaces.

As noted above, each of BFD sessions 33 A and 33 B are executed within a VLAN. BFD session 33 A is configured within VLAN 35 while BFD session 33 B is configured within VPLS 21 A. For the same reasons as those discussed above with respect to BFD sessions 33 A and 33 B, VPLS 67 may configure may configure VPLS 21 A and VLAN 35 within the same one of the interface cards 51 , PFES 53 and/or line cards 50 . Because PE router 16 A is not visible within VPLS 21 A the separate VLAN 35 is configured in the example of FIG. 1 to allow BFD visibility between CE router 18 A and PE router 16 A.

BFD 73 may configure BFD session 33 A in one of two ways. BFD 73 may configure BFD session 33 A as a static BFD session or as a dynamic seamless BFD (S-BFD) session. When created as a static BFD session, an administrator may configure a static router on each of CE router 18 A via U/I 62 and PE router via a U/I similar to U/I 62 . The administrator may configure BFD 73 to establish BFD session 33 A for the static route. The administrator may also configure IGP 75 to associate BFD session 33 A to BFD session 33 B between CE router 18 A and CE router 18 B.

The following is an example command line interface command for configuring a static BFD session:

set protocols ospf area 0 interface ge-0/0/0.1 bfd-liveness-detection associate-static vlan 100 address 10.10.1.2

The command is provided in for the OSPF protocol, but may be similar to the command for configuring other IGPs, such as IS-IS. The command associates the static BFD session to a VLAN 100 (which is the identifier assumed for purposes of example assigned to VLAN 35 ) and IP address 10.10.1.2 (which is associated with PE router 18 A in this example).

A dynamic BFD session is initiated by BFD 73 . As such, an administrator configures BFD 73 and configures a similar BFD 73 on PE router 18 A to be a so-called BFD reflector to request a dynamic BFD session between CE router 18 A and PE router 16 A over VLAN 35 . BFD 73 may then configure an S-BFD session 33 A dynamically, keeping IGP 75 as the client on CE router 18 A and interfacing with IGP 75 to associate the S-BFD session 33 A to IGP BFD session 33 B between CE router 18 A and CE router 18 B.

The following is an example command line interface command for configuring a static BFD session:

set protocols ospf area 0 interface ge-0/0/0.1 bfd-liveness-detection associate-dynamic vlan 100 address 10.10.1.2

The command is provided in for the OSPF protocol, but may be similar to the command for configuring other IGPs, such as IS-IS. The command associates the dynamic S-BFD session to a VLAN 100 (which is the identifier assumed for purposes of example assigned to VLAN 35 ) and IP address 10.10.1.2 (which is associated with PE router 18 A in this example).

Potential benefits of configuring BFD session 33 A as a dynamic S-BFD session is that a static route is not necessary and that PE router 18 A does not need to execute IGP 75 for VLAN 35 (because PE router 18 A is acting as a BFD reflector). Also, S-BFD does not require a three-way handshake prior to establish BFD sessions.

Control unit 42 may also provide an operating environment for execution of a routing engine 43 (“RE 43 ”) that controls L3 routing and L2 forwarding functions. In general, routing engine 43 maintains a routing information based (RIB) 44 that stores L3 routing information and L2 topology data representing a logical topology of the L2 network, e.g., a spanning tree, from the perspective of the interfaces. RIB 44 may also store updated MAC tables, MPLS label allocations and pseudowire information. Based on RIB 44 , RE 43 generates forwarding information based (FIB) 45 to contain forwarding data structures for installing within (e.g., programming) PFEs 53 .

In the example of FIG. 2 , control unit 42 includes a user interface (“U/I”) 62 with which an administrator interacts, either directly or by way of a provisioning system or software agent, to configure CE router 18 A. User interface 62 stores the information as configuration data 64 . Link Aggregation Control Protocol (LACP) 69 (optional) operates in a modified manner to bundle logical interfaces associated with the selected pseudowires and treat the bundle as a single logical interface in terms of L2 forwarding. In the case router 40 is coupled to the multi-homed, active-active access network, the administrator may interact with U/I 62 to form the customer-facing LAG of multi-homed L2 access links RE 43 may generate FIB 45 to include forwarding information that is used by control unit 42 to configure LC's 50 .

In the example of FIG. 2 , control unit 42 is connected to each of LCs 50 by a dedicated internal communication link 54 . For example, dedicated link 54 may comprise a 200 Mbps or Gigabit Ethernet connection for internal communication between the multiple components of router 40 . In one embodiment, control unit 42 communicates data representative of a software copy 45 ′ of FIB 45 into PFEs 53 to program the PFEs and thereby control forwarding of traffic by the corresponding components within the data plane. This allows the software FIB stored in memory (e.g., on-chip RAM) of in each of PFEs 53 to be updated without degrading packet-forwarding performance of CE router 18 A. In some instances, control unit 42 may derive separate and different software FIBs for each respective PFEs 53 . In addition, one or more of PFEs 53 may include packet-forwarding ASICs (not shown) that PFEs 53 program with a hardware-copy of FIB based on the software FIBs (i.e., hardware versions of the software FIBs) copied to each respective PFE 30 . In other, more complex embodiments, L2 switch may have many more LCs 50 (e.g., 48 or 64 FPCs), each of which may have four PFEs 50 that each couple to up to sixteen interface cards 51 .

CE router 18 A may be configured by an administrator, using U/I 62 , to join an IGP domain, such as IGP domain 29 . In FIG. 2 , IGP may run as an overlay network on VPLS pseudowires. Accordingly, PE routers that are configured to provide the VPLS pseudowires may not be visible to CE router 18 A in the IGP domain. CE router 18 A may exchange link state messages with other routers in the IGP domain based on topology of the network and changes to the topology of the network. In some examples, control unit 42 may store information in FIB 45 that associates identifiers of interfaces in LCs 50 with identifiers of physical links. In some examples, control unit 42 may store information in FIB 45 that associates identifiers of interfaces in LCs 50 with identifiers of logical links.

In the example of FIG. 2 , PE router 16 A may receive one or more instructions to initiate maintenance on PE router 16 A from an administrator or as a result of a scheduled or asynchronous event. As described in FIG. 1 , CE router 18 A may determine or otherwise be notified by PE router 16 A (via the diagnostic bit set in a BFD message received via BFD session 33 A) that PE router 16 A is being taken offline for maintenance. In some examples, PE router 16 A may determine or otherwise be notified that PE router 16 A is being taken offline for maintenance in a BFD session that uses BFD packets.

In response to being notified that PE router 16 A is being taken offline for maintenance, CE router 18 A may stop forwarding network traffic to PE router 16 A. In particular, RE 43 may determine information identifying logical link 30 in FIB 45 and set the link state of logical link 30 to a “link overload” state. For instance if PFE 53 A includes or is otherwise coupled to an interface that is further coupled to physical link 26 A between PE router 16 A and CE router 18 A, RE 43 may determine the identified interface is used to send and receive network traffic for logical link 30 . RE 43 may associate or otherwise assign a metric to data in FIB 45 that represents logical link 30 that causes PE router 16 A to stop sending network traffic to PE router 16 A using the interface coupled to physical link 26 A. In the example of FIG. 2 , RE 43 may associate or otherwise assign a maximum metric to data in FIB 45 that represents logical link 30 and/or the interface of PFE 53 A that is coupled to PE router 16 A. RE 43 may configure one or more of LCs 50 based on the updated information in FIB 45 .

To continue sending and receiving network traffic with CE router 18 B, RE 43 may configure one or more of LCs 50 to re-route any network traffic to CE router 18 B using logical link 31 that is present in IGP domain 29 . In particular, RE 43 may determine information identifying logical link 31 in FIB 45 and update FIB 45 to forward network traffic to CE router 18 B using logical link 31 . For instance, RE 43 may identify an interface of interfaces 51 that is coupled to physical link 26 B, which is used to carry network traffic for logical link 31 . RE 43 may update FIB 45 to forward network packets using the identified interface. RE 43 may configure one or more of LCs 50 based on the updated information in FIB 45 . In this way, CE router 18 A may re-route traffic to CE router 18 B using logical link 31 and bypass PE router 16 A.

CE router 18 A also advertises the link overload state to other nodes in IGP domain 29 by flooding the information that indicates the link overload state via link-state messages, such as LSAs and LSPs. RE 43 may cause one or more of LCs 50 to send link-state messages that include the information that indicates the link overload state to other routers in IGP domain 29 using one or more of interfaces 51 . RE 43 may generate the link-state message that includes a link-overload TLV.

In the case CE router 18 A runs OSPF as IGP 75 , RE 43 may include a link-overload TLV in the link state message that defines a set of fields with information that define a type, a length, and a remote IP address. The remote IP address may be the IP address of CE router 18 B. RE 43 may determine that CE router 18 B is other endpoint of logical link 30 and include the IP address of CE router 18 B in remote IP address field of the link-overload TLV. The type field may include a value that indicates that the sub TLV is a link-overload TLV. The length field may include a value that indicates the length of link-overload TLV or a portion of the overload sub TLV, such as the length of the remote IP address field or the length of the link-overload TLV itself. Further details of the link-overload TLV for OSPF are described in FIGS. 3A-3B . In the case that CE router 18 A runs IS-IS as IGP 75 , RE 43 may define or set one or more new link overload bits within a TLV of an IS-IS link state message, as further described in FIGS. 3A-3B . In any case, RE 42 causes one or more of LCs 50 to flood link state messages to other routers in IGP domain 29 , such as CE router 18 B.

CE router 18 B receives a link-state message that includes the link-overload TLV from CE router 18 A. The receiving node, CE router 18 B may process the link-overload TLV based on the link type for which the “link overload” information is received. Based on the information included in the link-overload TLV, CE router 18 B may set or otherwise assign a metric to logical link 30 , such that CE router 18 B stops sending network traffic to CE router 18 A using logical link 30 . For instance, CE router 18 B may set the metric for logical link 30 to a maximum metric. For point-to-point links and P2MP, the metric in the outgoing direction may be set to the maximum metric. CE router 18 B may also advertise the link overload state to other nodes in IGP domain 29 by flooding the information via link-state messages, such as LSAs and LSPs. By advertising the link overload state to other nodes in IGP domain 29 , CE router 18 B may cause traffic from other nodes to be diverted using links other than logical link 30 .

As an alternative to or in conjunction with the above link-overload TLV aspects of the techniques, CE router 18 A and CE router 18 B may execute BFD 33 B in the example of FIG. 1 and utilize BFD session 33 B to signal that PE router 16 A adjacent to CE router 18 A is going to become nonoperational. BFD session 33 A may receive the BFD message including a diagnostic bit signaling that PE router 18 A is going to become nonoperational and provide this information to BFD 73 . BFD 73 , due to the associate between BFD sessions 33 A and 33 B, may interface with BFD session 33 B so that BFD session 33 B generates a BFD message including a diagnostic bit set to indicate that a PE router adjacent to CE router 16 A along logical link 30 , i.e., PE router 16 A in the example of FIG. 1 , is going to become nonoperational. BFD session 33 B transmits the generated BFD message to CE router 18 B, thereby informing CE router 18 B that PE router 16 A is going to become nonoperational.

When BFD session 33 B sends the BFD message with the diagnostic bits set, BFD 73 starts a POLL sequence to ensure that the BFD message reaches the BFD state machine. BFD session 33 B between CE router 18 A and CE router 18 B does not change the state due to this diagnostic bit indicating that PE router 18 A is going to become nonoperational. The foregoing applies whether the BFD session 33 A is configured as a static BFD session or as a dynamic S-BFD session.

BFD session 33 B may, upon receiving the BFD message, interface with IGP 75 of CE router 18 B to inform IGP 75 of CE router 18 B that a router adjacent to CE router 18 A is going to become nonoperational. IGP 75 of CE router 18 B may then determine the interface via which the BFD message was sent and update via the routing information that PE router 16 A is adjacent from CE router 18 A along the logical link over which BFD session 33 B is executed. CE router 18 B may then update the routing information (i.e., the IGP topology in this example) to indicate that PE router 16 A is going to become nonoperational. IGP 75 of CE router 18 B may perform this update by setting or otherwise assigning a metric to the interface over which the BFD message was sent (e.g., the same interface supporting logical link 30 ) in a manner similar to that described above based on the diagnostic bit in the BFD message, such that CE router 18 B stops sending network traffic to CE router 18 A using logical link 30 . For instance, CE router 18 B may set the metric for logical link 30 to a maximum metric. In some examples, the traffic reroute performed by CE router 18 B may occur due to the metric increase on link 30 .

To continue sending and receiving network traffic with CE router 18 A, CE router 18 B may determine an alternate path to CE router 18 A. For instance, CE router 18 B may perform a Shortest Path First (SPF) calculation based on the topology of the network in example system 8 . In the example of FIG. 1 , CE router 18 B may determine that logical link 31 is available to send and receive network traffic with CE router 18 A. In particular, CE router 18 B may determine that logical link 31 is a part of the shortest path between CE router 18 A and 18 B.

In response to determining that logical link 31 is available to send and receive network traffic with CE router 18 A, CE router 18 B may configure one or more of its packet forwarding engines to forward network traffic to CE router 18 A using logical link 31 . In this way, CE router 18 B may re-route network traffic from CE router 18 B to CE router 18 A to bypass PE router 16 A that is going offline for maintenance.

The operating environment of control unit 42 may be implemented solely in software, or hardware, or may be implemented as a combination of software, hardware or firmware. For example, control unit 42 may include one or more processors which execute software instructions. In that case, control unit 42 may include various software modules or daemons executing on an operating system, and may include a non-transitory computer-readable storage device, such as computer memory or hard disk, for storing executable instructions.

The architecture of router 40 illustrated in FIG. 2 is shown for exemplary purposes only. The disclosure is not limited to this architecture. In other embodiments, router 40 may be configured in a variety of ways. In one embodiment, for example, some of the functionally of control unit 42 may be distributed within PFEs 53 . Elements of control unit 42 may be implemented solely in software, or hardware, or may be implemented as combinations of software, hardware, or firmware. For example, control unit 42 may include one or more processors, one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, or any combination thereof, which execute software instructions. In that case, the various software modules of control unit 42 may comprise executable instructions stored, embodied, or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer-readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), non-volatile random access memory (NVRAM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, a solid state drive, magnetic media, optical media, or other computer-readable media. Computer-readable media may be encoded with instructions corresponding to various aspects of router 40 , e.g., protocols. Control unit 42 , in some examples, retrieves and executes the instructions from memory for these aspects.

Although described as representing CE router 18 A, FIG. 2 may more generally represent any router, include PE routers 18 . In this respect, PE routers 18 may each include similar components to those of CE router 18 A, including control unit 42 , interface cards 51 , PEFs 53 , line cards 50 , and switch fabric 48 . Likewise, control unit 42 of PE routers 18 may include similar components to those shown in the example of FIG. 2 .

Control unit 42 of PE router 18 A, for example, may therefore execute or otherwise include a VPLS 67 , a BGP 66 and an IGP 75 similar to those shown in the example of FIG. 2 . VPLS 67 of PE router 18 A may configure VLAN 35 . BGP 66 of PE router 18 A may then configure BFD session 33 A in a manner similar to that described above with respect to CE router 18 A although from the perspective of PE router 18 A. BFD session 33 A may operate to generate and transmit the BFD message that includes the diagnostic bit indicating or otherwise signaling that PE router 18 A is going to become nonoperational.

FIGS. 3A-3B illustrate example link-overload TLVs that may be used to prevent transient black-holing of traffic in an overlay network, in accordance with the techniques described herein. As previously described in this disclosure, the link-overload TLV may be a sub-TLV or portion of another TLV included in a network packet. In other examples, the link-overload TLV may be the only TLV in a network packet. In some examples, a link-overload TLV may be generated in a network packet, such as a link-state message, by RE 43 , or one or more of LCs 50 , as shown in FIG. 2 . Techniques of the disclosure provide protocol extensions for new link-overload TLVs that are defined in IS-IS and OSPF to carry link overload information.

FIG. 3A illustrates a link-overload TLV 100 that is generated by CE router 18 A for an OSPFv2 link-state message, in accordance with techniques of this disclosure. In some examples, CE router 18 A includes link-overload TLV 100 as part of the extended link TLV defined in “OSPFv2 Prefix/Link Attribute Advertisement,” draft-ietf-ospf-prefix-link-attr, Feb. 2, 2015, https://datatracker.ietf.org/doc/draft-ietf-ospf-prefix-link-attr/?include_text=1, which is hereby incorporated by reference. As shown in FIG. 3A , link-overload TLV 100 may include a type field 100 , length field 104 , and remote IP address field 106 (e.g., the value field of TLV 100 ). Link-overload TLV 100 may indicate that the link (e.g., logical link 30 ) which carries the link-overload TLV 100 is overloaded and the metric for the corresponding link (e.g., logical link 30 ) identified by the remote IP address should be set to a maximum metric for SPF calculation.

Although shown as a 16-bit field, type field 102 may be of any size in other examples. Type field 102 may include a value that indicates that the sub TLV is a link-overload TLV. In this way, CE router 16 B, when processing a link-state message, may determine that the link-state message includes a link-overload TLV, and perform one or more techniques as described in this disclosure, such as setting a metric and/or re-routing network traffic to name only a few example operations.

Although shown as a 16-bit field, length field 104 may be of any size in other examples. Length field 104 may include a value that indicates the length of link-overload TLV or a portion of the overload sub TLV, such as the length of the remote IP address field or the length of the link-overload TLV itself. In some examples, the value of length field 104 may be 4. In the example where the length of the remote IP address field 106 is indicated in length field 104 , CE router 16 B, when processing a link-state message, may determine the total number of bits in remote IP address field 106 that represent an IP address of a router, such as CE router 16 B.

Although shown as a 32-bit field, remote IP address field 106 may be of any size in other examples. Remote IP address field 106 may include an IP address of an endpoint router of a link (e.g., a logical link) in an IGP domain. In some examples, the IP address may be an IPv4 address. For instance, logical link 30 , which is visible in IGP domain 29 of FIG. 1 , includes endpoint routers CE router 18 A and CE router 18 B. When generating the value for remote IP address field 106 , a first one of the endpoints, e.g., CE router 18 A, determines the IP address of the second endpoint, such as the IP address of CE router 18 B, and includes the remote IP address in remote IP address field 106 . By incorporating the remote IP address into link-overload TLV 100 , techniques of the disclosure enable CE router 18 B in IGP domain 29 to identify the particular link that will no longer be used by CE router 18 B to forward network traffic to CE router 18 A. CE router 18 B, when processing a link-state message, may use the value of remote IP address field 106 to identify the particular link and associate a metric, such as a maximum metric with the particular link. In this way, CE router 18 B may stop forwarding network traffic to CE router 18 A using the particular link.

FIG. 3B illustrates a link-overload TLV 110 that is generated by CE router 18 A for an OSPFv3 link-state message, in accordance with techniques of this disclosure. In some examples, CE router 18 A includes link-overload TLV 110 in the Router-link TLV as defined in “OSPFv3 LSA Extendibility,” draft-ietf-ospf-ospfv3-1sa-extend, Feb. 16, 2015, https://datatracker.ietf.org/doc/draft-ietf-ospf-ospfv3-1sa-extend/?include_text=1, which is hereby incorporated by reference. As shown in FIG. 3B , link-overload TLV 110 may include a type field 112 , length field 114 , and remote IP address field 116 (e.g., the value field of TLV 110 ). Link-overload TLV 110 may indicate that the link (e.g., logical link 30 ) which carries the link-overload TLV 110 is overloaded and the metric for the corresponding link (e.g., logical link 30 ) identified by the remote IP address should be set to a maximum metric for SPF calculation.

Although shown as a 16-bit field, type field 112 may be of any size in other examples. Type field 112 may include a value that indicates that the sub TLV is a link-overload TLV. In this way, CE router 16 B, when processing a link-state message, may determine that the link-state message includes a link-overload TLV, and perform one or more techniques as described in this disclosure, such as setting a metric and/or re-routing network traffic to name only a few example operations.

Although shown as a 16-bit field, length field 114 may be of any size in other examples. Length field 114 may include a value that indicates the length of link-overload TLV or a portion of the overload sub TLV, such as the length of the remote IP address field or the length of the link-overload TLV itself. In some examples, the value of length field 114 may be 16. In the example where the length of the remote IP address field 116 is indicated in length field 114 , CE router 16 B, when processing a link-state message, may determine the total number of bits in remote IP address field 116 that represent an IP address of a router, such as CE router 16 B.

Although shown as a 32-bit field, remote IP address field 116 may be of any size in other examples. Remote IP address field 116 may include an IP address of an endpoint router of a link (e.g., a logical link) in an IGP domain. In some examples, the IP address may be an IPv6 address. For instance, logical link 30 , which is visible in IGP domain 29 of FIG. 1 , includes endpoint routers CE router 18 A and CE router 18 B. When generating the value for remote IP address field 116 , a first one of the endpoints, e.g., CE router 18 A, determines the IP address of the second endpoint, such as the IP address of CE router 18 B, and includes the remote IP address in remote IP address field 116 . By incorporating the remote IP address into link-overload TLV 110 , techniques of the disclosure enable CE router 18 B in IGP domain 29 to identify the particular link that will no longer be used by CE router 18 B to forward network traffic to CE router 18 A. CE router 18 B, when processing a link-state message, may use the value of remote IP address field 116 to identify the particular link and associate a metric, such as a maximum metric with the particular link. In this way, CE router 18 B may stop forwarding network traffic to CE router 18 A using the particular link.

In some examples, CE routers 18 A and 18 B may use IS-IS as the IGP. RFC 5029 defines a link-attributes sub-TLV. RFC 5029 entitled “Definition of an IS-IS Link Attribute Sub-TLV,” https://datatracker.ietf.org/doc/rfc5029/?include_text=1, Mar. 2, 2013, is hereby incorporated by reference. In accordance with techniques of the disclosure CE router 18 A, when generating a link-state message to indicate a link overload state, may define or otherwise set a new “link overload bit” in the link-attributes sub-TLV. In some examples, the link overload bit may be set at bit position 0x04. The link attributes sub TLV may be carried within TLV-22 as defined by RFC 5029. TLV-22 is further described in RFC 5305 entitled “IS-IS Extensions for Traffic Engineering,” https://datatracker.ietf.org/doc/rfc5305/?include_text=1, Mar. 2, 2013. By incorporating the setting the link overload bit, techniques of the disclosure enable CE router 18 B in IGP domain 29 to identify the particular link that will no longer be used by CE router 18 B to forward network traffic to CE router 18 A. The link overload bit may enable CE router 18 B to determine that the link (e.g., logical link 30 ) which carries the link overload bit is overloaded should be set to a maximum metric for SPF calculation. In this way, CE router 18 B may stop forwarding network traffic to CE router 18 A using the particular link based on the link overload bit.

FIG. 4 illustrates a BFD message 120 that is generated by a CE router 18 or a PE router 16 in accordance with techniques of this disclosure. BFD message 120 may also be referred to as a BFD control packet as set forth in section 4 and particularly section 4.1 of the above incorporated RFC 5880. Section 4.1 of the above incorporated RFC 5880 discusses each of the various fields shown in the example of FIG. 4 , including the “Vers” version field, the “Diag” diagnostic field, the “Sta” state field, the “P” Poll flag, the “F” final flag, the “C” control plane independent flag, the “A” authentication present flag, the “D” demand flag, the “M” multipoint flag, the “Length” field, the “My Discriminator” field, the “Your Discriminator” field, the “Desired Min TX Interval” field, the “Required Min RX Interval,” and the “Required Min Echo RX Interval” field.

With respect to the BFD aspects of the techniques described in this disclosure, CE router 18 A may for example configure diagnostic field 122 to signal that PE router 18 A is going to become nonoperational. In some examples, CE router 18 A may set diagnostic field 122 to a specify a value of nine (9) to signal that PE router 18 A is going to become nonoperational in BFD control packet 120 . CE router 18 A may generate BFD control packet 120 in this manner and transmit BFD control packet 120 via BFD session 33 B. Although described with respect to CE router 18 A, PE router 18 A may generate BFD control packet 18 A in the above described manner to signal that PE router 18 A is going to become nonoperational and transmit BFD control packet 18 A via BFD session 33 A. Moreover, although described as specifying a value of nine to signal that PE router 18 A is going to become nonoperational, any value not presently used (or, in other words, reserved for future use) may be used to signal that PE router 18 A is going to become nonoperational or more generally signal that the active path is going down (meaning, going to become nonoperational).

FIG. 5 is flowchart illustrating example operations implemented by multiple network devices to prevent transient black-holing of traffic in an overlay network, in accordance with the techniques described herein. Example operations in accordance with techniques of the disclosure are illustrated for example purposes with respect to PE router 16 A, CE router 18 A, and CE router 18 B, as described in this disclosure. Initially, CE routers 18 A and 18 B may join IGP domain 29 . IGP domain 29 may include multiple logical links such as logical links 30 and 31 . IGP domain 29 may operate as an overlay network on top of underlying layer 2 network. The underlying layer 2 network may implement VPLS and/or IP2SEC running at one or more provider routers, such as PE routers 16 A- 16 D. PE routers 16 A- 16 D may implement one or more pseudowires 21 A and 21 B, which may comprise a portion of an underlying network on which the overylay network runs.

Upon each of CE routers 18 A and 18 B joining IGP domain 29 , CE router 18 A may forward network traffic to CE router 18 B using logical link 30 ( 150 ), and CE router 18 B may forward network traffic to CE router 18 B using logical link 30 ( 152 ). Network traffic carried by logical link 30 in the IGP overlay network is also carried by pseudowire 21 A in the underlying layer 2 network. At a later time, PE router 16 A may receive the one or more instructions in response to user input from an administrator or automatically as a result of a scheduled or asynchronous event ( 154 ).

CE router 18 A may determine that PE router 16 A will be undergoing maintenance, and therefore logical link 30 will be unavailable to forward network traffic due to maintenance of PE router 16 A ( 158 ). For instance, PE router 16 A may notify CE router 18 A by sending one or more messages to CE router 18 A, or CE router 18 A may determine that PE router 16 A is no longer sending messages that would otherwise indicate that PE router 16 A is online and/or available to forward network traffic.

CE router 18 A may flood link-state messages to other routers in IGP domain 29 that include link overload information to overload logical link 30 and/or send a BFD message, such as BFD message 120 shown in the example of FIG. 4 , signaling that an active path is going down ( 160 ). In some examples, the link-overload information may include a link-overload TLV and/or one or more link-overload bits as described in FIGS. 3A-3B . The link-state messages may include an extension that stores the link-overload information. To continue sending and receiving network traffic with CE router 18 B, CE router 16 A may also configure one or more of its forwarding units to re-route any network traffic to CE router 18 B using logical link 31 that is present in IGP domain 29 ( 164 ). In this way, CE router 18 A may re-route traffic to CE router 18 B using logical link 31 and bypass PE router 16 A. Accordingly, CE router 18 A may forward network traffic to CE router 18 B using logical link 31 ( 166 ).

CE router 18 B receives a link-state message that includes the link-state information and/or BFD message 120 from CE router 18 A. CE router 18 B may process the link-overload information and/or BFD message 120 and set or otherwise assign a metric to overload logical link 30 , such that CE router 18 B stops sending network traffic to CE router 18 A using logical link 30 ( 162 ). For instance, CE router 18 B may set the metric for logical link 30 to a maximum metric. To continue sending and receiving network traffic with CE router 18 A, CE router 18 B may determine an alternate path to CE router 18 A. For instance, CE router 18 B may perform a Shortest Path First (SPF) calculation based on the topology of the network in example system 8 . CE router 18 B may determine that logical link 31 is available to send and receive network traffic with CE router 18 A, and begin forwarding traffic using logical link 31 ( 168 ).

As shown in FIG. 5 , PE router 16 A may continue forwarding packets during the time period from when PE router 16 A initially notifies CE router 18 A is being taken offline for maintenance until PE router 16 A no longer receives network packets from PE router 16 B or CE router 18 A ( 170 ). In other examples, PE router 16 A may continue forwarding packets during the time period from when PE router 16 A initially notifies CE router 18 A is being taken offline for maintenance until a timer of a defined time duration expires. The defined time duration may be set by an administrator or may be a hardcoded value in PE router 16 A. In any case, PE router 16 A may start the timer when PE router 16 A notifies CE router 18 A that PE router 16 A is being taken offline for maintenance, and may continue forwarding network traffic until the timer expires.

FIG. 6 is flowchart illustrating example operations of a network device that may prevent transient black-holing of traffic in an overlay network, in accordance with the techniques described herein. For purposes of illustration only, the example operations are described below within the context of CE router 18 A in this disclosure. Initially, CE router 18 A may join IGP domain 29 ( 200 ). IGP domain 29 may include multiple logical links such as logical links 30 and 31 . As previously described in this disclosure, IGP domain 29 may operate as an overlay network on top of underlying layer 2 network. PE routers 16 A- 16 D may implement one or more pseudowires 21 A and 21 B, which may comprise a portion of an underlying network on which the overylay network runs.

CE router 18 A may configure one or more of its forwarding units to forward network traffic to CE router 18 B using logical link 30 ( 202 ). Network traffic carried by logical link 30 in the IGP overlay network is also carried by pseudowire 21 A in the underlying layer 2 network. At a later time, PE router 16 A may receive the one or more instructions in response to user input from an administrator or automatically as a result of a scheduled or asynchronous event. CE router 18 A may determine that PE router 16 A will be undergoing maintenance, and therefore logical link 30 will be unavailable to forward network traffic due to maintenance of PE router 16 A ( 204 ).

CE router 18 A may generate link-state messages that include link overload information to overload logical link 30 and/or BFD message 120 signaling that the active path is going down ( 206 ). In some examples, the link-overload information may include a link-overload TLV and/or one or more link-overload bits as described in FIGS. 3A-3B . The link-state messages may include an extension that stores the link-overload information. CE router 18 A may send the generated link-state messages to other routers in IGP domain 29 or BFD message 120 to CE router 18 B participating in BFD session 33 B ( 208 ). To continue sending and receiving network traffic with CE router 18 B, CE router 16 A may also configure one or more of its forwarding units to re-route any network traffic to CE router 18 B using logical link 31 that is present in IGP domain 29 ( 210 ). In this way, CE router 18 A may re-route traffic to CE router 18 B using logical link 31 and bypass PE router 16 A. Accordingly, CE router 18 A may forward network traffic to CE router 18 B using logical link 31 ( 212 ).

FIG. 7 is block diagram of multiple network devices that may implement operations prevent transient black-holing of traffic in a broadcast network, in accordance with the techniques described herein. FIG. 7 illustrates CE routers 252 A- 252 D and network devices 270 (e.g., routers, switches, etc.) that implement a broadcast network 264 . CE routers 252 A- 252 D are coupled to broadcast network 264 by links 262 A- 262 D. Broadcast network 264 may operate or otherwise be implemented within, as a part of, or otherwise support an IGP domain. CE routers 252 A and 252 B may be included in enterprise network 250 A, and CE routers 252 C and 252 D may be included in enterprise network 252 B. CE routers 252 A and 252 B may be multi-homed one or more of network devices 270 , and CE routers 252 C and 252 D may be multi-homed to one or more of network devices 270 that implement broadcast network 264 . In the example of FIG. 6 , CE router 252 D may be the designated router for broadcast network 264 . Accordingly, CE router 252 D may originate network link advertisements on behalf of broadcast network 264 . Using VPLS and/or IP2SEC, enterprise networks 250 A and 250 B may appear to be directly attached to the same local area network (LAN), and therefore hosts 260 A and 260 B may appear to be attached to the same LAN. CE routers 252 A- 252 D may run an IGP, such as OSPF or IS-IS, such that each of CE routers 252 A- 252 D are included in the same IGP domain.

In the example of FIG. 7 , enterprise network 250 A may be multi-homed to broadcast network 264 with a broadcast link 262 B. When broadcast link 262 B is going to be replaced or otherwise taken down for maintenance, techniques of the disclosure may cause other network devices, such as CE routers 252 C and 252 D in broadcast network 264 to divert network traffic to CE router 252 A, rather than sending network traffic to CE router 252 B. For instance, broadcast link 262 B may become unavailable due to maintenance at CE router 262 B.

In response to CE router 252 B determining that it is being taken offline for maintenance, CE router 252 B may overload link 262 B. For instance, CE router 252 B may set the metric to link 262 B to a maximum metric. Accordingly, CE router 252 B may stop sending network traffic using link 262 B. However, before CE router 252 B stops sending network traffic using link 262 B, CE router 252 B may send link-state messages to other routers in the same IGP domain, such as CE router 252 A, 252 C, and 252 D. The link-state messages may include link-overload information as described in accordance with techniques of this disclosure. For instance, the link-state messages may include link-overload TLVs or link-overload bits, as described in FIGS. 3A-3B .

Each of routers CE 252 A, 252 C, and 252 D may receive the link-state messages that include the link-overload information. In the example of FIG. 7 , because CE router 252 D is the designated router, CE router 252 D may remove CE router 252 B from its list of neighbors in broadcast network 264 . CE router 252 D may perform an SPF computation without CE router 252 B and update its forwarding plane to forward network traffic to enterprise network 250 A using CE router 242 A and link 262 A. CE router 252 D may also flood information indicating the removal of CE router 252 B in link-state messages to other routers in broadcast network 264 , such as CE routers 252 A and 252 C. Upon receiving the link-state message, other routers of broadcast network 264 , such as CE router 252 C may perform an SPF computation without CE router 252 B and update its forwarding plane to forward network traffic to enterprise network 250 A using CE router 242 A and link 262 A. In this way, CE routers other than CE router 252 B in broadcast network 264 may be notified in advance that link 262 B will be unavailable, and re-route network traffic to enterprise network 250 A prior to link 262 B being taken offline. Accordingly, such techniques may reduce or prevent transient black-holing of traffic at CE router 252 B.

In the techniques of the disclosure described in FIG. 7 , CE router 252 B may continue forwarding packets during the time period from when CE router 252 B initially notifies other CE routers in broadcast network 264 that link 262 B is being taken offline for maintenance until CE router 252 B no longer receives network packets from other routers. In some examples, CE router 252 B may continue forwarding packets during the time period from when CE router 252 B initially notifies other routers in broadcast network 264 that link 262 B is being taken offline for maintenance until a timer of a defined time duration expires. The defined time duration may be set by an administrator or may be a hardcoded value in CE router 252 B. In any case, CE router 252 B may start the timer when CE router 252 B determines that link 262 B is being taken offline for maintenance, and may continue forwarding network traffic until the timer expires.

It some examples, it may be necessary that the originator and receiver of link-overload TLV understand the extensions defined in this document and in case of broadcast links the originator and the designated router may need to understand the extensions. Other nodes in the network, however, may not have to understand the extensions. If the receivers of the link-overload TLV do not understand it, they may ignore it without causing other impacts to the network.

The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components. The term “processor” or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit including hardware may also perform one or more of the techniques of this disclosure.

Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various techniques described in this disclosure. In addition, any of the described units, modules or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware, firmware, or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware, firmware, or software components, or integrated within common or separate hardware, firmware, or software components.

The techniques described in this disclosure may also be embodied or encoded in an article of manufacture including a computer-readable medium encoded with instructions. Instructions embedded or encoded in an article of manufacture including a computer-readable medium encoded, may cause one or more programmable processors, or other processors, to implement one or more of the techniques described herein, such as when instructions included or encoded in the computer-readable medium are executed by the one or more processors. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a compact disc ROM (CD-ROM), a floppy disk, a cassette, magnetic media, optical media, or other computer readable media. In some examples, an article of manufacture may include one or more computer-readable storage media. In some examples, a computer-readable storage media may include non-transitory media. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

It is to be recognized that depending on the embodiment, certain acts or events of any of the methods described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the method). Moreover, in certain embodiments, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

Various embodiments of the invention have been described. These and other embodiments are within the scope of the following claims.