Receive-side processing for encapsulated encrypted packets

Virtualization allows the abstraction and pooling of hardware resources to support virtual machines in a Software-Defined Networking (SDN) environment, such as a Software-Defined Data Center (SDDC). For example, through server virtualization, virtual machines (VMs) running different operating systems may be supported by the same physical machine (e.g., referred to as a “host”). Each VM is generally provisioned with virtual resources to run an operating system and applications. Further, through SDN, benefits similar to server virtualization may be derived for networking services. For example, logical overlay networks may be provisioned, changed, stored, deleted and restored programmatically without having to reconfigure the underlying physical hardware architecture. In practice, network device(s) may be deployed in the SDN environment to perform packet processing to facilitate communication among endpoints, such as VMs, etc. However, existing approaches for processing encapsulated encrypted packets may lack efficiency.

FIG. 1 is a schematic diagram illustrating an example software-defined networking (SDN) environment in which receive-side processing for encapsulated encrypted packets may be performed;

FIG. 2 is a schematic diagram illustrating an example physical view of the SDN environment in FIG. 1 ;

FIG. 3 is a flowchart of an example process for a network device to perform receive-side processing for encapsulated encrypted packets;

FIG. 4 is a schematic diagram illustrating an example architecture of a computer system configured to perform receive-side processing for encapsulated encrypted packets;

FIG. 5 is a flowchart of an example detailed process for receive-side processing for encapsulated encrypted packets; and

FIG. 6 is a schematic diagram illustrating an example of receive-side processing for encapsulated encrypted packets according to the example in FIG. 5 .

In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the drawings, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

Challenges relating to packet processing will now be explained using FIG. 1 and FIG. 2 . In particular, FIG. 1 is a schematic diagram illustrating example software-defined networking (SDN) environment 100 in which receive-side processing for encapsulated encrypted packets may be performed. FIG. 2 is a schematic diagram illustrating example physical view 200 of SDN environment 100 in FIG. 1 . It should be understood that, depending on the desired implementation, SDN environment 100 may include additional and/or alternative components than that shown in FIG. 1 and FIG. 2 . In practice, SDN environment 100 may include any number of hosts (also known as “computer systems,” “computing devices”, “host computers”, “host devices”, “physical servers”, “server systems”, “transport nodes,” etc.). Each host may be supporting any number of virtual machines (e.g., tens or hundreds).

Referring first to FIG. 1 , SDN environment 100 includes multiple hosts, such as host-A 110 A, host-B 110 B and host-C 110 C. In practice, hosts 110 A-C may be located at geographically dispersed sites, such hosts 110 A-B at a first site and hosts 110 C-D at a second site. To facilitate communication among hosts 110 A-D over physical network 182 , EDGE 1 150 may be deployed at the edge of the first site, and EDGE 2 160 at the edge of the second site. Throughout the present disclosure, EDGE 150 / 160 may be an entity that is implemented using one or more virtual machines (VMs) and/or physical machines (also known as “bare metal machines”) and capable of performing functionalities of a switch, router, bridge, gateway, edge appliance, any combination thereof, etc. In practice, EDGE 150 / 160 may implement a centralized service router (SR) to provide networking services such as firewall, load balancing, network address translation (NAT), intrusion detection, deep packet inspection, etc.

Referring also to FIG. 2 , EDGE 1 150 (“first computer system”) and EDGE 2 160 (“second computer system”) may be VMs supported by respective host-B 110 B and host-C 110 C. Hosts 110 A-C may each include suitable hardware and virtualization software (e.g., hypervisors 112 A-C) to support various VMs. For example, at the first site (see 201 ), host-A 110 A supports VM 1 131 and VM 3 133 , and host-B 110 B supports EDGE 1 150 and VM 2 132 . Hosts 110 A-B may be connected via any suitable physical network 203 . At the second site (see 202 ), host-C 110 C supports EDGE 2 160 and VM 4 134 , and host-D 110 D supports VM 5 135 (not shown in FIG. 2 for simplicity). In alternative implementations, EDGE 1 150 and/or EDGE 2 160 may be bare metal machines or supported by different hosts than that shown in FIG. 2 .

Hypervisor 112 A/ 112 B/ 112 C maintains a mapping between underlying hardware 111 A/ 111 B/ 111 C and virtual resources allocated to the VMs. Hardware 111 A/ 111 B/ 111 C includes various physical components, such as central processing unit(s) or processor(s) 120 A/ 120 B/ 120 C; memory 122 A/ 122 B/ 122 C; physical network interface controllers (NICs) 124 A/ 124 B/ 124 C; and storage disk(s) 128 A/ 128 B/ 128 C accessible via storage controller(s) 126 A/ 126 B/ 126 C, etc. Virtual resources are allocated to each virtual machine to support a guest operating system (OS) and applications, such as virtual central processing unit (CPU), guest physical memory, virtual disk(s) and virtual network interface controller (VNIC). Hypervisor 112 A/ 112 B/ 112 C further implements virtual switch 114 A/ 114 B/ 114 C and logical distributed router (DR) instance 116 A/ 116 B/ 116 C to handle egress packets from, and ingress packets to, respective VMs.

In practice, logical switches and logical distributed routers may be implemented in a distributed manner and can span multiple hosts 110 A-C to connect the VMs. For example, a logical switch may be configured to provide logical layer-2 connectivity to VMs supported by different hosts. The logical switch may be implemented collectively by virtual switches 114 A-C of respective hosts 110 A-C and represented internally using forwarding tables (e.g., 115 A-C) at the respective virtual switches 114 A-C. Further, logical distributed routers that provide logical layer-3 connectivity may be implemented collectively by distributed router (DR) instances (e.g., 116 A-C) of respective hosts 110 A-C and represented internally using routing tables (e.g., 117 A-C) at the respective DR instances. Routing tables 117 A-C may be each include entries that collectively implement the respective logical distributed routers.

The VMs (e.g., VMs 131 - 134 , 150 and 160 ) may send and receive packets via respective logical ports 141 - 146 . As used herein, the term “logical port” may refer generally to a port on a logical switch to which a virtualized computing instance is connected. A “logical switch” may refer generally to an SDN construct that is collectively implemented by virtual switches of hosts 110 A-C, whereas a “virtual switch” (e.g., 114 A-C) may refer generally to a software switch or software implementation of a physical switch. In practice, there is usually a one-to-one mapping between a logical port on a logical switch and a virtual port on a virtual switch. However, the mapping may change in some scenarios, such as when the logical port is mapped to a different virtual port on a different virtual switch after migration of the corresponding virtualized computing instance (e.g., when the source and destination hosts do not have a distributed virtual switch spanning them).

Although examples of the present disclosure refer to virtual machines, it should be understood that a “virtual machine” running on a host is merely one example of a “virtualized computing instance” or “workload.” A virtualized computing instance may represent an addressable data compute node or isolated user space instance. In practice, any suitable technology may be used to provide isolated user space instances, not just hardware virtualization. Other virtualized computing instances may include containers (e.g., running within a VM or on top of a host operating system without the need for a hypervisor or separate operating system or implemented as an operating system level virtualization), virtual private servers, client computers, etc. Such container technology is available from, among others, Docker, Inc. The virtual machines may also be complete computational environments, containing virtual equivalents of the hardware and software components of a physical computing system.

As used herein, the term “hypervisor” may refer generally to a software layer or component that supports the execution of multiple virtualized computing instances, including system-level software in guest virtual machines that supports namespace containers such as Docker, etc. Hypervisors 114 A-C may each implement any suitable virtualization technology, such as VMware ESX® or ESXi™ (available from VMware, Inc.), Kernel-based Virtual Machine (KVM), etc. The term “packet” may refer generally to a group of bits that can be transported together from a source to a destination, such as message, segment, datagram, etc. The term “traffic” may refer generally to a flow of packets. The term “layer 2” may refer generally to a Media Access Control (MAC) layer; “layer 3” to a network or Internet Protocol (IP) layer; and “layer-4” to a transport layer (e.g., using transmission control protocol (TCP) or user datagram protocol (UDP)) in the Open System Interconnection (OSI) model, although the concepts described herein may be used with other networking models.

Depending on the desired implementation, tunnel 180 may be established between a first tunnel endpoint at EDGE 1 150 and a second tunnel endpoint at EDGE 2 160 . In practice, the second tunnel endpoint may be any other endpoint or non-edge router, not just EDGE 2 160 . Tunnel 180 may be established using any suitable tunneling protocol. For example, a Virtual Private Network (VPN) based on Internet Protocol Security (IPSec) may bridge traffic in a hybrid cloud environment between first site 201 (e.g., on-prem data center) and second site 202 (e.g., public cloud environment). In practice, IPSec is a secure network protocol suite that provides data authentication, integrity and confidentiality between a pair of entities (e.g., data centers, gateways) across an IP-based network. One example in the IPSec protocol suite is Encapsulating Security Payload (ESP), which provides origin authenticity using source authentication, data integrity and confidentiality through encryption protection for IP packets. Although various examples will be discussed using IPSec-based VPN, it should be understood that any alternative and/or additional protocol(s) may be used.

In the example in FIG. 1 , some example cross-site packet flows are shown. A first packet flow is between VM 4 134 on host-C 110 C and VM 1 131 on host-A 110 A. A second packet flow is between VM 5 135 on host-D 110 D and VM 2 132 on host-B 110 B. At the second site, EDGE 2 160 may perform transmit-side processing by performing encryption and encapsulation on packets originating from source endpoints VM 4 134 and VM 5 135 . At the first site, EDGE 1 150 may perform receive-side processing by performing decryption and decapsulation before forwarding (decrypted) packets towards destination endpoints VM 1 131 and VM 2 132 . Conventionally, however, packet processing at EDGE 1 150 may lack parallelism and efficiency. This causes a bottleneck at EDGE 1 150 , especially when there is a large number of packets that require receive-side processing.

Receive-Side Processing for Encapsulated Encrypted Packets

According to examples of the present disclosure, receive-side processing for encapsulated encrypted packets may be performed in a more efficient manner. In particular, post-cryptography processing may be distributed among multiple processing units of EDGE 1 150 based on (cleartext) content of decrypted packets. This way, receive-side scaling (RSS) may be implemented for post-cryptography operations to improve parallelism at EDGE 1 150 and therefore performance. Examples of the present disclosure should be contrasted against conventional approaches that assign packets to different processing units based on their non-encrypted outer header because the rest of the packets are encrypted.

Examples of the present disclosure may be performed by any suitable “computer system” configured to perform receive-side processing. In the following, an example “computer system” will be explained using EDGE 1 150 , which is deployed at the edge of first site 201 to facilitate communication among VMs 131 - 135 . Depending on the desired implementation, EDGE 1 150 may implement any suitable data-plane packet processing engine(s) to perform packet processing. One example is the Data Plane Development Kit (DPDK), which is an open-source Linux Foundation project that provides a set of data plane libraries and (physical or virtual) NIC drivers to accelerate fast packet processing.

As used herein, the term “receive-side processing” may be used generally to include various operations performed by a computer system in response to receiving ingress or incoming encapsulated encrypted packets, including authentication, decryption, decapsulation, encapsulation, firewall, load balancing, forwarding to destination, etc. To support RSS, EDGE 150 may include multiple processing units labelled as core 1 , . . . , coreN (see 151 in FIG. 1 ). Any number (N>1) of processing units may be configured. The term “processing units” or “cores” may refer generally to a set of components capable of supporting parallel processing. Each “processing unit” or “core” may be hardware-implemented (e.g., processors, CPU cores) and/or software-implemented (e.g., threads executed in parallel, virtual CPUs). Using DPDK (see FIG. 4 ), poll mode driver (PMD) threads may run on top of processing units 151 .

In more detail, FIG. 3 is a flowchart of example process 300 for a computer system to perform receive-side processing for encapsulated encrypted packets. Example process 300 may include one or more operations, functions, or actions illustrated by one or more blocks, such as 310 to 360 . The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated depending on the desired implementation. In the following, example process 300 will be explained using the example packet flows in FIG. 1 . Throughout the present disclosure, an asterisk (*) is used to indicate encrypted packet content, such as encrypted header (e.g., I1* and I2*) and encrypted payload (e.g., P1* and P2*). The absence of the asterisk (e.g., O1, I1, P1) indicates decrypted or cleartext packet content.

At 310 and 320 in FIG. 3 , in response to receiving a first encapsulated encrypted packet (see ENCAP 1 102 ) over tunnel 180 , EDGE 1 150 may perform decryption and decapsulation on the first encapsulated encrypted packet. Using the example in FIG. 1 , ENCAP 1 102 includes a first outer header (O1) and a first encrypted inner packet (I1*, P1*), where I1*=encrypted inner header, and P1*=encrypted payload. Block 320 may involve decapsulation to remove the first outer header (O1) and decryption to generate a first decrypted inner packet (I1, P1), thereby extracting the same inner packet content that is encrypted by EDGE 2 160 . Block 320 may also include authentication and other receive-side processing operation(s). At 330 in FIG. 3 , EDGE 1 150 may assign the first encrypted inner packet (I1, P1) to a first processing unit (e.g., core 1 ) of EDGE 1 150 based on (cleartext) content of the first encrypted inner packet. See 101 - 104 in FIG. 1 .

At 340 and 350 in FIG. 3 , in response to receiving a second encapsulated encrypted packet (see ENCAP 2 106 ) over tunnel 180 , EDGE 1 150 may perform decryption and decapsulation. Using the example in FIG. 1 , ENCAP 2 106 includes a second outer header (O2) and a second encrypted inner packet (I2*, P2*), where I2*=encrypted inner header and P2*=encrypted payload. Block 350 may involve decapsulation to remove the second outer header (O2) and decryption to generate a second decrypted inner packet (I2, P2), thereby extracting the same inner packet content that is encrypted by EDGE 2 160 . Block 350 may also include authentication and other receive-side processing operation(s). At 350 in FIG. 3 , EDGE 1 150 may assign the second encrypted inner packet to a second processing unit (e.g., core 2 ) based on content of the second encrypted inner packet. This way, post-cryptography (e.g., post-decryption) processing may be distributed among multiple processing units based on (cleartext) content of decrypted inner packets before they are forwarded towards their respective destinations. See 105 - 108 in FIG. 1 .

As will be described using FIGS. 4, 5 and 6 , post-cryptography RSS may be performed for encapsulated encrypted packets (e.g., ENCAP 1 102 and ENCAP 2 106 ) belonging to a single security association (SA). In practice, the term “security association” may refer generally to a form of contract between a pair of tunnel endpoints (e.g., implemented by respective EDGE 1 150 and EDGE 2 160 ) detailing how to exchange and protect information exchange between them. By implementing post-cryptography RSS for a particular SA, network performance may be improved in scenarios where only one SA is negotiated per tunnel 180 . In practice, route-based VPN that has high throughput may lead to such configuration. Otherwise, endpoints may experience performance degradation over a single SA, especially when various stateful services (e.g., firewall) are enabled in addition to VPN.

As will be described using FIGS. 4, 5 and 6 , blocks 330 and 360 may involve any suitable assignment strategy. In one example, the first processing unit may be selected at block 330 based on header information (I1) of the first decrypted inner packet. Similarly, the second processing unit may be selected at block 360 based on header information (I2) of the second decrypted inner packet. The header information (e.g., 5 or 10 tuples) may include source address information, destination address information, source port number, destination port number and protocol information. This way, packets belonging to a particular flow may be assigned to the same processing unit. In another example, a load-based approach may be used to select the first processing unit based on the decrypted header information and a load level associated with the first processing unit to further improve resource utilization.

Example Architecture

FIG. 4 is a schematic diagram illustrating example architecture 400 of network device 150 configured to perform receive-side processing for encapsulated encrypted packets in SDN environment 100 . Example architecture 400 may include additional and/or alternative components than that shown in FIG. 4 . In the example in FIG. 4 , EDGE 1 150 may support any suitable packet processing engines, such as DPDK application 440 residing in user space 411 and kernel NIC interface (KNI) packet process 460 residing in kernel space 412 . Packets that travel between user-space DPDK application 440 and KNI packet process 460 is handled by network interface 450 (e.g., KNI). Network interface 450 supports transmit (TX) and receive (RX) queues 451 - 452 , which may be shared-memory-based queues that are accessible by both DPDK application 440 and KNI packet process 460 .

Packet processing by EDGE 1 150 will be explained using 401 - 408 . At 401 - 402 , in response to detecting a packet at first NIC 1 421 , the packet may be directed to DPDK application 440 in user space 411 . At 407 , after performing any necessary packet processing, DPDK application 440 may direct the processed packet towards second NIC 2 422 . Alternatively, at 403 , DPDK application 440 may direct the packet to network interface 450 via TX queue 451 . In this case, at 404 - 405 , KNI packet process 460 may retrieve (e.g., poll) packet(s) from TX queue 451 before performing any necessary packet processing and pushing the packet(s) into RX queue 452 of network interface 450 . At 406 , DPDK application 440 may retrieve (e.g., poll) packet(s) from RX queue for any necessary packet processing. At 407 - 408 , DPDK application 440 may perform necessary processing and forward packet(s) to second NIC 2 422 . It should be noted that a particular ingress packet (e.g., “PACKET IN” 401 ) is not necessarily the same as an egress packet (e.g., “PACKET OUT” 408 ) in FIG. 4 .

To support post-cryptography RSS, a receive-side datapath on DPDK application 440 may include various components 441 - 444 (to be explained below) and multiple processing units 445 denoted as core 1 , . . . , coreN (see also 151 in FIG. 1 ). Packets polled from NIC 1 421 (see 402 ) may be processed by DPDK application 440 using components 441 - 444 before the packets are directed towards second NIC 2 422 . It is not necessary to perform post-cryptography RSS on packets that are polled from RX queue 452 on KNI 450 (see 406 ). To facilitate post-cryptography RSS, it should be understood that DPDK application 440 may implement any alternative and/or additional components (or sub-components) than that shown in FIG. 4 . Depending on the desired implementation, example components 441 - 445 may have any suitable function names.

IPSec-Based VPN Example

Examples of the present disclosure will now be explained using IPSec-based VPN. It should be understood that any alternative and/or additional protocol(s) may be used. In more detail, FIG. 5 is a flowchart of example detailed process 500 for a network device to perform receive-side processing for encapsulated encrypted packets in SDN environment 100 . Example process 500 may include one or more operations, functions, or actions illustrated by one or more blocks, such as 510 to 580 . The various blocks may be combined into fewer blocks, divided into additional blocks, and/or eliminated depending on the desired implementation. The example in FIG. 5 will be explained using FIG. 6 , which is a schematic diagram illustrating example receive-side processing 600 for encapsulated encrypted packets according to the example in FIG. 5 .

(a) Tunnel Establishment

At 510 in FIG. 5 , tunnel 180 may be established between a first tunnel endpoint at EDGE 1 150 and a second tunnel endpoint at EDGE 2 160 to facilitate secure communication between them. During tunnel establishment, EDGE 1 150 and EDGE 2 160 may establish or negotiate an SA associated with tunnel 180 to specify various security attributes, such as cryptographic algorithms and keys. For example, EDGE 1 150 and EDGE 2 160 may negotiate which algorithms to encrypt packets and to check for data integrity, such as advanced encryption standard (AES), secure hash algorithm (SHA), AES Galois/Counter mode (AES-GCM), etc. The SA may be established manually (where tunnel attributes are static) or dynamically (where tunnel attributes are negotiated in real time).

Using IPSec for example, an SA may be uniquely identifiable using a security parameter index (SPI), source and destination address information, and a security protocol such as Encapsulating Security Payload (ESP) or Authentication Header (AH). Based on the SA, EDGE 2 160 may perform encryption and encapsulation for egress packets originating from source endpoints (e.g., VM 4 134 and VM 5 135 ) before forwarding encapsulated encrypted packets over tunnel 180 . At the receive-side, the reverse is performed. Based on the SA, EDGE 1 150 to perform decryption and decapsulation before forwarding decrypted packets towards destination endpoints (e.g., VM 1 131 and VM 2 132 ). To identify the SA, an associated SPI may be added an identification tag to an outer header of each encapsulated encrypted packet travelling over tunnel 180 . Although not shown for simplicity, each encapsulated encrypted packet may be padded with encryption-related data, such as ESP trailer data and ESP authentication data before being sent over tunnel 180 .

(b) Decryption and Decapsulation

At 520 and 530 in FIG. 5 , DPDK application 440 may receive (e.g., by polling) ingress packets from NIC 1 421 and determine whether they require decryption and decapsulation. If not (e.g., non-IPSec packets), a different datapath may be used to process the ingress packets. In the example in FIG. 6 , three example encapsulated encrypted packets are shown, i.e., labelled as ENCAP 1 (see 611 ), ENCAP 2 (see 612 ) and ENCAP 3 (see 613 ). Each encapsulated encrypted packet 611 / 612 / 613 includes an outer header (O1, O2, O3) that is addressed from source tunnel endpoint address=IP-TEP 2 associated with EDGE 2 160 to destination tunnel endpoint address=IP-TEP 1 associated with EDGE 1 150 . The outer header (O1, O2, O3) also specifies SPI=X that identifies a particular SA negotiated for tunnel 180 . See 601 , 603 in FIG. 6 .

ENCAP 1 611 , ENCAP 2 612 and ENCAP 3 613 include respective encrypted inner packets denoted as (I1*, P1*), (I2*, P2*) and (I3*, P3*). For ENCAP 1 611 and ENCAP 3 613 , their respective encrypted inner packets (I1*, P1*) and (I3*, P3*) are addressed from source address=IP-VM 4 associated with VM 4 134 to destination address=IP-VM 1 associated with VM 1 131 . For ENCAP 2 612 , encrypted inner packets (I2*, P2*) is addressed from source address=IP-VM 5 associated with VM 5 135 to destination address=IP-VM 2 associated with VM 2 132 . In other words, ENCAP 1 611 , ENCAP 2 612 and ENCAP 3 613 belong to the same SA (i.e., SPI=X), but two different packet flows. See 602 , 604 in FIG. 6 .

At 540 , 550 and 560 , DPDK application 440 may perform receive-side processing for ENCAP 1 611 , ENCAP 2 612 and ENCAP 3 613 , such as decapsulation, decryption, authentication, etc. Using IPSec as an example, DPDK application 440 may implement various modules for processing ENCAP 1 611 , ENCAP 2 612 and ENCAP 3 613 , including “ipsec_input” 441 , “crypto_pmd” 442 and “ipsec_input_finish” 443 . In this case, block 540 may involve “ipsec_input” 441 parsing the outer header (O1, O2, O3) of ENCAP 1 611 , ENCAP 2 612 and ENCAP 3 613 to identify an SA associated with SPI=X, and preparing any necessary transactions for a cryptography driver.

At 550 in FIG. 5 , DPDK application 440 may perform cryptography operations using “crypto_pmd” 442 , such as authentication (e.g., based on MAC computation) and decryption to convert an encrypted inner packet to a decrypted inner packet. In the example in FIG. 6 , ENCAP 1 611 , ENCAP 2 612 and ENCAP 3 613 may be transformed into respective decrypted packets denoted as DECRYPT 1 621 , DECRYPT 2 622 and DECRYPT 3 623 . Note that DECRYPT 1 621 includes a decrypted inner packet denoted as (I1, P1), DECRYPT 2 622 includes (I2, P2) and DECRYPT 3 623 includes (I3, P3).

At 560 in FIG. 5 , DPDK application 440 may perform decapsulation to remove the outer headers (O1, O2, O3) using “ipsec_input_finish” 443 . The results are denoted as INNER 1 631 , INNER 2 632 and INNER 3 633 in FIG. 6 . Block 550 may also involve completing various IPSec processing, such as checking a replay window, etc. In practice, any other packet information (header or trailer) not shown in FIG. 6 may be removed before being forwarded towards the destination. Using Generic Route Encapsulation (GRE) over IPSec as an example, GRE-related headers may be removed, such as a GRE header, etc. The GRE header is generally encrypted when sent over tunnel 180 .

(c) Post-Cryptography RSS

At 570 in FIG. 5 , DPDK application 440 may distribute post-cryptography processing over multiple processing units using “core_select” 444 . Block 570 may involve selecting a particular core_i from the set of [core 1 , . . . , coreN] based on cleartext or decrypted content of INNER 1 631 , INNER 2 632 and INNER 3 633 . Any suitable assignment or core selection approach may be used, such as tuple-based, load-based, etc. To reduce the amount of CPU time spent on dispatching packets to various processing units, it is not necessary to implement the same complex RSS algorithms used by hardware NICs. In practice, calculating RSS in software is generally very costly, and examples of the present disclosure may be used together with any suitable inner flow hashing approach for distributing (decrypted) inner packets among processing units 445 .

Using a tuple-based approach, block 570 may involve parsing the decrypted inner header to extract tuple information and calculating a hash value associated with the tuple information to select core 1 based on the hash value. One example is to map the tuple information to a particular processing core by applying an exclusive OR (XOR) operation as follows:

i=XOR of tuple modulo N.

Here, i∈{1, . . . , N} represents an index of the selected processing unit, N represents the total number of available processing units and tuple represents 5-tuple=(source IP address, destination IP address, source port number, destination port number, protocol) extractable from the decrypted inner header (I1, I2, I3).

In the example in FIG. 6 , INNER 1 631 and INNER 3 633 are both assigned to the same core 1 based on a first hash value (h 1 ) associated with a TCP flow between VM 4 134 and VM 1 131 . INNER 2 632 may be assigned to core 2 based on a second hash value (h 2 ) associated with a TCP flow between VM 5 135 and VM 2 132 . See 641 - 643 in FIG. 6 . Using the tuple-based approach, packets having the same tuple information and belonging to the same TCP flow may be assigned to the same processing core. This has various performance advantages compared to a random or round robin assignment, including reducing the likelihood of out-of-order packet processing, packet loss, etc.

In another example, a load-based approach may be implemented to select core_i based on a load level associated with each corei. For example, if the load level exceeds a predetermined threshold, a different corei may be selected. Subsequent packets from the same TCP flow may be assigned to the same corei. Depending on the desired implementation, the hash value may be calculated based on additional or alternative tuple information from inner packets 631 - 633 .

At 580 and 590 in FIG. 5 , decrypted inner packet 631 - 633 may be enqueued for subsequent post-cryptography processing by their selected corei. For example, an intercore data structure (e.g., a ring, buffer, etc.) may be configured to store pointers to respective decrypted inner packets 631 - 633 before they are processed by respective core 1 and core 2 . In this case, processing units 445 may poll packets from the intercore data structure for further processing. In practice, each corei may be a PMD thread configured to perform post-cryptography processing, such as to provide routing, firewall, policy-based routing, NAT and/or other networking service(s). If flow cache is enabled on EDGE 1 150 , a match-action type of processing may be implemented. This involves performing a lookup in a flow cache table and, if there is a match, the associated action(s). Once post-cryptography processing is completed, INNER 1 631 and INNER 3 633 may be forwarded by EDGE 1 150 towards destination VM 1 131 on host-A 110 A, and INNER 2 632 towards destination VM 2 132 on host-B 110 B.

Using examples of the present disclosure, RSS may be performed for packets belonging to the same SA, but different packet flows. This allows scaling of IPSec performance for a particular SA supported by tunnel 180 , such as in the case of route-based VPN. This should be contrasted against conventional approaches that are able to distribute traffic based on non-encrypted content of the outer header, such as the SPI. By performing load balancing or core selection based on decrypted content of packets, post-cryptography processing may be distributed more efficiently among multiple processing units 445 at EDGE 1 150 . Although explained using EDGE 1 150 , it should be understood that EDGE 2 160 may implement the examples of the present disclosure to perform receive-side processing for encapsulated encrypted packets on the reverse path, such as from VM 1 131 to VM 4 134 or from VM 2 132 to VM 5 135 .

Examples of the present disclosure may be implemented for tunnel 180 supporting multiple SAs. Using an example with two SAs, different packet flows belonging to a first SA may be distributed among a first subset of processing units 445 , such as core 1 , . . . , coreM, where M<N. Additionally, different packet flows belong to a second SA may be distributed among a second subset of processing units 445 , such as core(M+1), . . . , coreN. The size of the first subset (M) may be configured according to a priority level associated with the first SA. For example, more processing units may be allocated to the first SA to assign a higher priority level compared to the second SA.

Container Implementation

Although discussed using VMs 131 - 135 , it should be understood that receive-side processing for encapsulated encrypted packets may be performed for other virtualized computing instances, such as containers, etc. The term “container” (also known as “container instance”) is used generally to describe an application that is encapsulated with all its dependencies (e.g., binaries, libraries, etc.). For example, multiple containers may be executed as isolated processes inside VM 1 131 , where a different VNIC is configured for each container. Each container is “OS-less”, meaning that it does not include any OS that could weigh 10s of Gigabytes (GB). This makes containers more lightweight, portable, efficient and suitable for delivery into an isolated OS environment. Running containers inside a VM (known as “containers-on-virtual-machine” approach) not only leverages the benefits of container technologies but also that of virtualization technologies. Using the examples in FIG. 1 to FIG. 6 , receive-side processing for encapsulated encrypted packets may be performed to facilitate secure communication among containers located at geographically-dispersed sites in SDN environment 100 .

Computer System

The above examples can be implemented by hardware (including hardware logic circuitry), software or firmware or a combination thereof. The above examples may be implemented by any suitable computing device, computer system, etc. The computer system may include processor(s), memory unit(s) and physical NIC(s) that may communicate with each other via a communication bus, etc. The computer system may include a non-transitory computer-readable medium having stored thereon instructions or program code that, when executed by the processor, cause the processor to perform processes described herein with reference to FIG. 1 to FIG. 6 . For example, a computer system capable of acting as EDGE 1 150 or EDGE 2 160 may be deployed in SDN environment 100 to perform receive-side processing.

The techniques introduced above can be implemented in special-purpose hardwired circuitry, in software and/or firmware in conjunction with programmable circuitry, or in a combination thereof. Special-purpose hardwired circuitry may be in the form of, for example, one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), and others. The term ‘processor’ is to be interpreted broadly to include a processing unit, ASIC, logic unit, or programmable gate array etc.

The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or any combination thereof.

Those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computing systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure.

Software and/or other instructions to implement the techniques introduced here may be stored on a non-transitory computer-readable storage medium and may be executed by one or more general-purpose or special-purpose programmable microprocessors. A “computer-readable storage medium”, as the term is used herein, includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant (PDA), mobile device, manufacturing tool, any device with a set of one or more processors, etc.). A computer-readable storage medium may include recordable/non recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk or optical storage media, flash memory devices, etc.).

The drawings are only illustrations of an example, wherein the units or procedure shown in the drawings are not necessarily essential for implementing the present disclosure. Those skilled in the art will understand that the units in the device in the examples can be arranged in the device in the examples as described or can be alternatively located in one or more devices different from that in the examples. The units in the examples described can be combined into one module or further divided into a plurality of sub-units.