Methods, systems, and computer readable media for monitoring, adjusting, and utilizing latency associated with accessing distributed computing resources

The subject matter described herein relates to monitoring latency associated with accessing distributed computing resources. More particularly, the subject matter described herein relates to monitoring, adjusting, and utilizing latency associated with accessing distributed computing resources.

As computing has changed from an on-premises model to a cloud services model where resources are distributed, differences in latency can adversely affect a user's experience and even prevent applications from functioning. For example, a user accessing cloud service expects the latency in accessing the service to be the same or nearly the same each time the user accesses the service, regardless of the physical location from which the service is provided.

Applications, such as database mirroring and synchronization applications, likewise expect minimal variation in latency between accesses to database resources located at different sites. For example, a database application synchronizing its local database with databases located at different remote sites expects the latency associated with the synchronization operations to be substantially equal. If the latency associated with synchronizing a database located at one site is different from the latency associated with synchronizing a database located at a different site is not equal, the database synchronization application may block access to the database at all sites until all synchronization operations are complete.

Another problem associated with variations in latencies between distributed computing resources is that such variations can mask network security issues. For example, a non-deterministic latency variation can be exploited by an attacker to mask attacks that involve intercepting traffic, modifying the traffic to include malicious code, and retransmitting the modified traffic to a target network. Even though intercepting, modifying, and retransmitting packets introduces latency into transactions involving the packets, such latency may be difficult to detect if variation in the natural latency (i.e., latency not caused by attacks) is non-deterministic.

Accordingly, in light of these difficulties, there exists a need for monitoring, adjusting, and utilizing latency associated with accessing distributed computing resources.

Methods, systems, and computer readable media for monitoring, adjusting, and utilizing latency associated with accessing distributed computing resources are disclosed. One method includes measuring a first latency associated with accessing a first computing resource located at a first site. The method further includes the measuring a second latency associated with accessing a second computing resource located at a second site different from the first site. The method further includes selectively impairing transmission of packets to or processing of packets by at least one of the first and second computing resources in accordance with a performance, network security, or diagnostic goal.

The term “computing resource”, as used herein, refers to any one or more of processing, storage, or other resources involving one or more computers.

The subject matter described herein may be implemented in hardware, software, firmware, or any combination thereof. As such, the terms “function” “node” or “module” as used herein refer to hardware, which may also include software and/or firmware components, for implementing the feature being described. In one exemplary implementation, the subject matter described herein may be implemented using a computer readable medium having stored thereon computer executable instructions that when executed by the processor of a computer control the computer to perform steps. Exemplary computer readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media, such as disk memory devices, chip memory devices, programmable logic devices, and application specific integrated circuits. In addition, a computer readable medium that implements the subject matter described herein may be located on a single device or computing platform or may be distributed across multiple devices or computing platforms.

The subject matter described herein will now be explained with reference to the accompanying drawings of which:

FIG. 1 is a network diagram demonstrating an exemplary operating environment for monitoring, adjusting, and utilizing latency associated with accessing computing resources locating at different sites according to an aspect of the subject matter described herein;

FIG. 2 is a network diagram illustrating a method for measuring latencies associated with accessing computing resources located at different sites according to an aspect of the subject matter described herein;

FIG. 3 is a network diagram illustrating an alternate method for measuring latencies associated with accessing computing resources located at different sites including to an aspect of the subject matter described herein;

FIG. 4 is a network diagram illustrating selective impairment of packets transmitted to computing resources located at different sites according to an aspect of the subject matter described herein;

FIG. 5 is a network diagram illustrating an exemplary process for utilizing variations in latency to detect an attack according to an aspect of the subject matter described herein;

FIG. 6A is a flow chart illustrating an exemplary process for monitoring and adjusting latency associated with accessing computing resources located at different sites according to an aspect of the subject matter described herein; and

FIG. 6B is a flow chart illustrating an exemplary process for utilizing variations in latency to detect attacks according to an aspect of the subject matter described herein.

Methods, systems, and computable readable media for monitoring, adjusting, and utilizing latency associated with accessing distributed computing resources are disclosed. FIG. 1 is a network diagram illustrating an exemplary operating environment for the subject matter described herein. Referring to FIG. 1 , an enterprise data center 100 includes computing resources, such as one or more processors 102 and memory 104 for providing computing services, such as cloud computing and/or storage services. To provide such services in a geo-redundant manner, enterprise data center 100 will have access to computing resources located at geographically distributed sites. In the illustrated example, enterprise data center 100 has access to a first computing resource 106 located at a first site and a second computing resource 108 located at a second site, where the first and second computing sites are geographically distributed from each other. Each computing resource 106 and 108 may include one or more processors 102 and memory 104 . Together, data center 100 and remote computing resources 106 and 108 may be cloud computing resources that provide cloud computing services.

In addition, in the illustrated example, distributed computing resources 106 and 108 each include an application 110 that provides a service. Application 110 , in one example, may be a database application that accesses a database. Application 110 may provide the same service at either of the geographically distributed computing sites.

Because application 110 implemented at computing resource 106 provides the same service as application 110 at computing resource 108 , data center 100 can request service from either instance of application 110 . The latency experienced by data center 100 or the end user may vary depending on which instance of application 110 is providing the service. The latency variation can be caused by differences in relative congestion in the networks between data center 100 and computing resources 106 and 108 or by the relative loading of computing resources 106 and 108 . Such differences latency can cause applications to fail and or can result in security threat. Regarding security threats, an attacker can intercept and modify requests between data center 100 and computing resources 106 and 108 without detection if the latency between data center 100 and computing resources 106 and 108 is not deterministic.

FIG. 2 is a network diagram illustrating an aspect of the subject matter described herein where enterprise data center 100 includes a latency measurement module 200 that measures round trip packet latencies between data center 100 and remote computing resources 106 and 108 . In FIG. 2 , latency measurement module 200 may be implemented by processor 102 and stored in memory 104 . Latency measurement module 200 measures round trip latency by transmitting test packets, illustrated as test request A and test request B, to each of remote computing resources 106 and 108 , receiving responses from remote computing resources 106 and 108 , and measuring the round trip time from the times that requests are transmitted to remote computing resources 108 to the times that responses are received from remote computing resources 106 and 108 . In this example, the latency includes the network latency and the latency caused by processing at remote computing resources 106 and 108 . Latency measurement module 200 may maintain a data store 202 of per site latency measurements. In the illustrated example, the average round trip latency between data center 100 and remote computing resource 108 is 79 milliseconds, while the average round trip latency for the same type of transaction between data center 100 and remote computing resource 106 is 11 milliseconds.

Although in the example illustrated in FIG. 2 , latency measurement module 200 measures latency based on test packets generated by latency measurement module 200 , the subject matter described herein is not limited to such an implementation. In an alternate implementation, latency measurement module 200 may measure the latency of live network traffic between enterprise data center 100 and each of remote computing resources 106 and 108 . For example, if a non-test application associated with data center 100 sends a request packet to remote computing resource 108 , latency measurement module 200 may record the outgoing transmission timestamp of the request packet. When a response packet from remote computing resource 108 is received by data center 100 , latency measurement module 200 may record the time of receipt. The latency may then be recorded for the particular transaction and computing site to be the difference between the transmission time of the request packet and the time of receipt of the response packet. Similar measurements may be performed for transactions between data center 100 and remote computing resource 106 to calculate and store the latency between enterprise data center 100 and remote computing resource 106 .

It should be noted that in this example, the latency measured is round trip latency. It is also possible to measure one way latency. One way latency can be estimated by dividing the round trip latency by two, assuming that the delay in the network is relatively symmetric. If the network delay is not symmetric, one-way latency can be measured using the method described in commonly assigned, co-pending U.S. Patent Application Publication No. 2016/0301589, the disclosure of which is incorporated herein by reference in its entirety. Briefly, this method includes transmitting a packet from a first network device, intercepting the packet, and transmitting a copy of the packet back to the first network device. Another copy of the packet is sent to a second network device. The first network device records the time of receipt of the copy of the packet that it received. The second network device records the time of the receipt of the copy of the packet that it received. The second network device transmits the packet back to the first network device where the packet is intercepted and the difference between the recorded times of receipt of the two copies by the first and second network devices is computed as the one way link delay from the first network device to the second device. The one-way link delay may be used as a measurement of one-way latency.

In yet another alternate implementation, latency measurement module 200 may measure latency between enterprise data center 100 and each of the remote computing resources 106 and 108 using a bypass switch located at each of remote computing resources 106 and 108 to loop test packets transmitted by data center 100 back to data center 100 . FIG. 3 illustrates this example. In FIG. 3 , latency measurement module 200 transmits test packets to each of remote computing resources 106 and 108 . A bypass switch 300 located at each of remote computing resources 106 and 108 loops the test packet back to latency measurement module 200 . Latency measurement module 200 measures the round trip latency associated with a network between enterprise data center 100 and remote computing resources 106 and 108 based on the transmission time of each request message and the time that the request is looped back and received by data center 100 . Such round trip latency does not include processing latency at the remote computing resources. However, measuring network delay can be useful to isolate the causes of latency problems. Again, one way latency can also be determined using the methodology illustrated in FIG. 3 by dividing the round trip latency by two. However, such an approximation is only valid if the latency in the network is relatively symmetric. In the case of asymmetric network delay, an alternate method, such as the one described above can be utilized.

Once latency between distributed computing resources has been measured, the latency can be adjusted according to a desired performance, diagnostic, or network security goal. In one example, it may be desirable to ensure that the latency between services provided by different computing resources is substantially equal. FIG. 4 illustrates an exemplary implementation of a latency adjustment module 400 implemented by processor 102 in data center 100 that selectively impairs transmission of packets transmitted to remote computing resources 106 and 108 to achieve either latency equalization or a difference in latency that is within the tolerance of an application. Referring to FIG. 4 , latency measurement module 200 measures latency between data center 100 and remote computing resources 106 and 108 using any of the methods described above. Based on these measurements and a desired performance, diagnostic, or network security goal, latency adjustment module 400 adjusts the latency of packets transmitted to remote computing resources 106 and/or 108 . In the illustrated example, latency associated with accessing remote computing resource 106 is measured or calculated as 60 ms and latency associated with accessing computing resource 108 is measured or calculated as 80 ms. If it is desirable to equalize the latency, latency adjustment module 400 may add 20 milliseconds of delay to packets transmitted from data center 100 to remote computing resource 106 relative to packets transmitted from data center 100 to remote computing resource 108 . In the illustrated example, the adjusted latency for the transaction between data center 100 and remote computing resource 106 involving request C and response C is 80 ms. The adjusted latency matches the measured latency for the transaction involving request A and response A between data center 100 and remote computing resource 108 . Latency measurement and selective insertion of delay impairment may be performed continually so that the latency experienced by data center 100 for transactions with remote applications 110 is substantially equal.

In another example, it may not be desirable to maintain equal latency between transactions performed by different remote sites but instead to maintain deterministic but unequal latency. Maintaining deterministic but unequal latency may include selectively impairing packets between data center 100 and remote computing resources 106 and 108 so that the latency variation and/or the difference in measured latency is deterministic. In one example, it may be desirable to adjust latency so that the latency experienced by packets accessing application 110 at remote computing resource 106 is within the tolerance of an application with respect to delays experienced by packets accessing application 110 at remote computing resource 108 . In another example, it may be desirable to modulate latency by selective impairment of the transmission of packets from data center 100 so that the latency or the difference in latencies between data center 100 and remote computing resources 106 and 108 varies according to a schedule or deterministic pattern.

FIG. 5 is a network diagram illustrating the use of variations from adjusted latency to detect the presence of an attack. Referring to FIG. 5 , in addition to latency measurement module 200 and latency adjustment module 400 , data center 100 includes an attack detection module 500 . Attack detection module 500 monitors latency of transactions between data center 100 and remote computing resources 106 and 108 . Attack detection module 500 determines whether the monitored latency varies according to the predetermined schedule. If attack detection module 500 determines that latency associated with accessing remote computing resources 106 and 108 indicates the presence of an attack. Continuing with the example from FIG. 4 , if latency adjustment module 400 adds 20 ms of latency to request C, the expected latency for the transaction involving request C and response C would be 80 ms, as illustrated in FIG. 4 . However, in FIG. 5 , an attacker intercepts response C, modifies response C to include attack code, and retransmits response C as response C*.

When enterprise data center 100 receives response C*, attack detection module 500 determines that the round trip latency for response C* is 100 ms, which is a 25% increase over the expected round trip latency of 80 ms for transactions with remote computing resource 106 . Accordingly, attack detection module 500 may determine that the variation in latency indicates the presence of an attack and generate an alarm or take other action, such as quarantining response C* for offline analysis in a protected environment.

In yet another example, latency adjustment module 400 may adjust the latency for transactions between data center 100 and either or both of remote computing resources 106 and 108 to vary according to a predetermined schedule. For example, latency adjustment module 400 may adjust the latency for transactions between data center 100 and remote resource 106 to be 60 ms for a time period, 80 ms for a subsequent time period, and then back to 60 ms in a repeating pattern. If an attacker intercepts and inserts attack code into packets between data center 100 and remote computing resource 106 , attack detection module 500 will detect a variation in latency from the predetermined pattern. If such a variation is detected, attack detection module 500 may generate an alarm and or take other appropriate action.

FIG. 6A illustrates an exemplary process for measuring and adjusting latency associated with transactions involving distributed computing resources according to the aspect of the subject matter described herein. Referring to FIG. 6A , in step 600 , latencies associated with accessing remote computing resources are measured. The measuring can be performed by latency measuring module 200 using any of the methods described above. In step 602 , it is determined whether the measured latency achieves a desired latency goal, such as a performance, diagnostic, or network security goal. If the latency achieves the desired goal, control returns to step 600 where the process of measuring and comparing the latency to a desired goal is repeated.

If, in step 602 the latency does not achieve the desired goal, control proceeds to step 604 where the latency is adjusted in accordance with the goal. The adjustment will be effected using latency adjustment module 400 to selectively impair packets transmitted to one or both remote computing resources. Control then returns to step 600 where the process of measuring and comparing latency to the desired goal re-starts.

The process for determining whether variations in latency indicate an attack may be performed in parallel with the latency measured and adjustment steps in FIG. 6A . FIG. 6B illustrates exemplary steps that may be performed in determining whether variations in latency associated with accessing remote computing resources are indicative of an attack. Referring to FIG. 6B , in step 604 A, latency associated with accessing a remote computing resource is monitored. For example, attack detection module 500 may monitor latencies of packets accessing remote computing resources 106 and 108 . In step 604 B, it is determined whether the monitored latencies are indicative of an attack. As described above, in one example, an attack may be indicated if latency of a given packet varies more than a threshold amount from an average latency or from an adjusted latency. If the monitored latency is indicative of an attack, control proceeds to step 604 A where an indication of an attack (such as an alarm) is generated. Control then returns to step 600 A where latency monitoring is continued. If in in step 602 A, it is determined that an attack is not indicated, control returns to step 600 A where latency monitoring and attack detection continues.

Although in the examples described above, latency is measured and adjusted at enterprise data center 100 , the subject matter described herein is not limited to such an implementation. In an alternate implementation, latency can be measured in any computing site that contains computing resources. For example, in FIG. 2 , latency adjustment, measurement, and attack detection can be performed at any one or more of data center 100 , remote computing resources 106 , or remote computing resources 108 .

In the examples described above, latency adjustment includes equalizing latency associated with application computing resources at different sites. In another example, latency adjustment includes modulating latency so that latency between two or more computing resources varies deterministically. In yet another alternate example, latency adjustment module 400 may adjust latencies to give priority to one computing site over another. For example, in FIG. 4 , if remote computing resource 106 is assigned a higher priority than remote computing resource 108 , latency adjustment module 400 may adjust the latency so that transactions involving remote computing resource 106 experience less latency than transactions involving remote computing resource 108 .

In the examples described above, latency is measured and adjusted on two different network paths. However, the subject matter described herein is not limited to the measuring and adjusting latency on only two network paths. Measuring and adjusting latency on any number of network paths to achieve a desired performance, security, or diagnostic goal is intended to be within the scope of the subject matter described herein. For example, in a cloud computing network where computing resources are accessible via n different network paths, n being an integer, latency may be equalized across all n of the network paths. In another example, latency may be adjusted on the n network paths to achieve a desired priority among the n network paths or their associate resources.

In the examples described above, latency is adjusted to achieve a desired performance or security goal. Latency may also be adjusted according to a desired diagnostic goal. For example, in FIG. 4 , latency adjustment module 400 may gradually increase the latency experienced by transactions involving remote computing resources 106 and 108 . A performance monitor associated with enterprise data center 100 may monitor the response of data center 100 or computing resources 106 and 108 to the increase in latency. For example, the performance monitor may determine whether enterprise data center 100 properly fails over to another remote computing site when the latency associated with one remote computing site exceeds a latency threshold. Similarly, if latency adjustment module 400 decreases the latency from a value that caused a failover back to a natural latency level, the performance monitor may determine whether enterprise data center 100 fails back to using the remote computing resource for which the latency is decreased. The performance monitor may also monitor the performance of applications to see whether they crash or recover as latency is changed.

In the examples described above, latency is measured and adjusted on a per packet or per site basis. In an alternate implementation, latency may be measured and adjusted on a per link (physical or virtual), per port (physical or virtual), per traffic type (e.g., all hypertext transfer protocol (HTTP) traffic, all real time transport protocol (RTP) traffic, etc.), or per application (all Netflix traffic, all Twitter traffic, all Facebook traffic, etc.). Latency may be measured and adjusted, for example, on a per application basis and the performance of the application to the change in latency may be monitored.

In the security examples described above, variations in latency are used to detect an attack. The variations used for attack detection may be variations in latency over time on one link or across more than one link. In addition, a deterministic latency signature of a link or a group of links may be changed over time to change the latency baseline used to detect potential attacks. If an attack is detected, in the examples described above, an alarm is generated. Other actions that can be taken include isolating a resource, taking down a link, killing a virtual machine (VM) instance, etc. In one example, attack detection module 500 may access a data structure that contains security rules corresponding to different applications or computing resources. For example, if a potential attack is detected due to a variation in latency associated with one or more cloud sites, attack detection module 500 may look up the cloud sites in the data structure, access the corresponding rule and perform the security action or actions specified by the rule. Examples of security actions include any of the actions described above.

It will be understood that various details of the presently disclosed subject matter may be changed without departing from the scope of the presently disclosed subject matter. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation.