Ingesting data from managed elements into a data analytics platform

A portion of the disclosure of this patent document may contain command formats and other computer language listings, all of which are subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

This Application is related to U.S. patent application Ser. No. 13/832,433 entitled “METHOD AND APPARATUS FOR MULTI-DOMAIN WRITING OF APPLICATIONS USING HTML5 POSTMESSAGE” filed on Mar. 15, 2013 the teachings of which applications are hereby incorporated herein by reference in their entirety.

This application relates to data analytics, specifically communication of data for analysis in a computer network.

Time series data (e.g., a collection of CPU utilization measurements on a set of servers over a period of several days) is a key data source for IT analytics that helps data center administrators manage the health of their information systems and monitor the performance and availability of the services the information systems provide to an organization. Gathering time series data from its source (e.g., element managers responsible for monitoring individual devices and IT infrastructure components) into an analytics data warehouse is a difficult task.

One approach is to build bespoke (i.e., custom, build-to-order) collectors for each element manager, using the data export protocols exposed by those element managers. This approach can yield good runtime performance, but is expensive to produce and maintain as the number of different time series data sources increases, and is dependent on good performing data export protocols to be available from the data source. Furthermore, many element managers mask the inherent parallelism available, by aggregating data collected from multiple IT infrastructure components (e.g., routers, servers, virtual machines, network nodes, arrays, switches, etc.). This aggregation can prevent collecting information from the element manager in a scale out fashion. This may be referred to as a funneling effect.

Example embodiments of the present invention relate to a method, a system, and a computer program product for data analytics. The method includes receiving a plurality of first data streams from respective managed elements in a network and partitioning data of the plurality of first data streams according to an attribute regarding the data into a partitioned second data stream. The partitioned second data stream then may be streamed toward a data analytics platform for consumption by the data analytics platform. In a preferred embodiment, WebSockets are used.

The above and further advantages of the present invention may be better under stood by referring to the following description taken into conjunction with the accompanying drawings in which:

FIG. 1 is a block diagram of a network including an element manager for ingesting data from a plurality of managed elements into a data analytics platform according to an example embodiment of the present invention;

FIG. 2 is a flow diagram illustrating a method according to an example embodiment of the present invention;

FIG. 3 is a block diagram illustrating an apparatus according to an example embodiment of the present invention; and

FIG. 4 is an illustration of an example embodiment of the present invention embodied as a computer program product.

A recent trend is to make time series data available through REST APIs. ViPR by EMC Corporation of Hopkinton, Mass., for example, exports time series data through a REST API. Although REST APIs are simple and based on industry standards (e.g., HTTP/S), it is very difficult to get good performance. In particular, it is often critical to get a large volume of metrics data collected over a brief period of time, in support of near-real time performance and availability reporting. This is challenging to do through a REST API because it is extremely difficult to have multiple parallel ingestion processes work against the same REST API (i.e., the request/response paradigm of REST APIs is fundamentally suboptimal with respect to efficiency and latency). It should be understood that, while polling data directly from managed elements is naturally parallelizable, polling data from an element manager, which can be seen as a single entity, is not naturally parallelizable.

For scale-out systems, such as ViPR, importing data to another system (e.g., data analytics platform), such as Watch4Net by EMC Corporation of Hopkinton, Mass., restriction to a single “pipe” (e.g., API) for data transfer causes significant issues with respect to efficiency and latency because of the “funneling effect” (i.e., the workload to gather data from the plurality of managed elements that was done by a plurality of, for example, servers is now handled by a single stream). Therefore, example embodiments of the present invention overcome these and other deficiencies by using the WebSockets protocol to achieve a performant, real time, and parallelizable mechanism to ingest data. As understood in the art, Web Sockets is an HTML5 standard developed as a protocol upgrade over HTTP/S that provides a very efficient, low latency mechanism to communicate data bi-directionally between clients and servers and, like HTTP/S, leverages existing TCP/IP networking infrastructure and web application layer devices and software.

Example embodiments of the present invention leverage both the asynchronous nature of WebSockets in combination with REST. The asynchronous nature of WebSockets which allows data to be pushed from the data source, instead of polling for data, significantly reduces latency as compared to HTTP/S polling. Additionally, the use of REST allows the data source to provide data streams specific to a device or a set of devices, thereby allowing increased parallelism in the ingestion process.

FIG. 1 is a block diagram of a network 100 including an element manager 130 (as part of a scale-out system 120 ) for ingesting data from a plurality of managed elements (e.g., managed element 1 110 1 -managed element N 110 N ) (generally 110 ) into a data analytics platform 170 according to an example embodiment of the present invention. It should be understood that there may be thousands of managed elements 110 in the network 100 and that each managed element 110 uses an asynchronous link 115 (e.g., WebSocket) to push data to the element manager 130 . FIG. 1 may be studied in conjunction with FIG. 2 which is a flow diagram illustrating a method according to an example embodiment of the present invention.

As illustrated in FIGS. 1 and 2 , the element manager 130 may include a receiver module 140 , a partition module 150 , and a stream module 160 . The receiver module 140 may receive a plurality of first data streams 112 1 - 112 N (generally 112 ) from respective managed elements 110 1 - 110 N in the network 100 ( 202 ). As will be described in greater detail below, the partition module 150 may partition data of the plurality of first data streams 110 according to an attribute regarding the data streams 112 into a partitioned second data stream 122 1 - 122 M (generally 122 ) ( 204 ). The streamer module 160 then may stream the partitioned second data stream 122 toward a data analytics platform 170 for consumption by the data analytics platform 170 ( 206 ).

It should be understood that the receiver module 140 receives the plurality of first data streams 112 from respective managed elements 110 in the network 100 over respective asynchronous links 115 and, similarly, the streamer module 160 streams the partitioned second data streams 122 toward the data analytics platform 170 over respective asynchronous streams 125 for consumption by the data analytics platform 170 . In a preferred embodiment, these links 115 and streams 125 are WebSockets.

To partition the data of the plurality of first data streams 110 , the partition module 150 may determine the attribute regarding the data streams 122 according to which the plurality of first data streams 112 are to be partitioned and then filter the data of the plurality of first data streams 112 according to the determined attribute. To determine which attributes over which the partition module 150 should partition the data of the plurality of first data streams 112 , the receiver module may receive a subscription request 172 1 - 172 M (generally 172 ) from the data analytics platform 170 identifying the attribute for filtering the data of the plurality of first data streams 112 . Therefore, example embodiments of the present invention partition the data of the plurality of first data streams 112 based on a subscription 172 to the partition instead of performing a bulk copy (e.g., SQL ETL).

The partition module 150 , optionally in conjunction with the streamer module 160 , then may generate the partitioned second data stream 122 for streaming toward the data analytics platform 170 according to the filtered data of the plurality of first data streams 112 filtered by the partition module 150 , such as by selecting (i.e., allocating) for inclusion in the partitioned second data stream 122 only the selected data of the plurality of first data streams 112 having the identified attribute as a data attribute.

It should be understood that the partition module 150 may partition the data of the plurality of first data streams 110 according to one or more attributes (e.g., individual plural attributes or a hierarchy of attributes) regarding the data streams 112 into one or more respective partitioned second data streams 122 for parallel streaming toward the data analytics platform 170 .

Additionally, the one or more attributes regarding the data streams 112 may be regarding the data, itself, of the data streams 112 or regarding the managed element 110 sending the data streams 112 . For example, the one or more attributes regarding the data streams 112 may be regarding the content of the data or attributes of the managed element 110 .

Further, although N first data streams 112 and M partitioned second data streams 122 are shown in FIG. 2 , the ratio of M:N is not limited to a 1:1 ratio and may be any ratio of N first data streams 112 to M partitioned second data streams 122 such that the data of the plurality of first data streams 112 received by the receiver module 140 is parallelized over the plurality of M partitioned second data streams 122 streamed from the streamer module 160 toward the data analytics platform 170 for consumption by the data analytics platform. It should be understood that the data analytics platform 170 may direct how many partitioned data streams 122 it receives from the streamer module 160 based on resources available at the data analytics platform 170 . Therefore, example embodiments of the present invention are able to distribute the data of the plurality of first data streams 112 without a restriction of a single API and without exposing the underlying architecture of the scale out system 120 .

FIG. 3 is a block diagram of an example embodiment apparatus 305 according to the present invention. The apparatus 305 may be part of a system 300 and includes memory 310 storing program logic 315 , a processor 320 for executing a process 325 , and a communications I/O interface 330 , connected via a bus 335 .

The methods and apparatus of this invention may take the form, at least partially, of program code (i.e., instructions) embodied in tangible non-transitory media, such as floppy diskettes, CD-ROMs, hard drives, random access or read only-memory, or any other machine-readable storage medium. When the program code is loaded into and executed by a machine, such as the computer of FIG. 3 , the machine becomes an apparatus for practicing the invention. When implemented on one or more general-purpose processors, the program code combines with such a processor to provide a unique apparatus that operates analogously to specific logic circuits. As such, a general purpose digital machine can be transformed into a special purpose digital machine.

FIG. 4 shows program logic 455 embodied on a computer-readable medium 460 as shown, and wherein the logic 455 is encoded in computer-executable code configured for carrying out the methods of this invention, thereby forming a computer program product 400 .

The logic for carrying out the method may be embodied as part of the aforementioned system, which is useful for carrying out a method described with reference to embodiments shown. For purposes of illustrating the present invention, the invention is described as embodied in a specific configuration and using special logical arrangements, but one skilled in the art will appreciate that the device is not limited to the specific configuration but rather only by the claims included with this specification.

Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present implementations are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.