Efficient approach with the toleration of stale data to dynamically transform and unify data quality in client and server with continuous transaction flows

The invention relates to a method for increasing the consistency between client replication data quality information and server replication data quality information.

In the financial and trading and banking business, as well as media like Bloomberg®, CNBC®, and others, clients use time delay as a metric for replication data quality, where a delay of seconds is higher replication data quality than a delay of minutes. On the server side, the number of transactions behind represents replication data quality because transaction flow dictates the quality of the data. The fewer transactions behind, the higher the replication data quality. At any given time there could be a great number of transactions per second that require replication, for example 50,000 transactions per second, or very few transactions to replicate, for example 10 transactions per second. For a highly demanded stock like IBM®, transaction flow may be high, for example 10,000 transactions per second, whereas transaction flow for other stocks may be as low as, for example, 10 transactions per day. During peak hours, thousands of transactions may take place every minute. During off-peak hours, there may not be any transactions, for example, during the night.

In addition to media websites such as Bloomberg® and CNBC®, trading company websites such as E*Trade®, NYSE® and Fidelity® all use replicated data to accommodate a large number of clients. The replication process requires the transfer of transaction logs from the master server into various replica servers, and applies those transactions to the replica servers at different rates depending on the network, geographical distance, and the transaction rate. Accordingly, some replica servers get more data at a greater rate than other servers.

Under server snapshot replication, the server periodically takes a snapshot of the data and replicates this snapshot to other servers at some pre-defined interval. This differs from data backup because the snapshot replicas are for client access. Banks, for example, traditionally take a snapshot of the server once every 24 hours. One media traditionally took a snapshot of all headlines every 30 minutes and a stock price snapshot every 5 minutes. Snapshot replication, however, has many drawbacks that have driven most enterprises away from this replication method. The periodic nature of snapshot replication makes inefficient use of network and machines resources and causes uneven, oscillating server utilization by overloading the network and the machine during taking the snapshot while leaving the network and machine idle at other times. Snapshots also have a lower replication data quality and are less current than continuously replicated data.

Continuous replication, which has replaced snapshot replication in many applications because it better utilizes machine and network resources and provides better quality of data with less delay, continuously wires transaction logs to replicated servers, fully using network and machine capacity. Transactions may accumulate on one or more servers in the network, but once network and machine capacity allows, these accumulated transaction logs will be transferred and applied to replicated servers. Continuous replication, however, makes it difficult to measure server data quality because it does not have the periodic qualities of snapshot replication anymore; instead, server data quality changes every second as replication progresses and as network and machine capacities allow. Accordingly, there exists a large gap between clients' understanding and use of replication data quality information and the server's perspective on the replication data quality that continuous replication provides.

The present invention addresses the discrepancy between client replication data quality information and server replication data quality information by recording and transmitting the number of transactions the server is processing each second. This information may be cached and used to convert the delay into how many transactions lag the continuous replication. The present invention avoids calculation problems due to stale transactions per second data by, for example, recalculating the number of transactions the server is processing each second each time a request is received by the server. The recalculated values may be inserted into the replication data stream. The cached values may be invalidated and updated when, for example, a new transactions per second value is calculated.

Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the invention with advantages and features, refer to the description and to the drawings.

The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

FIG. 1 illustrates a flow chart according to the preferred method.

The detailed description explains the preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings.

The flowchart of FIG. 1 illustrates one embodiment of how a common client that uses time delay as a metric for replication data quality may communicate with a server that uses the number of transactions behind as a metric for replication data quality and obtain useful replication data quality information. Data reflecting the number of transactions the server is behind, which may be referred to as transaction lag TB, and number of transactions the server processes each second, which may be referred to as the transaction processing rate TPS, may be used to dynamically calculate what transaction lag corresponds to a maximum allowable time delay T a . Because transaction lag TB and transaction processing rate TPS both change constantly, this information may be updated each time the client makes a request to avoid stale data.

Referring to FIG. 1 , 10 a router may receive a client request. The router may then 20 determine the maximum time delay that the client that made the request will allow, which may be referred to as the maximum allowable time delay T a . If the request context includes explicit information regarding T a , the router may extract this information from the request context. If the request context does not include such information, the router may determine T a by determining the class of the client from the request context and using that class information to look up the maximum allowable time delay for that class of clients. The router may use a map table that provides the maximum allowable delays for various client classes to look up the maximum allowable delay for that class of clients.

The router may then 30 check its cache to see if the transaction processing rate TPS for this class of clients is available locally. If a member of this class has previously made a request, the cache may contain the TPS. Similarly, if the client has previously made a request, cache at the client may contain the TPS. If the client request is the first request by a member of the client's class, 40 b the transaction processing rate TPS is fetched from the server and cached. Otherwise, 40 a the TPS is fetched from the cache.

Next, 50 the router may convert the maximum allowable time delay T a into the maximum allowable transaction lag TBA by multiplying T a by the transaction processing rate TPS:

The router may then 60 insert TBA and T a into the client request stream context. The router may then 70 search its cache to identify a server that with a cached transaction lag TB that is less than the maximum allowable transaction lag TBA and is therefore capable of handling the request. The router may then 80 route the request to the identified server.

The server, which has current transaction processing rate information TPS current , may then 90 calculate a new value for a maximum allowable transaction lag TBA new using TPS current and the maximum allowable time delay T a extracted from the request context:

TBAnew=T a * TPS current

The server may then 100 determine the difference between the client-calculated maximum allowable transaction lag TBA and the server-calculated maximum allowable transaction lag TBA new :

=TBA new −TBA

If 110 the difference is positive, the server's current transaction processing rate TPS current is greater than the transaction processing rate TPS cached at the router or client. If the difference is negative, however, the server's current transaction processing rate TPS current is less than the cached transaction processing rate TPS.

If TPS current is less than the cached transaction processing rate TPS, the server may 120 check to determine whether the current transaction lag TB current is less than the server-calculated maximum allowable transaction lag TBA new and the server therefore remains capable of handling the request. If TB current is greater than the server-calculated maximum allowable transaction lag TBA new , the server 130 forwards the request to another server. Otherwise, the server 140 processes the request. The server 150 inserts the current transaction processing rate TPS current and the current transaction lag TB current into its response to the request. The router and the client may extract and cache TPS current and TB current for use with future requests. The response may be routed to users of the data.

While the preferred embodiment of the invention has been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described.