Automatic capture of minimal portable executable bug repros

This invention relates generally to query optimizers, and more particularly to analyzing optimizers and other types of software components upon the occurrence of errors or crashes to determine the causes, i.e., to “troubleshoot” or “de-bug” the component.

Reproducing bugs (defects, exceptions, runtime errors, etc.) in software components is typically difficult and time-consuming. Obtaining the necessary information to reproduce the problem is often difficult, if not impossible, and requires significant manual effort. Collecting bug details must usually be done after a crash when the system is in an unstable state and some of the necessary troubleshooting information may be unrecoverable.

Query optimizers are among the most complex software components in any database system, and they naturally tend to contain software defects which cause unexpected errors. Despite significant efforts in quality assurance, unexpected errors are occasionally encountered. A self-contained “bug repro” that enables replication of the problem is often the best approach to a speedy resolution. Without a live reproduction scenario, diagnosing and de-bugging software defects in a query optimizer is usually very time consuming and costly. Capturing all the information needed to recreate an issue in the laboratory setting that was originally encountered on a production system is a difficult challenge that requires significant manual interaction, skill, and expertise. This is particularly so for query optimizers. Repros are notoriously difficult to obtain as they require schema definition, the offending query, the system configuration, and many other pieces of data that are difficult to capture in a consistent and accurate way.

There is a need for a mechanism for automatically capturing self-contained executable repro files for problems that occur in software components, such as query optimizers, which is triggered automatically upon the occurrence of a problem with the software component, or upon request. The repro files need to capture the necessary input to the software components and the relevant system state necessary to reproduce the problem, and they should be minimal and portable. It is to these ends that the present invention is directed.

FIG. 1 is a block diagram giving the architecture of an optimizer of the type with which the invention may be employed and showing its relationship to a database system;

FIG. 2 is a diagrammatic view illustrating a workflow of an embodiment of a process in accordance with the invention; and

FIG. 3 is a block diagram illustrating the replay of an executable repro file in accordance with the invention.

The invention is particularly well adapted for use in troubleshooting and debugging query optimizers, and will be described in that context. As will become apparent, however, this is illustrative of only one utility of the invention and that the invention may be used with other types of software components.

As will be described in more detail below, the invention provides an automatic capture process for obtaining minimal, portable and executable bug repros upon the occurrence of a software crash. It overcomes the major challenges discussed above by providing an automatic capture mechanism that captures the problem environment accurately, i.e., in a transactionally consistent manner, and does so while the system is potentially in a degraded state. The problem capture mechanism of the invention is accurate, which is crucial for its usefulness and the success in a highly concurrent system with constantly changing data sets where the time window for collecting the evidence may be very small and often renders collection of data after the fact useless. The invention also meets the challenge of capturing data while the system itself is in an unstable state, be it because of a lack of resources, e.g., being out of memory, or due to inconsistent internal data structures that were either detected before or triggered an error or crash. For capture, the invention does not require additional resources to be allocated, and minimizes data structure accesses to avoid causing additional errors.

Preferably, the automatic capture mechanism of the invention is built into the query optimizer, and the architecture of the optimizer is preferably such that it serializes, as part of its regular workflow, all of the input parameters of an optimization, i.e., the query statement, the referenced metadata and the configuration parameters, as well as a synopsis of the execution environment prior to performing the actual optimization. When an exception occurs, the previously serialized artifacts may be automatically combined and supplemented with optimization run-time information, such as stack traces of involved threads and states of transient objects, using pre-allocated resources. The results may be then dumped to an executable, self-contained, fully portable capture file (a repro) that contains all the parameters needed to rerun the optimizer process and recreate the problem in a troubleshooting environment.

FIG. 1 is a block diagram illustrating a query optimizer 100 embodying the invention, and showing the relationship of the optimizer to a database system 110 . The optimizer 100 may comprise an optimizer engine 114 and a metadata cache 116 . It preferably runs as a separate process completely independently of the database system 110 and does not rely upon any database management system (DBMS) facilities. The optimizer is also preferably authored in a different language from the database system and has its own operating system (OS) abstraction layer 120 .

Database system 110 may be a standard database system, such as available from EMC/Greenplum (and referred to as a GPDB) comprising a computer system connected to data stores (not shown in the figure), and a memory comprising computer readable media for storing executable instructions for controlling the operations of the computer. The instructions control the computer to provide a query parser 124 , a catalog and an executor 128 , among other components as will be described.

The query optimizer preferably communicates with the database system using an XML-based encoding referred to herein as DXL for data exchange. The DXL encoding provides an abstraction between the optimizer and a database system that interacts with the optimizer, and because it is XML-based, it affords a universal description of metadata and relational expressions, yet it may also have custom extensions for system specific intricacies such as binary representations of user-defined types. By using DXL to encode metadata and queries, it is unnecessary for the source database system to be accessible for replay.

As shown in FIG. 1 , an incoming SQL query 130 to parser 124 of the database system is communicated to the query optimizer 100 along with referenced metadata. In the query optimizer, the query is converted to a DXL query at 134 and communicated to the optimizer engine 114 , and the accompanying metadata is transmitted to the metadata cache 116 . The metadata cache may request, at 136 , DXL metadata 138 from the database catalog 128 . The requested metadata may be supplied by the catalog 126 to the parser 124 and also transmitted at 140 to the metadata DXL converter 138 . Metadata converter 138 transforms the metadata to DXL format and supplies it to the metadata cache 116 . The optimizer engine 114 uses the DXL metadata and query from converter 134 to generate a query plan which is transmitted at 144 to a DXL plan converter 146 that forwards the plan to an executor 128 for execution. The database system outputs the results of the query at 150 .

A variety of error situations can terminate the optimization process of a query statement. These include internal errors that are actively detected by the optimizer, as well as signals or other exceptional situations. In the optimizer, errors are converted into an exception that, unless caught, triggers a data capture handler of the invention to produce a repro file. When an error occurs, the data capture handler harvests relevant information and input data for an ongoing optimization to produce the repro file. This data may include the actual query in algebraic form, the metadata referenced in the query including table and index information, functions, etc., table statistics used by a cardinality estimation module of the optimizer, configuration information, process and thread specific information such as stack traces or thread-control blocks, and environmental information such as control settings and optimizer hints.

For a repro to be useful, it is important that the required information be collected in a way that is consistent with the transaction in which the original optimization occurred. Furthermore, collection must occur with the system in a degraded state where the integrity of data structures may no longer be guaranteed and where additional resources such as memory allocations are not possible as they may trigger further internal errors. The data capture process of the invention satisfies these needs, as will be explained.

Since the query optimizer runs in a separate process form the database, all input and output data must be serialized into DXL form when transferred. This requires that input data to the optimizer be serialized before it is used. The invention advantageously stores and uses this pre-serialized data, instead of having to collect the required information at the time of an exception, to harvest a snapshot that is consistent and represents literally the exact input to the optimizer at the time of the exception. Any component of the optimizer can participate in the data capture process by registering a capture object that will be called as part of handling exceptions with the data capture error handler. Objects can register with the handler by implementing a simple API which is invoked during a crash and serializes the object state to the handler output. In the optimizer, objects such as the input query, metadata, configuration settings, etc. will be registered with the handler. Objects may register with handler at any point during execution. If a crash occurs for object before it is registered, that object did not contribute to crash and may safely be excluded from the repro file. At the time of the crash, the handler requests each registered object to serialize itself into pre-allocated memory space. Capture objects retrieve relevant data, calculate the data size, and copy the data into a designated memory-mapped output buffer. The capture mechanism also causes the OS layer to capture objects to retrieve stack traces and other process details. Unlike pre-serialized DXL documents, the capture objects for system information have to serialize their contents on-the-fly. As will be described, the error handling mechanism of the invention provides support for this by automatic storing required information on a per-thread error context object within a pre-allocated memory scratch space for signal handling and stack trace analysis. Once triggered, the capture objects simply copy the information from the error context to the pre-allocated space in target memory.

The invention interacts with the capture handler on a per-query basis for creating the executable repro file (referred to also as a “dumpfile”), properly positioning DXL entries for each thread inside allocated space within the file with no overlap between entries, and adding an appropriate DXL header and footer to the file. In addition to providing the memory medium where artifacts are recorded in the form of a memory-mapped file or a pre-allocated memory buffer, the handler controls the position where each artifact is recorded within the buffer. Since different threads may raise exceptions concurrently, the handler preferably uses atomic operations to synchronize the artifacts extracted by the different capture objects.

FIG. 2 illustrates the details of an embodiment of a workflow process 200 of a data capture mechanism in accordance with the invention. In a first step 202 , a main processing thread 204 initializes the capture handler, and registers the stack trace, the input query, the metadata accessor, and configuration parameters as artifact objects. At 206 , the optimizer spawns one or more processing sub-threads T 1 , T 2 , T 3 and T 4 as part of the normal optimization process. At 208 , each spawned sub-thread registers its stack trace with the handler. If during optimization, one of the sub-threads, such as T 1 , triggers an exception, as at 210 , for example, due to a corrupted internal data structure, the error handler calls all registered captured objects of the sub-thread which stores the stack trace T 1 at 216 in a buffer memory 220 . When it crashes, the sub-thread T 1 exits and may notify the main thread 204 at 222 , which automatically aborts the remaining sub-threads at 224 and the optimization process at 226 . Sub-threads T 2 -T 4 serialize themselves as entries into corresponding locations in buffer memory 220 , as shown. Once all of the sub-threads T 1 -T 4 have terminated, the main thread walks the chain of its registered capture objects that collected all pre-serialized information, and copies it into a corresponding location 232 in buffer memory 220 . The entries in memory 220 are preferably in the form of a DXL document that comprises the executable repro file (dumpfile), to which the handler may add appropriate header, footer and system description information.

The Appendix gives an example of a simplified listing of the handler dumpfile that includes the stack trace only for the main thread. As will be appreciated, the actual dumpfile will also include the stack trace information for the other sub-threads.

Since the dumpfile generated by the handler upon a crash is an executable DXL document file, any optimizer instance can load the file to retrieve the input query, the metadata and configuration parameters. These can be used directly to invoke an optimization process that is identical to the one that triggered the crash so that the crash scenario can be replayed. This replay process 300 is illustrated in FIG. 3 .

As shown, the optimizer 100 may load the input query 312 from the dumpfile 310 , create a file-based provider for the metadata 314 , and set the same configuration parameters into the optimizer. The optimizer may then spawn the same optimization threads from the DXL stack traces 320 recorded in the dumpfile. Since the input query and all other parameters and conditions are the same, executing the repro on the optimizer allows the optimizer operation to be replayed just as it did at the time of the crash, so that the error may be reproduced, analyzed and corrected.

Moreover, the repro file is minimal, portable and executable. Since it contains a synopsis of the original system, the repro file may be transmitted to another site and run on any sized system to reproduce the error and the precise conditions under which it occurred. This enables developers to replay, investigate and troubleshoot errors on regular development laptops without the necessity of access to a large database system. Additionally, the invention may be integrated with an automatic problem reporting software to transmit the generated executable repro file from the customer site to a support center.

An embodiment of the invention affords a computer storage product comprising computer readable physical (non-transitory) storage medium storing the workflow framework as executable instructions for controlling the operations of a computer to perform the processing operations described herein. The computer readable medium may be any standard well-known storage media, including, but not limited to magnetic media, optical media, magneto-optical media, and hardware devices configured to store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices, and semiconductor memory such as ROM and RAM devices.

While the foregoing has been with reference to preferred embodiments of the invention, it will be appreciated by those skilled in the art that changes to these embodiments may be made without departing from the principles and spirit the invention, the scope of which is defined in the appended claims.

<?xml version=“1.0”  encoding = “UTF-8”?>

<dxl:DXLMsg xmlns:dxl=“http://greenplum.com/dxl/v1”>

<dxl:Thread  Id=“0” >

<dxl:Stacktrace>

1   0x000e8106df  gpos : : CException::Raise

2   0x000137d853  COptTasks : : PvOptimizeTask

3   0x000e81cb1c  gpos : : CTask : : Execute

4   0x000e8180f4  gpos : : CWorker : : Execute

5   0x000e81e811  gpos : : CAutoTaskProxy : : Execute

</dxl:Stacktrace>

<dxl:TraceFlags Value=“gp_optimizer_hashjoin” />

<dxl:Metadata System Ids=“0.GPDB”>

<dxl:Type Mdid=“0.9.1.0” Name=“int4”

IsRedistributable=“true” Length=“4” />

<dxl:RelStats Mdid=“2.688.1.1” Name=“r”Rows=“10”/>

<dxl:Relation Mdid=“0.688.1.1” Name=“r”

Distribution Policy=“Hash” DistributionColumns=“0”>

<dxl:Columns>

<dxl:Column Name=“a” Attno=“1” Mdid=“0.9.1.0” />

</dxl:Columns>

</ d x l : R e l a t i o n>

</dxl:Metadata>

<dxl:Query>

<dxl:OutputColumns>

<dxl:Ident Colld=“1” Name=“a” Mdid=“0.9.1.0”/>

</dxl:OutputColumns>

<dxl:LogicalGet>

<dxl:TableDescriptor Mdid=“0.688.1.1” Name=“r”>

<dxl:Columns>

<dxl:Column Attno=“1” Name=“a” Mdid=“0.9.1.0”/>

</dxl:Columns>

</dxl:TableDescriptor>

</dxl:LogicalGet>

</dxl:Query>

</dxl:Thread>

</dxl:DXLMsg>