Data transmission between memory and on chip memory of inference engine for machine learning via a single data gathering instruction

This application claims the benefit of U.S. Provisional Patent Application No. 62/675,076, filed May 22, 2018, which is incorporated herein in its entirety by reference.

This application is a continuation-in-part of U.S. patent application Ser. No. 16/226,508, filed Dec. 19, 2018, and entitled “Single Instruction Set Architecture (ISA) Format for Multiple ISAs in Machine Learning Inference Engine,” which is incorporated herein in its entirety by reference.

Applied Machine Learning (ML) is a booming field that utilizes a cascade of layers of nonlinear processing units and algorithms for feature extraction and transformation with a wide variety of usages and applications. ML typically involves two phases, training, which uses a rich set of training data to train a plurality of machine learning models, and inference, which applies the trained machine learning models to actual applications. Each of the two phases poses a distinct set of requirements for its underlying infrastructures. Various infrastructures may be used, e.g., graphics processing unit (GPU), a central processing unit (CPU), a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), etc. Specifically, the training phase focuses on, as a non-limiting example, GPU or ASIC infrastructures that scale with the trained models and retraining frequency, wherein the key objective of the training phase is to achieve high performance and reduce training time. The inference phase, on the other hand, focuses on infrastructures that scale with the applications, user, and data, and the key objective of the inference phase is to achieve energy (e.g., performance per watt) and capital (e.g., return on investment) efficiency.

Inference phase of ML is usually very computationally and data intensive. Unfortunately, as the input data and model sizes grow, data movement becomes a bottleneck and data processing increases because in order to perform simple processing, three operations or instructions are performed for each data, e.g., load, processing, and store. As the amount of data grows, performing these three operations or instructions becomes burdensome. Moreover, the current computing architecture is not scalable and are not well suited for ML and its applications, since a lot of time goes in loading and storing the data in comparison to processing the data.

The foregoing examples of the related art and limitations related therewith are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent upon a reading of the specification and a study of the drawings.

Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is noted that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.

FIG. 1 depicts an example of diagram of a hardware-based programmable architecture configured to support inference acceleration for machine learning according to one aspect of the present embodiments.

FIG. 2 depicts an example illustrating delivery of data and instructions from the data streaming engine and the instruction streaming engine to the inference engine, respectively, according to one aspect of the present embodiments.

FIG. 3 depicts an example of pseudocode that implements the data gathering instruction based on the input parameters according to one aspect of the present embodiments.

The following disclosure provides many different embodiments, or examples, for implementing different features of the subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.

Before various embodiments are described in greater detail, it should be understood that the embodiments are not limiting, as elements in such embodiments may vary. It should likewise be understood that a particular embodiment described and/or illustrated herein has elements which may be readily separated from the particular embodiment and optionally combined with any of several other embodiments or substituted for elements in any of several other embodiments described herein. It should also be understood that the terminology used herein is for the purpose of describing the certain concepts, and the terminology is not intended to be limiting. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood in the art to which the embodiments pertain.

FIG. 1 depicts an example of a diagram of a hardware-based programmable system/architecture 100 configured to support inference acceleration for machine learning. Although the diagrams depict components as functionally separate, such depiction is merely for illustrative purposes. It will be apparent that the components portrayed in this figure can be arbitrarily combined or divided into separate software, firmware and/or hardware components. Furthermore, it will also be apparent that such components, regardless of how they are combined or divided, can execute on the same host or multiple hosts, and wherein the multiple hosts can be connected by one or more networks.

Each of the engines/programmable processors in the architecture 100 is a dedicated hardware block/component including one or more microprocessors and on-chip memory units storing software instructions programmed by a user for various machine learning operations. When the software instructions are executed by the microprocessors, each of the hardware components becomes a special purposed hardware component for practicing certain machine learning functions as discussed in detail below. In some embodiments, the architecture 100 is on a single chip, e.g., a system-on-chip (SOC).

In the example of FIG. 1 , the architecture 100 may include a host 110 coupled to a memory (e.g., Double Data Rate or DDR) 120 and a core engine 130 via a PCIe controller and/or a direct memory access (DMA) module. The host 110 is a processing unit configured to receive or generate data to be analyzed and/or inferred by architecture 100 via machine learning. The DDR memory 120 includes a plurality of memory blocks each accessible via a memory address or pointer. The DDR memory 120 is coupled to a data streaming engine 140 configured to transfer/stream data between the DDR memory 120 and on-chip memory (OCM) 210 of an inference engine 160 discussed below via DDR-to-OCM DMA or DoD. The core 130 is a processing engine configured to receive and interpret a plurality of ML commands from the host 110 into instructions for a ML operation. The core 130 is also configured to process a plurality of performance non-critical operations, e.g., data/instruction preparatory work, data collection, data mapping, etc. The core 130 is coupled to an instruction-streaming engine 150 , which accepts instructions destined for the inference engine 160 from the core 130 and distributes the instructions to the appropriate units within the inference engine 160 . The inference engine 160 is configured to perform dense and sparse operations on received stream of data, e.g., to identify a subject in an image, by using the training data and executing the programming instructions received from the instruction-streaming engine 150 .

In some embodiments, the inference engine 160 includes a two-dimensional computing array of processing tiles, e.g., tiles 0, . . . , 63, arranged in, e.g., 8 rows by 8 columns. Each processing tile (e.g., tile 0) includes at least one on-chip memory (OCM) e.g., 210 , one POD engine (or POD), e.g., 220 , and one processing engine/element (PE), e.g., 230 . Here, the OCMs in the processing tiles are configured to receive data from the data streaming engine 140 in a streaming fashion. The OCMs 210 enable efficient local access to data per processing tile. The PODs 220 are configured to perform dense or regular computations on the received data in the OCMs 210 , e.g., matrix operations such as multiplication, matrix manipulation, tan h, sigmoid, etc., and the PEs are configured to perform sparse/irregular computations and/or complex data shape transformations of the received data in the OCMs 210 , e.g., memory transpose, addition operation, operations on irregular data structures (such as trees, graphs, and priority queues), respectively. Both the PODs 220 and the PEs 230 can be programmed according to the programming instructions received from the instruction-streaming engine 150 as discussed in detail below. Accordingly, the data is received and processed by each processing tile as an input data stream from the DDR memory 120 and the result is output by each processing tile as a stream of data to the DDR memory 120 .

In some embodiments, a plurality of (e.g., four) processing tiles in the inference engine 160 together form a processing block or quad 250 , e.g., processing tiles 0-3 form processing block 250 , wherein the processing tiles within each processing block 250 are coupled to one another via a routing element 240 . In some embodiments, all the routing elements are connected together as a mesh 260 of interconnect to connect the processing blocks in the same row or column as a two-dimensional array. It is appreciated that the number and/or types of components within each processing tile, the formation of the processing blocks, the number of processing tiles in each processing block, and the number of processing blocks in each row and column of the inference engine 160 as shown in FIG. 1 are exemplary and should not be construed as limiting the scope of the embodiments. In some embodiments, the same number of PE and POD may be used for each tile, and the same number of blocks may be used in each row and column in order to provide flexibility and scalability.

FIG. 2 depicts an example illustrating delivery of data (represented by solid line) and instructions (represented by dotted lines) from the data streaming engine 140 and the instruction streaming engine 150 to the inference engine 160 , respectively. In some embodiments, the instructions are sent from the instruction streaming engine 150 with a tile mask, indicating which processing tiles the instructions should be delivered to. Each router 240 is configured to check the tile mask to determine whether each instruction should be sent to one or more connected processing tiles or to a downstream router so that the instructions are always delivered to all the intended processing tiles in the right order. In some embodiments, each router 240 is also configured to check if there is enough buffer space in the processing tiles and/or the downstream router. Each router can start sending the next instructions only after the current instruction is sent to all intended processing tiles connected to the router and/or its downstream router.

In the example of FIGS. 1 and 2 , the core 130 is configured to program various components, e.g., PODs 220 and PEs 230 of the inference engine 160 via a set of programming instructions translated according to an instruction set architecture (ISA) designed for efficient data processing in a data-path. In some embodiments, the ISA is a predominantly asynchronous instruction set, wherein each instruction in the ISA programs a state-machine, which then runs asynchronously with respect to other state machines. It is appreciated that a series of instructions do not necessarily imply sequential execution. In some embodiments, the ISA provides separate synchronizing instructions to ensure order between instructions where needed.

In some embodiments, the ISA enables programming of each component, e.g., PODs 220 or PEs 230 , of the inference engine 160 in three steps: (i) programming one or more input data streams to the component to fetch input data into queues or registers associated with a computing block/operator of the component; (ii) programming the operator to perform the operations to be performed on the input data streams; and (iii) programming one or more output data streams to write the output of the operations into the OCM 210 of the inference engine 160 .

In some embodiments, the ISA includes at least three classes of programming instructions: (i) programming instructions executed by the PODs 220 , (ii) programming instructions executed by the PEs 230 , and (iii) common programming instructions executed before the tasks are dispatched to either the PODs 220 or the PEs 230 . Note that each of the programming instructions can be executed by one or more or all of the PODs 220 and/or PEs 230 at the same time.

In some embodiments, the ISA includes a data gathering instruction, e.g., DMA_DDR_Gather_to_OCM, wherein the data gathering instruction is configured to perform a data gathering operation to gather data from the DDR memory 120 and write the gathered data into the OCM 210 of each of one or more processing tiles via DoD by the data streaming engine 140 . In some embodiments, the gathering instruction has the following format: DMA_DDR_Gather_to_OCM (ddr_pointer_array_address, OCM_addr, num_pointers, linelength). Here, the ddr_pointer_array_address is the address in the DDR memory 120 where pointers or addresses to the data lines to be gathered are stored. The number of pointers stored is given by num_pointers. The gathering instruction reads num_pointers number of pointers from ddr_pointer_array_address and then reads data lines pointed to by those pointers or addresses. The length of the data lines is given by linelength. In some embodiments, the one or more processing tiles which OCMs 210 have the gathered data written into can be specified by the core 130 via a separate programming instruction, e.g., DMATaskBcst, which informs the data streaming engine 140 of the OCMs of the processing tiles to be written into. In some embodiments, the data gathering instruction is configured to gather data from a plurality of memory blocks in the DDR memory 120 via a single instruction at the same time. For a non-limiting example, the plurality of memory blocks may be maintaining a large table across multiple memory blocks in the DDR memory 120 . In some embodiments, the addresses/pointers to the plurality of memory blocks are maintained by the core 130 in an array of addresses, wherein each address can be, e.g., a 64-bit address. The array of addresses helps to create a two-level indirect reference to the entire list of the plurality of memory blocks from which the data is to be gathered in the data gathering instruction irrespective of the number of the memory blocks from which the data is to be gathered. In some embodiments, the array of pointers to the memory blocks from which data is to be gathered is contained in the DDR memory 120 . In some embodiments, the address of the array of pointers to the data gathering memory blocks in the DDM memory 120 is provided as an input parameter to the single data gathering instruction by the core 130 .

In some embodiments, in addition to the address of the array of pointers, the data gathering instruction may further take one or more of the following as its input parameters: number of pointers to the memory blocks from which data is to be gathered, e.g., numptrs, length of the line in each memory block to be loaded by each pointer to the memory block, e.g., linelen, which can be up to 1K bytes, address of the OCM 210 in each processing tile, e.g., ocm_addr, which the data gathered from the DDR memory 120 is to be written into sequentially, and a Boolean indicator, e.g., signed, which signifies if the data being transferred from the DDR memory 120 to the OCMs 210 of the processing tiles is either signed or unsigned. FIG. 3 depicts an example of pseudocode that implements the data gathering instruction based on the input parameters listed above, wherein the data is written into the OCMs of processing tiles as specified by the DMATaskBcst instruction.

The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to the practitioner skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical application, thereby enabling others skilled in the relevant art to understand the claimed subject matter, the various embodiments and the various modifications that are suited to the particular use contemplated.