Processor circuit, information processing apparatus, and operation method of processor circuit

This application is a continuation application of International Application PCT/JP2018/000987 filed on Jan. 16, 2018 and designated the U.S., the entire contents of which are incorporated herein by reference. The International Application PCT/JP2018/000987 is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2017-013397, filed on Jan. 27, 2017, the entire contents of which are incorporated herein by reference.

The embodiments discussed herein are related to a processor circuit, an information processing apparatus, and an operation method of the processor circuit.

Deep learning (hereinafter referred to as DL) is performed by a processor circuit in an information processing apparatus performing arithmetic processing. Algorithms that use a neural network (hereinafter referred to as a deep neural network (DNN)) having multiple layers are collectively referred to as DL. Among various types of DNNs, convolution neural networks (CNNs) are employed in many cases. The CNNs are widely used as DNNs for, for example, determining features in images.

With use of a CNN for determining features in images, a convolution operation is performed for an input image by using filters, and features in the image (for example, edge features) are detected. The convolution operation of the CNN is performed by, for example, a processor circuit. Japanese Laid-open Patent Publication No. 2013-156783 discloses inter-processor communications in a multi-core processor system.

Examples of the related art include International Publication Pamphlet Nos. WO2016/075813 and WO2012/001835, and Japanese Laid-open Patent Publication No. 08-147178.

According to an aspect of the embodiments, a processor circuit includes: (a) multiple processor cores; (b) multiple individual memories, each of the multiple individual memories being associated with one of the multiple processor cores and being configured to be accessed from the associated one of the multiple processor cores; (c) multiple shared memories, each of the multiple shared memories being associated with a first processor core, the first processor core being any one of the multiple processor cores, each of the multiple shared memories being configured to be accessed from either the first processor core or a first adjacent processor core, the first adjacent processor core being one of the multiple processor cores and being adjacent to the first processor core in a first direction among the multiple processor cores; (d) multiple memory control circuits, each of the multiple memory control circuits being provided between the first processor core and an associated individual memory of the multiple individual memories and being configured to output a read request from the first processor core to the associated individual memory belonging to the first processor core; (e) multiple selectors, each of the multiple selectors being associated with one of the multiple shared memories and being configured to select a read request from one of the first processor core, to which the associated one of the multiple shared memories belong, and the first adjacent processor core, output the selected read request to the associated one of the multiple shared memories, select a transfer request from one of a specific memory control circuit of the multiple memory control circuits and another memory control circuit of the multiple memory circuits belonging to a second adjacent processor core adjacent to the specific memory control circuit in a second direction, and output the selected transfer request to the associated one of the multiple shared memories; and (f) a control core configured to control the multiple processor cores; (g) wherein, in a case where the control core sets, in each of the multiple memory control circuits, a transfer source address of one of the multiple individual memories and the multiple shared memories that store transfer data to be transferred among the multiple processor cores and a transfer destination address of one of the multiple shared memories to which the transfer data is to be transferred and also sets transfer selection information in each of the multiple selectors, (h) with respect to each of the multiple memory control circuits, when an address of the read request from the first processor, to which a specific memory control circuit belongs, is identical to the transfer source address, the specific memory control circuit controls the transfer data in accordance with the read request to be transferred to the transfer destination address via a specific selector of the multiple selectors in which the transfer selection information is set, (i) wherein, in a case where the control core sets read selection information in each of the multiple selectors, with respect to each of the multiple shared memories, read data is read by one of the first processor core, to which an associated shared memory of the multiple shared memories belongs, and the first adjacent processor core from the associated shared memory via a specific selector of the multiple selectors in which the read selection information is set.

The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.

FIG. 1 illustrates a configuration of an information processing apparatus (a deep learning server) according to an embodiment;

FIG. 2 is a flowchart illustrating an outline of processing of a DL operation program;

FIG. 3 illustrates a configuration of a graphic processing unit (GPU) and a configuration of a processor core in the GPU;

FIG. 4 illustrates an example of a CNN;

FIG. 5 is a diagram for describing convolution operation;

FIG. 6 illustrates an example of convolution operation of a DL apparatus of the embodiment;

FIG. 7 illustrates a configuration of a graphic processor circuit of the embodiment;

FIG. 8 is a sequence chart illustrating an outline of reading operation and convolution operation for images that are performed by four processor cores illustrated in FIG. 7 ;

FIG. 9 illustrates a configuration among the processor cores, and individual memories and shared memories in the graphic processor circuit;

FIG. 10 illustrates transfer operation for read data to a shared memory performed by a memory control circuit;

FIG. 11 is an diagram for describing a transfer control circuit and a configuration register;

FIG. 12 is a flowchart illustrating operation of the transfer control circuit;

FIG. 13 is a sequence diagram illustrating operations in times in a sequence illustrated in FIG. 8 ;

FIG. 14 illustrates data reading operation and read data transfer operation in relation to two memory banks of the shared memories and the individual memories that belong to the four respective processor cores;

FIG. 15 illustrates data the reading operation and the read data transfer operation in relation to the two memory banks of the shared memories and the individual memories that belong to the four respective processor cores;

FIG. 16 illustrates the data reading operation and the read data transfer operation in relation to the two memory banks of the shared memories and the individual memories that belong to the four respective processor cores; and

FIG. 17 illustrates operation of a processor circuit according to a second embodiment.

The aforementioned convolution operation is such that, while the location of a coefficient filter in an image is moved in a raster scan direction of the image, sum-of-product operations are repeated by using values of pixel data of an adjacency matrix centered at a target pixel of the image and coefficients (weights) of the coefficient filter. The convolution operation involves operations performed by using multiple images and multiple coefficient filters, and thus, performing the convolution operations in parallel by using multiple sum-of-product operation units contributes to the reduction in operation time. In such a case, multiple cores having respective operation units individually read multiple images from memory and repeat sum-of-product operations by using respective coefficient filters.

The processing performed by multiple cores to read multiple identical images from memory is a factor in the reduction in processing efficiency of the convolution operation. Worse still, since the operations in a DNN involve a multitude of convolution operations, the degree of the reduction in processing efficiency due to reading processing increases in proportion.

One object of an embodiment is to provide a processor circuit, an information processing apparatus, and an operation method of the processor circuit that contribute efficient image reading processing performed by cores in the processor circuit.

FIG. 1 illustrates a configuration of an information processing apparatus (a deep learning server) according to the present embodiment. A server 1 is capable of communicating with a sensing device group 30 and a terminal device 32 via a network. The sensing device group 30 , for example, captures an image by using an imaging element, generates image data, and transmits the image data to the server 1 . The terminal device 32 receives a determination result of features in the image from the server 1 and outputs the determination result.

The server 1 includes a central processing unit (CPU) 10 , which is a general processor circuit, and a GPU 11 , which is a graphics processor circuit. The server 1 also includes a main memory 12 such as a dynamic random-access memory (DRAM), a network interface 14 such as a network interface card (NIC), and a large-capacity auxiliary storage device 20 such as a hard disk or a solid state drive (SSD), and a bus BUS that connects these devices to one another.

The auxiliary storage device 20 stores, for example, a DL operation program 22 and a DL parameter 24 . The auxiliary storage device 20 stores, in addition to the aforementioned program and the parameters, an operating system (OS) and various kinds of middleware programs, which are not illustrated in the diagram. The CPU 10 and the GPU 11 load the program and the parameters into the main memory 12 and execute the program in accordance with the parameters.

FIG. 2 is a flowchart illustrating an outline of processing of the DL operation program 22 . The DL operation program 22 performs, for example, a DNN operation. The CPU 10 and the GPU 11 execute the DL operation program 22 and perform processing in a learning mode and a determination mode. An example of DL is described by using a DNN for determining features in an image.

In the learning mode, the CPU 10 and the GPU 11 read an initial value of an operation parameter (for example, a filter coefficient (weight)) from the main memory 12 and write the initial value of the operation parameter to a high-speed memory SRAM in the GPU 11 (S 10 ). The CPU 10 and the GPU 11 also read from the main memory 12 an image transmitted from the sensing device group 30 and write the image to the high-speed memory SRAM (S 11 ). The CPU 10 and the GPU 11 subsequently perform format conversion of the image and generate an adjacency-matrix image (operation processing data) to be input to an arithmetic logic unit (S 12 ) and perform operation processing in a convolutional layer, a pooling layer, a fully connected layer, and a softmax layer (an output layer) of a DNN (S 13 ). The operation is performed individually for a predetermined number of images. The operation result is expressed, for example, such that a specific image corresponds to either a 0 or a 1.

The CPU 10 and the GPU 11 determine whether the difference between the operation result and training data as correct image data is equal to or less than a threshold (S 14 ), and in a case where the difference is more than the threshold (NO in S 14 ), update the operation parameter by performing a backward operation in the DNN in accordance with the difference (S 15 ). Subsequently, by using the updated operation parameter, steps S 11 to S 13 are repeated. Here, the difference between the operation result and the training data is, for example, the sum of differences between 1000 operation results, which are obtained by performing operations for 1000 images, and 1000 kinds of training data.

If the above-described difference is equal to or less than the threshold (YES in S 14 ), it is determined that the operation parameter is set to an optimum value and the learning mode is ended. By using the optimum value of the operation parameter, the following operation processing in the determination mode is performed.

In the determination mode, the CPU 10 and the GPU 11 read the image targeted for determination from the main memory 12 (S 16 ), perform format conversion of the image and generate an adjacency-matrix image to be input to an arithmetic logic unit (S 17 ), and performs operation processing in the convolutional layer, the pooling layer, the fully connected layer, and the softmax layer of the DNN (S 18 ). The CPU 10 and the GPU 11 repeat the determination processing until the determination processing has been performed for all images targeted for determination (S 19 ). The determination result is transmitted and output to the terminal device 32 .

FIG. 3 illustrates a configuration of the GPU 11 and a configuration of a processor core CORE in the GPU 11 . The GPU 11 is accessible to main memories M_MEM. The GPU 11 includes, for example, eight processor cores CORE, multiple high-speed memories SRAM, which are disposed in association with the respective processor cores CORE, an internal bus I_BUS, and memory controllers MC, which perform access control for main memories M_MEM. The GPU 11 also includes L1 cache memories in the respective processor cores CORE, an L2 cache memory shared among the eight processor cores CORE, and various kinds of peripheral resource circuits, which are all not illustrated in FIG. 3 . The GPU 11 also includes a direct memory access control circuit DMA for controlling, for example, internal data transfer among the high-speed memories SRAM and data transfer among the main memories M_MEM and the high-speed memories SRAM.

The processor core CORE includes, similarly to a general processor core, an instruction fetch circuit FETCH that retrieves an instruction from a memory, a decoder DEC that decodes the retrieved instruction, multiple arithmetic logic units ALU that perform arithmetic operations for the instruction in accordance with the decoding result, a register group REG for the multiple arithmetic logic units ALU, and a memory access control circuit MAC for accessing the high-speed memory SRAM.

The GPU 11 is implemented as, for example, a semiconductor chip and is a DL apparatus of the present embodiment. The GPU 11 reads an image from the main memories M_MEM storing the image transmitted from the sensing device group 30 and writes the image to the high-speed memories SRAM. The image written to the high-speed memories SRAM is input to the arithmetic logic units ALU in the processor cores CORE. The arithmetic logic units ALU perform operation processing in each layer in the DNN and generate an output of the DNN.

FIG. 4 illustrates an example of a CNN. The CNN for performing determination processing of an image has an input layer INPUT_L to which an image IM_D is input as input data, multiple pairs each including a convolutional layer CNV_L and a pooling layer PL_L, a fully connected layer C_L, and an output layer (a softmax layer) OUT_L.

The convolutional layer CNV_L generates an image F_IM_D having a specific feature obtained by filtering the image IM_D by using a coefficient filter FLT. By performing filtering with multiple coefficient filters FLT_ 0 to FLT_ 3 , multiple images F_IM_D having respective features are generated. The pooling layer PL_L, for example, selects a representative value (for example, a maximum value) in each node in the convolutional layer CNV_L. In the output layer OUT_L, for example, a determination result expressed by numbers (any of 0 to 9) in the image is output as described above.

In the convolutional layer CNV_L, sum-of-product operations are performed such that values of pixel data of, for example, a 3×3 adjacency matrix in the image IM_D composed of pixels of an M×N two-dimensional pixel matrix and values of coefficient data of a coefficient filter FLT having the same 3×3 structure as that of the adjacency matrix are respectively multiplied and the multiplication results are added together, and accordingly, a patch of the image F_IM_D of the adjacency matrix centered at a target pixel is generated. This filtering processing is performed for pixels of the entire image IM_D while the coefficient filter FLT is moved in a raster scan direction of the image IM_D. Such processing constitutes the convolution operation.

FIG. 5 is a diagram for describing the convolution operation. FIG. 5 illustrates as an example an input image IN_DATA formed by adding a padding P around a 5×5 image, a coefficient filter FLT 0 having weights W 0 to W 8 in a 3×3 matrix, an output image OUT_DATA obtained by performing the convolution operation. The convolution operation is such that values of multiple pixels in an adjacency matrix centered at a target pixel and the multiple coefficients (weights) W 0 to W 8 of the coefficient filter FLT 0 are respectively multiplied and the multiplication results are added together while the coefficient filter FLT 0 is moved in a raster scan direction of the image.

When a value of a pixel in an adjacency matrix is Xi (where i=0 to 8) and a coefficient of a coefficient filter is Wi (where i=0 to 8), a sum-of-product operation equation is as follows:

Xi =Σ( Xi*Wi )  (1)

where Xi on the right-hand side is a value of a pixel of the input image IN_DATA, Wi is a coefficient, Σ denotes the sum when i=0 to 8, and Xi on the left-hand side is a sum-of-product operation value, that is, a value of a pixel of the output image OUT_DATA.

In a case where the target pixel of the image is X 6 , the value of the pixel X 6 obtained by performing sum-of-product operation in accordance with Equation (1) is as follows:

X 6= X 0* W 0+ X 1* W 1+ X 2* W 2+ X 5* W 3+ X 6* W 4+ X 7* W 5+ X 10* W 6+ X 11* W 7+ X 12* W 8.

Embodiment

FIG. 6 illustrates an example of convolution operation of the DL apparatus of the present embodiment. FIG. 6 illustrates an input layer INPUT_L and multiple convolutional layers CNV_L and omits the pooling layers between the convolutional layers. The input layer INPUT_L may be a convolutional layer CNV_L 0 . Successively, two convolutional layers CNV_L 1 and CNV_L 2 are provided.

Multiple images X 0 to X 3 are input to or generated in the input layer INPUT_L or the convolutional layer CNV_L 0 as the first layer. The images X 0 to X 3 in FIG. 6 each have pixels in an N×M matrix similarly to the image IM_D in FIG. 4 . Accordingly, although pixels X 0 to X 24 of the image IM_D in FIG. 4 and the images X 0 to X 3 in FIG. 6 use identical reference characters, the same reference characters denote a pixel in FIG. 4 and an image in FIG. 6 and the subsequent drawings.

In the convolutional layer CNV_L 1 as the second layer, four kinds of convolution operations are performed as follows:

(1) with respect to the four images X 0 to X 3 in the first layer, the convolution operations are performed by using respective coefficient filters F 00 to F 03 of a first coefficient filter group, images ΣF 00 *X 0 to ΣF 03 *X 3 are accordingly generated, and an image XA 0 as an operation result is generated by adding the images ΣF 00 *X 0 to ΣF 03 *X 3 together;

(2) with respect to the four images X 0 to X 3 in the first layer, the convolution operations are performed by using respective coefficient filters F 10 to F 13 of a second coefficient filter group, images ΣF 10 *X 0 to ΣF 13 *X 3 are accordingly generated, and an image XA 1 as an operation result is generated by adding the images ΣF 10 *X 0 to ΣF 13 *X 3 together;

(3) similarly to the above, with respect to the four images X 0 to X 3 in the first layer, the convolution operations are performed by using respective coefficient filters F 20 to F 23 of a third coefficient filter group, and an image XA 2 as an operation result is generated by adding together images generated by performing the convolution operations; and

(4) similarly to the above, with respect to the four images X 0 to X 3 in the first layer, the convolution operations are performed by using respective coefficient filters F 30 to F 33 of a fourth coefficient filter group, and an image XA 3 as an operation result is generated by adding together images generated by performing the convolution operations.

In the above example of the convolution operations in the convolutional layer, the convolution operations are performed for the four images X 0 to X 3 by using the four respective coefficient filter groups. To increase the efficiency of many convolution operations, it is preferable that the (1) to (4) convolution operations described above be performed in parallel by using four processor cores CORE 0 to CORE 3 . In such a case, the four processor cores CORE 0 to CORE 3 each read the four images X 0 to X 3 from a memory.

GPU Configuration

FIG. 7 illustrates a configuration of a graphic processor circuit GPU of the present embodiment. The graphic processor circuit GPU is a deep learning apparatus for performing deep learning operation.

The graphic processor circuit GPU illustrated in FIG. 7 includes four processor cores CORE 0 to CORE 3 and a control core C_CORE for controlling the processor cores CORE 0 to CORE 3 , and in addition, similarly to the configuration illustrated in FIG. 3 , a direct memory access control circuit DMA, a memory controller MC, an internal high-speed memory (SRAM) I_MEM, and an internal bus I_BUS.

The graphic processor circuit GPU also includes multiple individual memories IMEM 0 to IMEM 3 associated with the respective processor cores CORE 0 to CORE 3 and accessed from the processor cores CORE 0 to CORE 3 to which the multiple individual memories IMEM 0 to IMEM 3 belong individually. The graphic processor circuit GPU also includes multiple shared memories SMEM 0 to SMEM 3 associated with the respective processor cores CORE 0 to CORE 3 . The multiple shared memories SMEM 0 to SMEM 3 are each accessed from a processor core to which the specific shared memory belongs and a first adjacent processor core adjacent in a first direction to the processor core to which the specific shared memory belongs. For example, the shared memory SMEM 0 , which belongs to the processor core CORE 0 , is accessed from the processor core CORE 0 and the processor core CORE 3 as the first adjacent processor core.

The graphic processor circuit GPU also includes multiple memory control circuits MAU 0 to MAU 3 each provided between the respective processor cores CORE 0 to CORE 3 and the respective individual memories IMEM 0 to IMEM 3 . The multiple memory control circuits MAU 0 to MAU 3 each output a read request from a specific processor core to an individual memory belonging to the specific processor core.

The graphic processor circuit GPU also includes multiple selectors SL 0 to SL 3 associated with the respective shared memories SMEM 0 to SMEM 3 . Each of the multiple selectors SL 0 -SL 3 (for example, the selector SL 0 ) selects one of a read request from a processor core (for example, the processor core CORE 0 ) to which a specific shared memory belongs and a read request from the first adjacent processor core (the processor core CORE 3 ) described above and outputs the selected read request to the specific shared memory (the shared memory SMEM 0 ). Each of the multiple selectors SL 0 -SL 3 (for example, the selector SL 0 ) also selects one of a transfer request from a memory control circuit (the memory control circuit MAU 0 ) and a transfer request from another memory control circuit (the memory control circuit MAU 1 ) belonging to a second adjacent processor core (the processor core CORE 1 ) adjacent in a second direction to the processor core (the processor core CORE 0 ) and outputs the selected transfer request to the specific shared memory (the shared memory SMEM 0 ).

The individual memories IMEM 0 to IMEM 3 and the shared memories SMEM 0 to SMEM 3 are all high-speed static random-access memories (SRAM) and accessed from the processor cores CORE 0 to CORE 3 . Furthermore, for example, contiguous address areas in a memory space are allocated to the individual memories IMEM 0 to IMEM 3 and the shared memories SMEM 0 to SMEM 3 as described later. As a result, the processor cores CORE 0 to CORE 3 are each able to perform an access request for any of its individual memory, its shared memory, and another shared memory belonging to its second adjacent processor core by specifying the address of a specific memory to be accessed.

When the graphic processor circuit GPU performs the convolution operation in the convolutional layer CNV_L 1 illustrated in FIG. 6 , the four processor cores CORE 0 to CORE 3 in the graphic processor circuit GPU perform in parallel the convolution operations for the four images X 0 to X 3 by each using four filters that are assigned to the specific processor core of the respective processor cores. In this case, as described above, the four processor cores CORE 0 to CORE 3 all sequentially read the four images X 0 to X 3 and repeat the convolution operations by using their filters. At this time, by transferring the images X 0 to X 3 between adjacent processor cores via a shared memory associated with each processor core, the number of steps performed by the four processor cores CORE 0 to CORE 3 to read the four images X 0 to X 3 is reduced.

Convolution Operation Performed by GPU

By using the convolution operation illustrated in FIG. 6 as an example, there is described an outline of reading operation and convolution operation for the images X 0 to X 3 that are performed by the four processor cores CORE 0 to CORE 3 illustrated in FIG. 7 .

FIG. 8 is a sequence chart illustrating an outline of reading operation and convolution operation for the images X 0 to X 3 that are performed by the four processor cores CORE 0 to CORE 3 illustrated in FIG. 7 . A time axis TIME is indicated in the horizontal direction and the four processor cores CORE 0 to CORE 3 , the individual memories IMEM 0 to IMEM 3 , the shared memories SMEM 0 to SMEM 3 are indicated in the vertical direction. With respect to each of times T 0 to T 4 , details of operation are indicated for each of the processor cores CORE 0 to CORE 3 and stored data is indicated for each of the individual memories IMEM 0 to IMEM 3 , and the shared memory CMEM. The shared memories SMEM 0 to SMEM 3 each have two of memory banks B 0 a , B 0 b to B 3 a , and B 3 b . The memory bank is an individual memory unit for which an access request (a read request or a write request) is input and that is accessed accordingly.

The direct memory access control circuit DMA controls in advance the four images X 0 to X 3 and the four coefficient filter groups in the external main memory M_MEM to be transferred to the internal high-speed memory I_MEM. In the time T 0 , the direct memory access control circuit DMA controls the four images X 0 to X 3 and the four coefficient filter groups of the coefficient filters F 00 to F 03 , F 10 to F 13 , F 20 to F 23 , and F 30 to F 33 in the external main memory M_MEM to be transferred to the four respective individual memories IMEM 0 to IMEM 3 . For example, the image X 0 and the coefficient filters F 00 to F 03 of the first coefficient filter group are transferred to the individual memory IMEM 0 , the image X 1 and the coefficient filters F 10 to F 13 of the second coefficient filter group to the individual memory IMEM 1 , the image X 2 and the coefficient filters F 20 to F 23 of the third coefficient filter group to the individual memory IMEM 2 , the image X 3 and the coefficient filters F 30 to F 33 of the fourth coefficient filter group to the individual memory IMEM 3 . In the following description, coefficients of four coefficient filters in each group are denoted by W 0 to W 3 .

In the time T 1 as a first operation cycle, the processor cores CORE 0 to CORE 3 read the respective images X 0 to X 3 and the respective coefficient filters F 00 to F 03 , F 10 to F 13 , F 20 to F 23 , and F 30 to F 33 from the respective individual memories IMEM 0 to IMEM 3 , which belong respectively to the processor cores CORE 0 to CORE 3 , and perform respective convolution operations ΣW 0 *X 0 , ΣW 1 *X 1 , ΣW 2 *X 2 , and ΣW 3 *X 3 . The processor cores CORE 0 to CORE 3 write the convolution operation results to the respective individual memories IMEM 0 to IMEM 3 . The memory control circuits MAU 0 to MAU 1 control the images X 0 to X 3 , which are read from the individual memories IMEM 0 to IMEM 3 , to be transferred to second memory banks B#b (#=0 to 3) of the respective shared memories SMEM 1 to SMEM 3 , and SMEM 0 belonging to the second adjacent processor cores. As a result, the image X 3 is stored in the shared memory SMEM 0 , the image X 0 is stored in the shared memory SMEM 1 , the image X 1 is stored in the shared memory SMEM 2 , and the image X 2 is stored in the shared memory SMEM 3 .

In the time T 2 as a second operation cycle, the processor cores CORE 0 to CORE 3 read the respective images X 3 , X 0 to X 2 from the second memory banks B#b of the shared memories SMEM 0 to SMEM 3 , which belong to the respective processor cores CORE 0 to CORE 3 , and perform the convolution operations ΣW 3 *X 3 , ΣW 0 *X 0 , ΣW 1 *X 1 , ΣW 2 *X 2 by using the respective coefficient filter groups. The processor cores CORE 0 to CORE 3 add the convolution operation results to the previous convolution operation results obtained in the time T 1 and written to the respective individual memories IMEM 0 to IMEM 3 and write the convolution operation results to the individual memories IMEM 0 to IMEM 3 ; in other words, the processor cores CORE 0 to CORE 3 perform read-modify-write. The memory control circuits MAU 0 to MAU 1 subsequently control the images X 3 , X 0 to X 2 , which are read from the second memory banks B#b of the individual memories IMEM 0 to IMEM 3 , to be transferred to first memory banks B#a of the respective shared memories SMEM 1 to SMEM 3 , and SMEM 0 belonging to the second adjacent processor cores. As a result, the image X 2 is stored in the shared memory SMEM 0 , the image X 3 is stored in the shared memory SMEM 1 , the image X 0 is stored in the shared memory SMEM 2 , and the image X 1 is stored in the shared memory SMEM 3 .

In the times T 3 and T 4 as third and fourth operation cycles, similarly to the second operation cycle, the processor cores CORE 0 to CORE 3 read the images from the shared memories SMEM 0 to SMEM 3 , which belong to the respective processor cores CORE 0 to CORE 3 , perform the convolution operations by using the respective coefficient filter groups, add the operation results to the previous operation results in the preceding operation cycle, and stores the operation results in the individual memories IMEM 0 to IMEM 3 . The memory control circuits MAU 0 to MAU 1 control the images read from the shared memories SMEM 0 to SMEM 3 to be transferred respectively to the shared memories SMEM 1 to SMEM 3 , and SMEM 0 belonging to the second adjacent processor cores.

The first memory banks B#a and the second memory banks B#b in the shared memories SMEM 0 to SMEM 3 are switched between a write destination of transfer data and a read destination of an image in every operation cycle. This change is performed in accordance with transfer selection information or read selection information that is set in the selectors for the shared memories by the control core.

In such a manner, the four images X 0 to X 3 that the four processor cores target for operation are transferred to the adjacent shared memories in each of the operation cycles T 1 to T 4 , thereby circulating among the four shared memory synchronously with the operation cycles. As a result, the four images X 0 to X 3 are transferred only at the first time from the internal high-speed memory I_MEM to the four individual memories IMEM 0 to IMEM 3 under the control of the direct memory access control circuit DMA, and in the following operation cycles, transferred and circulated among the four shared memories. The transfer among the shared memories does not involve the access to the internal bus, resulting in efficient processing and saving power.

Details of GPU Configuration

FIG. 9 illustrates a configuration among the processor cores, and the individual memories and the shared memories in the GPU. While centered around the processor core CORE 0 , FIG. 9 illustrates the first adjacent processor core CORE 3 in the first direction and the second adjacent processor core CORE 1 in the second direction opposite to the first direction. The individual memory IMEM 0 , the shared memory SMEM 0 , the memory control circuit MAU 0 , the selector SL 0 , which all belong to the processor core CORE 0 , are described below.

Between the processor core CORE 0 and the individual memory IMEM 0 belonging to the processor core CORE 0 , the memory control circuit MAU 0 is provided to output an access request from the processor core to the individual memory IMEM 0 . Between the processor core CORE 0 and the shared memory SMEM 0 belonging to the processor core CORE 0 , the selector SL 0 is also provided. The shared memory SMEM 0 includes, for example, the two memory banks B 0 a and B 0 b , which individually perform processing in accordance with the access requests. Thus, it is possible that, while one of the memory banks is a read destination of an image, the other of the memory banks is a write destination of transfer data.

The memory control circuit MAU 0 includes a core-bus arbitration circuit CB_ARB that, when an access request from the processor core CORE 0 and an access request from the internal bus I_BUS conflict with each other, arbitrates between the processor core CORE 0 and the internal bus I_BUS and outputs one access request to the individual memory IMEM 0 . The core-bus arbitration circuit CB_ARB performs arbitration, for example, such that the access request from the internal bus I_BUS takes priority. Otherwise, the arbitration may be performed such that one of access requests takes priority in accordance with a set value.

The processor core CORE 0 performs an access request to the individual memory IMEM 0 belonging to the processor core CORE 0 via the memory control circuit MAU 0 . The processor core CORE 0 also performs an access request to the shared memory SMEM 0 belonging to the processor core CORE 0 via the selector SL 0 and an access request to the shared memory SEME 1 belonging to the second adjacent processor core CORE 1 via the selector SL 1 .

Discrete logical addresses in a memory space are assigned to the shared memories SMEM 0 to SMEM 3 and the individual memories IMEM 0 to IMEM 3 . For example, contiguous addresses are assigned to the two memory banks B 0 a and B 0 b of the shared memory SMEM 0 and the individual memory IMEM 0 that both belong to the processor core CORE 0 . Accordingly, contiguous addresses are assigned to the shared memories and the individual memories that belong to each of the four processor cores. As a result, it is possible for the processor core CORE 0 to select data in the individual memory IMEM 0 and the shared memory SMEM 0 , which belong to the processor core CORE 0 , and the shared memory SMEM 1 , which belongs to the second adjacent processor core CORE 1 , by using unique addresses.

The selector SL 0 is provided for the shared memory SMEM 0 . The selector SL 0 selects one of four inputs in accordance with selection information set by the control core C_CORE.

For example, access selection information (the read selection information) is set in the selector SL 0 by the control core C_CORE. In accordance with this setting, the selector SL 0 selects one of an access request from the processor core CORE 0 and an access request from the first adjacent processor core CORE 3 in the first direction and outputs the selected access request to the shared memory SMEM 0 . The same holds for the other selectors.

Accordingly, in a case where the access selection information for selecting the access request from the processor core CORE 0 is set to the selectors SL 0 and SL 1 , when the processor core CORE 0 outputs a control signal for transmitting an address ADD and a read/write signal R/W as a form of memory access, the control signal is output to the shared memory SMEM 0 , the individual memory IMEM 0 , and the shared memory SMEM 1 adjacent in the second direction. As a result, the processor core CORE 0 is able to access an specific data area in the memory identified by the address ADD. The processor core CORE 0 outputs wright data WDT in a case of a write request, whereas the processor core CORE 0 receives read data RDT in a case of a read request.

For example, transfer selection information is set in the selector SL 0 by the control core C_CORE. In accordance with this setting, the selector SL 0 selects one of a transfer request from the memory control circuit MAU 0 belonging to the processor core CORE 0 and a transfer request from the memory control circuit MAU 3 belonging to the first adjacent processor core CORE 3 and outputs the selected transfer request to the shared memory SMEM 0 . The same holds for the other selectors.

Since the shared memory SMEM 0 includes the two memory banks B 0 a and B 0 b , the selector SL 0 is provided for the both two memory banks and thus performs ¼ selection with respect to each memory bank.

Transfer Operation for Read Data to Shared Memory

FIG. 10 illustrates transfer operation for read data to the shared memory performed by the memory control circuit. The memory control circuit MAU 0 belonging to the processor core CORE 0 includes a transfer control circuit TRN_CN. The transfer control circuit TRN_CN controls read data RDT, which is read from the individual memory IMEM 0 or the shared memory SMEM 0 belonging to the processor core CORE 0 , or the shared memory SMEM 1 belonging to the second adjacent processor core CORE 1 , to be transferred to either the shared memories SMEM 0 or SMEM 1 .

When the processor core CORE 0 performs a read request to the individual memory IMEM 0 to receive the read data RDT_I 0 and perform an predetermined operation, the transfer control circuit TRN_CN in the memory control circuit detects the read request and controls the read data RDT_I 0 to be transferred to either the two shared memories SMEM 0 or SMEM 1 ; in other words, the read data is transferred in either a direction of transfer TRN 0 or a direction of transfer TRN 1 , which are indicated by arrows in FIG. 10 . The control core C_CORE sets in a configuration register CFG of the transfer control circuit TRN_CN a specific address from which read data is read and another specific address of a shared memory to which the read data is transferred. The control core C_CORE sets in the selectors SL 0 and SL 1 the transfer selection information in accordance with transfer destinations.

When the processor core CORE 0 performs a read request to the shared memory SMEM 0 to receive read data RDT_S 0 and perform an predetermined operation, the transfer control circuit TRN_CN detects the read request and controls the read data RDT_S 0 to be transferred to the shared memory SMEM 1 as the other shared memory; in other words, the read data is transferred in a direction of transfer TRN 2 , which is indicated by an arrow in FIG. 10 . This transfer operation is also controlled in accordance with a set value that is set in the configuration register CFG.

When the processor core CORE 0 performs a read request to the shared memory SMEM 1 to receive read data RDT_S 1 and perform an predetermined operation, the transfer control circuit TRN_CN detects the read request and controls the read data RDT_S 1 to be transferred to the shared memory SMEM 0 as the other shared memory; in other words, the read data is transferred in a direction of transfer TRN 3 , which is indicated by an arrow in FIG. 10 . This transfer operation is also controlled in accordance with a set value that is set in the configuration register CFG.

Function and Operation of Transfer Control Circuit of Memory Control Circuit

Next, a function and an operation of transfer control circuit are described.

FIG. 11 is an diagram for describing the transfer control circuit TRN_CN and the configuration register CFG. FIG. 12 is a flowchart illustrating the operation of the transfer control circuit. FIG. 12 also illustrates a table indicating set values that are set in the configuration register CFG. A transfer range T_RNG denotes a transfer source address of transfer data. A transfer flag T_FLG determines whether transfer is desired or not. A transfer offset T_OFST is information on a transfer destination address and a transfer destination address is calculated by adding the transfer offset T_OFST to a read address RADD. Accordingly, by setting the transfer offset T_OFST with respect to the transfer range T_RNG as a certain value, a memory bank of a shared memory as a transfer destination is selected.

As illustrated in FIG. 11 , the control core C_CORE sets set values respectively in the configuration register CFG for the transfer control circuit TRN_CN and configuration registers C of the selectors SL 0 and SL 1 via a control bus C_BUS. In the configuration register CFG, the transfer range T_RNG, the transfer flag T_FLG, and the transfer offset T_OFST are set. For example, the access selection information (the read selection information), which is used for selecting an access request from one of the processor cores on both sides of a specific selector, and the transfer selection information, which is used for selecting a transfer request (a write request) from the transfer control circuit TRN_CN of one of the memory control circuits on both sides of the specific selector, are set in the configuration registers C of the selectors SL 0 and SL 1 . As described above, the shared memories SMEM 0 and SMEM 1 each include at least two memory banks, and it is possible to set selection information individually for the memory banks in each of the selectors SL 0 and SL 1 .

As illustrated in FIGS. 11 and 12 , the transfer control circuit TRN_CN in the memory control circuit MAU observes the read/write signal R/W for transmitting an access request from the processor core. In a case where the transfer control circuit TRN_CN detects a read request performed by an load instruction (YES in S 20 ), the transfer control circuit TRN_CN determines whether the transfer flag T_FLG denotes that transfer is desired (S 21 ) and determines whether the read address RADD is an address within the transfer range T_RNG (S 22 ). If YES in the both determination, the transfer control circuit TRN_CN calculates a transfer destination address TADD by adding the transfer offset T_OFST to the read address RADD and transfers the read data RDT as transfer data TDT to the transfer destination address TADD (S 23 ).

In this manner, as illustrated in FIG. 11 , the transfer control circuit TRN_CN controls the transfer destination address TADD, a write signal WT, and the transfer data TDT to be transferred to the shared memory as the transfer destination via the selector SL and controls the transfer data TDT to be written to the shared memory.

Configuration Set by Control Core and Transfer Operation in Accordance with Configuration

Configurations that are set by the control core and operations of the transfer control circuit in the memory control circuit are collectively described as follows. Also in the following description, the individual memory and the shared memory corresponding to the processor core CORE 0 are denoted by reference characters in parentheses.

Firstly, with respect to each of the transfer control circuits in the multiple memory control circuits:

(1) the control core C_CORE sets the transfer source address T_RNG of one of the individual memory (IMEM 0 ), the shared memory (SMEM 0 ), and the shared memory (SMEM 1 ) belonging to the second adjacent processor core (CORE 1 ) adjacent in the second direction, which stores the read data RDT to be transferred between the processor cores;

(2) the control core C_CORE also sets the transfer destination address information T_OFST of either the shared memory (SMEM 0 ) or the shared memory (SMEM 1 ) belonging to the second adjacent processor core (CORE 1 ); and

(3) in a case where the control core C_CORE sets the transfer selection information in multiple selectors (SL 0 and SL 1 ), when the read addresses RADD of the read requests from the multiple processor cores are identical to the transfer source addresses T_RNG, the transfer control circuits of the multiple memory control circuits control the read data RDT, which is read from any one of the individual memory (IMEM 0 ), the shared memory (SMEM 0 ), and the shared memory (SMEM 1 ) belonging to the second adjacent processor core (CORE 1 ), to be transferred as the transfer data TDT to one of the shared memory (SMEM 0 ) and the shared memory (SMEM 1 ) belonging to the second adjacent processor core (CORE 1 ) that corresponds to the transfer destination address information T_OFST.

Secondly, in a case where the control core C_CORE sets the read selection information in the multiple selectors (SL 0 and SL 1 ), the read data in the multiple shared memories (SMEM 0 and SMEM 1 ) are read via the selectors (SL 0 and SL 1 ), in which the read selection information is set, by either the processor core (CORE 0 ) to which the shared memory belongs or the first adjacent processor core (CORE 3 ).

Overall Operations of Control Core, Processor Core, Memory Control Circuit, and Selector

Next, in the present embodiment, operations of the control core C_CORE, operations of the four processor cores CORE 0 to CORE 3 , operations of the four memory control circuit MAU 0 to MAU 3 (the transfer control circuits TNS_CN) and the selectors SL 0 to SL 3 are collectively described.

FIG. 13 is a sequence diagram illustrating operations in the times T 0 , T 1 , and T 2 in the sequence illustrated in FIG. 8 . FIGS. 14, 15, and 16 illustrates data reading operation and transfer operation in relation to the two memory banks B#a and B#b of the shared memories SMEM# and the individual memories IMEM# that belong to the four respective processor cores. FIG. 14 corresponds to the time T 1 , FIG. 15 corresponds to the time T 2 , and FIG. 16 corresponds to the time T 3 .

Time T 0

In the time T 0 , the control core C_CORE instructs the four processor cores CORE 0 to CORE 3 to load from the internal memory I_MEM the respective images X 0 to X 3 and four respective filter groups F 0 to F 3 into the individual memories IMEM 0 to IMEM 3 belonging to the four respective processor cores CORE 0 to CORE 3 (S 30 ). In response to the instruction, the processor cores CORE 0 to CORE 3 loads the respective images X 0 to X 3 and the four respective filter groups F 0 to F 3 into the individual memories (S 31 ).

Time T 1

Next, in the time T 1 , the control core C_CORE sets the configuration for data transfer in the memory control circuit MAU 0 to MAU 3 and the selector SL 0 to SL 3 that belong to the four respective processor cores (S 32 ). In response to this, in each of the memory control circuits, the transfer range (the transfer source address), the transfer flag, the transfer offset (the transfer destination address information), which are described above, are set in the configuration register CFG for the transfer control circuit (S 33 ). In addition, the transfer selection information is set in the selectors. In the time T 1 , the transfer range is set as an address in the individual memory and the transfer selection information is set as the memory control circuit (the transfer control circuit) side that belongs to the second adjacent processor core adjacent to the processor core to which the selector SL# belongs.

Subsequently, the control core C_CORE instructs the four processor cores CORE 0 to CORE 3 to start the convolution operations (S 34 ). In response to the start instruction, the four processor cores load (read) the respective images X 0 to X 3 and the respective filter groups F 0 to F 3 from the individual memories belonging to the respective four processor cores. At the same time, the transfer control circuits of the memory control circuits each detect an read operation, detect that the transfer range in the configuration register CFG is identical to the read address and that the transfer flag denotes that transfer is desired, calculate the transfer destination address TADD, and control each of the images X 0 to X 3 as read data to be output together with the write signal WT to the shared memory (the memory bank B#b) that belongs to the second adjacent processor core and that is at the transfer destination address TADD. In this manner, the read data is transferred to the shared memory (S 36 ).

Subsequently, the processor cores perform the convolution operations for the respective read images X 0 to X 3 and the respective coefficients W 0 to W 1 of the read filter groups F 0 to F 3 (S 37 ) and stores the operation results in the individual memories belonging to the respective processor cores (S 38 ). The processor cores then each transmit an end signal END to the control core C_CORE. The end signal may be transmitted to the control core at both the timing of ending transfer and the timing of ending the operation. When the operation processing times of the respective processor cores differ from one another, the processor cores do not necessarily perform the operation processing synchronously.

FIG. 14 illustrates, with respect to each of the processor cores, addresses of the two memory banks B#a and B#b of the shared memory SMEM# belonging to a specific processor core and an address of the individual memory IMEM# and also illustrates the read operation (a solid-line arrow) of the image X and the filter coefficient W in the time T 1 and the transfer operation (a dashed-line arrow) of the image X to the memory bank B#b of the adjacent shared memory SMEM#. The start address of the memory bank B 0 a of the shared memory SMEM 0 belonging to the processor core CORE 0 is 0x2_0000, the start address of the memory bank B 0 b is 0x2_1000, the start address of the individual memory IMEM 0 is 0x2_2000, and the end address of the individual memory IMEM 0 is 0x2_6000. The addresses relating to the processor cores CORE 1 to CORE 3 are similarly expressed.

The processor cores each read the image X from the own individual memory and the corresponding transfer control circuit of the memory control circuit simultaneously controls the read image X to be transferred to the memory bank B#b of the shared memory of the second adjacent processor core.

Time T 2

Next, in the time T 2 , the control core C_CORE sets the configuration similar to that of the time T 1 in the configuration registers CFG of the memory control circuits (S 42 and S 43 ). However, since the transfer destination is, unlike the configuration in the time T 1 , changed to the memory bank B#a of the shared memory, the transfer offset as the transfer destination address information is a value corresponding to the memory bank B#a. In addition, the control core sets the read selection information and the transfer selection information in the selectors SL# (S 42 and S 43 ). In this manner, the processor cores are each able to read the subsequent image from the shared memory belonging to the specific processor core and the transfer control circuit is able to control the image to be transferred to the shared memory belonging to the second adjacent processor core.

The control core performs an instruction for resuming the operation (S 44 ). The processor cores each load the image and the filter coefficient therein (S 45 ), perform the convolution operation (S 47 ), add the operation result in the time T 2 to the previous operation result in the time T 1 that is read from the individual memory, and write back the operation result to the individual memory (the read-modify-write operation) (S 48 ). The transfer control circuits of the memory control circuits each control the image that the processor core is to load therein to be transferred to the shared memory (S 48 ). After completing the convolution operation and writing back the operation result in the individual memory, the processor cores each transmit an end notification END to the control core.

As illustrated in FIG. 15 , the processor cores each read the image X from the memory bank B#b of the own shared memory and the corresponding transfer control circuit of the memory control circuit simultaneously controls the read image X to be transferred to the memory bank B#a of the shared memory of the second adjacent processor core.

Times T 3 and T 4

The operations in the times T 3 and T 4 are the same as the operations described above. FIG. 16 illustrates the operation in the time T 3 . The processor cores each read the image X from the memory bank B#a of the own shared memory and the corresponding transfer control circuit of the memory control circuit simultaneously controls the image X to be transferred to the memory bank B#b of the shared memory of the second adjacent processor core. The operation in the time T 4 is the same as in FIG. 15 .

As illustrated in FIGS. 14, 15, and 16 , the four images X 0 to X 3 are transferred among the shared memories belonging to the four respective processor cores in each of the operation cycles T 1 to T 4 and circulated among the shared memories.

Second Embodiment

FIG. 17 illustrates an operation of a processor circuit according to a second embodiment. In the operation illustrated in FIG. 17 , the processor core CORE 0 reads the coefficient W from the individual memory IMEM 0 and the image X from the memory bank B 0 a of the shared memory SMEM 0 and performs predetermined processing PR such as the convolution operation. The processor core CORE 0 then transfers the processing result to the memory bank B 1 a of the shared memory SMEM 1 belonging to the adjacent processor core CORE 1 .

Next, the processor core CORE 1 reads the coefficient W from the individual memory IMEM 1 and the image X as the operation result of the processor core CORE 0 from the memory bank B 1 a of the shared memory SMEM 1 and performs the predetermined processing PR such as the convolution operation. The processor core CORE 1 then transfers the processing result to the memory bank B 2 a of the shared memory SMEM 2 belonging to the adjacent processor core CORE 2 .

Subsequently, the processor core CORE 2 reads the coefficient W from the individual memory IMEM 2 and the image X as the operation result of the processor core CORE 1 from the memory bank B 2 a of the shared memory SMEM 2 and performs the predetermined processing PR such as the convolution operation. The processor core CORE 2 then transfers the processing result to the memory bank B 3 a of the shared memory SMEM 3 belonging to the adjacent processor core CORE 3 .

Finally, the processor core CORE 3 reads the coefficient W from the individual memory IMEM 3 and the image X as the operation result of the processor core CORE 2 from the memory bank B 3 a of the shared memory SMEM 3 and performs the predetermined processing PR such as the convolution operation. The processor core CORE 3 then transfers the processing result to the memory bank B 0 a of the shared memory SMEM 0 belonging to the adjacent processor core CORE 0 .

By performing the operations described above, the four processor cores transfer the operation results among the processor cores, and therefore it is possible to considerably reduce the processing for reading the operation results from the internal memory I_MEM.

As described above, in the present embodiment, the shared memory is provided to the processor core in addition to the individual memory, and when the processor core reads data from the individual memory and the shared memory and performs a predetermined operation, the processor core simultaneously transfers the read data to the shared memory belonging thereto or the shared memory belonging to the adjacent processor core. The adjacent processor core then reads the data transferred to the shared memory and performs a predetermined operation. The access operation to the shared memory performed by the processor core and the transfer operation for the read data to the shared memory are performed in accordance with control values set by the control core.

As described above, the data transfer among the processor cores is performed by using control values that are set by the processing core and performing, in accordance with the control values, a transfer request based on an address and a read/write signal that are output by the control core, and therefore, the data transfer may achieve less control overhead.

All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.