The photoelectric conversion device includes a pixel unit configured to output first and second signals from columns of first and second regions, and a processing circuit including a first processing circuit including first and third memories, a second processing circuit including second and fourth memories, and a data exchange circuit. The first memory stores the first and second signals output from the first region, and the second memory stores the first and second signals output from the second region. The data exchange circuit stores the first signals read out from the first and second memories in the third memory, and stores the second signals read out from the first and second memories in the fourth memory. The processing circuit outputs the first signals stored in the third memory from the first processing circuit, and outputs the second signals stored in the fourth memory from the second processing circuit.

Description

18 parts

BACKGROUND

›Field

The present invention relates to a photoelectric conversion device and an imaging system.

›Description of the Related Art

In a stacked photoelectric conversion device in which a pixel substrate provided with a pixel array and a signal processing substrate provided with a signal processing circuit are stacked, the signal processing circuit provided on the signal processing substrate may be divided into a plurality of regions. For example, a vertical scanning circuit may be disposed in a portion overlapping with the vicinity of the center of the pixel array for improvement in performance and convenience in manufacturing. In such a case, the column circuit unit and the data processing unit are divided into two in the horizontal direction and arranged on the signal processing substrate. Japanese Patent Application Laid-Open No. 2015-216334 discloses a solid-state imaging device in which a signal processing circuit is divided into two and arranged on a logic substrate.

When the column circuit corresponding to each column of the pixel array is divided in the horizontal direction, horizontal scanning is performed for each divided area to output data. When the data divided in the horizontal direction are combined, for example, as described in Japanese Patent Application Laid-Open No. 2015-216334, it is conceivable to collect the data in the data processing unit of one of the signal processing circuits.

However, when the signal processing circuit is configured in this manner, the other divided signal processing unit is not used. Further, when plural kinds of data independent from each other are read out from the pixel array and processed by the data processing unit, it is necessary to mount a circuit for processing plural kinds of data in one of the signal processing circuits, which increases the circuit scale.

›SUMMARY

An object of the present disclosure is to provide a photoelectric conversion device and an imaging system capable of efficiently outputting pixel data without increasing the circuit scale.

According to an aspect of the present disclosure, there is provided a photoelectric conversion device including a pixel unit in which a plurality of pixels each including a photoelectric converter is arranged to form a plurality of rows and a plurality of columns, and a processing circuit configured to process a signal output from the pixel unit, wherein the pixel unit includes a first region and a second region defined by columns, and is configured to output a plurality of kinds of signals including a first signal and a second signal from each of columns belonging to the first region and the second region, wherein the processing circuit includes a first processing circuit including a first memory and a third memory, a second processing circuit including a second memory and a fourth memory, and a data exchange circuit, wherein the first memory is configured to store the first signal and the second signal output from each column of the first region, wherein the second memory is configured to store the first signal and the second signal output from each column of the second region, wherein the data exchange circuit is configured to read out the first signal stored in the first memory and the first signal stored in the second memory to store them in the third memory, and read out the second signal stored in the first memory and the second signal stored in the second memory to store them in the fourth memory, and wherein the processing circuit is configured to output the first signals of the first region and the second region stored in the third memory from the first processing circuit, and to output the second signals of the first region and the second region stored in the fourth memory from the second processing circuit.

Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

›BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device according to a first embodiment.

FIG. 2 is a circuit diagram illustrating a configuration example of a pixel in the imaging device according to the first embodiment.

FIG. 3 is a perspective view schematically illustrating the imaging device according to the first embodiment.

FIG. 4 is a diagram illustrating a connection relationship between a pixel unit and a vertical scanning unit in the imaging device according to the first embodiment.

FIG. 5 is a diagram illustrating a connection relationship among a pixel unit, an AD conversion unit, and a horizontal scanning unit in the imaging device according to the first embodiment.

FIG. 6A and FIG. 6B are block diagrams illustrating a configuration example of processing circuits in the imaging device according to the first embodiment.

FIG. 7A and FIG. 7B are block diagrams illustrating a configuration example of processing circuits in an imaging device according to a second embodiment.

FIG. 8 is a circuit diagram illustrating a configuration example of a pixel in an imaging device according to a third embodiment.

FIG. 9 is a schematic top view of the pixel in an imaging device according to the third embodiment.

FIG. 10 is a diagram illustrating a connection relationship among a pixel unit, a vertical scanning unit, an AD conversion unit, and a horizontal scanning unit in the imaging device according to the third embodiment.

FIG. 11 is a block diagram illustrating a configuration example of processing circuits in an imaging device according to the third embodiment.

FIG. 12 , FIG. 13 , and FIG. 14 are plan views illustrating layout examples of the functional blocks on the signal processing substrate of the imaging device according to a fourth embodiment.

FIG. 15 is a block diagram illustrating a schematic configuration of an imaging system according to a fifth embodiment.

FIG. 16A is a diagram illustrating a configuration example of an imaging system according to a sixth embodiment.

FIG. 16B is a diagram illustrating a configuration example of a movable object according to the sixth embodiment.

›DESCRIPTION OF THE EMBODIMENTS · 1 of 12

Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.

First Embodiment

An imaging device according to a first embodiment will be described with reference to FIG. 1 to FIG. 6B . FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device according to the present embodiment. FIG. 2 is a circuit diagram illustrating a configuration example of a pixel in the imaging device according to the present embodiment. FIG. 3 is a perspective view schematically illustrating the imaging device according to the present embodiment. FIG. 4 is a diagram illustrating a connection relationship between a pixel unit and a vertical scanning unit in the imaging device according to the present embodiment. FIG. 5 is a diagram illustrating a connection relationship among a pixel unit, an AD conversion unit, and a horizontal scanning unit in the imaging device according to the present embodiment. FIG. 6A and FIG. 6B are block diagrams illustrating a configuration example of processing circuits in the imaging device according to the present embodiment.

As illustrated in FIG. 1 , for example, an imaging device 10 according to the present embodiment may include a processing device (processor) 1 , a control unit 2 , a vertical scanning unit 3 , a pixel unit 4 , an AD conversion unit (analog-to-digital conversion unit) 5 , a horizontal scanning unit 6 , a data processing unit 7 , and an output unit 8 . The vertical scanning unit 3 and the AD conversion unit 5 are connected to the pixel unit 4 . A horizontal scanning unit 6 is connected to the AD conversion unit 5 . A data processing unit 7 is connected to the horizontal scanning unit 6 . An output unit 8 is connected to the data processing unit 7 . A control unit 2 is connected to the vertical scanning unit 3 , the AD conversion unit 5 , the horizontal scanning unit 6 , and the data processing unit 7 . The processing device 1 is connected to the control unit 2 .

The processing device 1 is, for example, a central processing unit (CPU), and controls the overall operation of the imaging device 10 . The processing device 1 outputs a control signal such as a synchronization signal and a setting signal such as an operation mode to the control unit 2 . The processing device 1 may be provided in an imaging system in which the imaging device 10 is mounted, that is, outside the imaging device 10 .

The control unit 2 is a control circuit that receives control signals and setting signals from the processing device 1 and outputs control signals for controlling the operations and timings of the vertical scanning unit 3 , the AD conversion unit 5 , the horizontal scanning unit 6 , and the data processing unit 7 .

In the pixel unit 4 , a plurality of pixels P each including a photoelectric converter are two-dimensionally arranged so as to form a plurality of rows and a plurality of columns. Each of the plurality of pixels P is configured to output a pixel signal corresponding to the amount of light incident on the photoelectric converter. FIG. 1 illustrates a pixel unit 4 including pixels P arranged in a matrix of m columns×n rows, and coordinates represented by (column numbers, row numbers) are added to reference numerals P of the pixels. Note that in this specification, an extending direction (row direction) of each row is defined as a horizontal direction, and an extending direction (column direction) of each column is defined as a vertical direction. It is also assumed that the row number of the head row is the first row, and the column number of the head column is the first column.

The vertical scanning unit 3 is a control circuit that operates in response to a signal from the control unit 2 and drives the plurality of pixels P constituting the pixel unit 4 in a row basis. The operations performed by the vertical scanning unit 3 for the plurality of pixels P constituting the pixel unit 4 include readout scan and shutter scan. The shutter scan refers to an operation of sequentially resetting the photoelectric converters on a row-by-row basis and starting exposure for a part or all of the pixels P of the pixel unit 4 . The readout scan refers to an operation of sequentially outputting signals based on charges accumulated in the photoelectric converters on a row-by-row basis from the pixels P in a part or all of the rows of the pixel unit 4 . The vertical scanning unit 3 supplies control signals to the pixels P on a row-by-row basis via control lines (not illustrated) arranged in respective rows of the pixel array. The vertical scanning unit 3 may be configured using a shift register or an address decoder.

The AD conversion unit 5 may include an amplifier circuit (not illustrated), an analog-to-digital conversion (AD conversion) circuit, and a memory. The AD conversion unit 5 performs amplification processing and AD conversion processing on a pixel signal which is an analog signal output from the pixel P of the pixel unit 4 via a vertical output line (not illustrated), and holds the pixel signal as digital data in the memory.

The horizontal scanning unit 6 is a control circuit that operates in response to a signal from the control unit 2 and sequentially transfers pixel signals converted into digital data by the AD conversion unit 5 to the data processing unit 7 on a column-by-column basis. The horizontal scanning unit 6 may be configured using a shift register or an address decoder.

The data processing unit 7 is a functional block that performs predetermined processing on digital data transferred from the horizontal scanning unit 6 . A specific configuration of the data processing unit 7 and specific processing in the data processing unit 7 will be described later.

The output unit 8 is an output circuit that digitally processes the signal transferred from the data processing unit 7 and outputs the serial data to the outside of the imaging device 10 . The output unit 8 may include a digital processing unit, a parallel/serial conversion circuit, and an output circuit such as LVDS (Low Voltage Differential Signaling).

›DESCRIPTION OF THE EMBODIMENTS · 2 of 12

In this specification, among these functional blocks constituting the imaging device 10 , the control unit 2 , the AD conversion unit 5 , the horizontal scanning unit 6 , the data processing unit 7 , and the output unit 8 may be collectively referred to as a processing circuit.

FIG. 2 is a circuit diagram illustrating a configuration example of a pixel P. FIG. 2 exemplifies the pixels P (M, N) arranged in the M-th column and the N-th row of the pixel unit 4 , but the same applies to other pixels P. Note that M is an integer greater than or equal to 1 and less than or equal to m, and N is an integer greater than or equal to 1 and less than or equal to n.

As illustrated in FIG. 2 , each pixel P may include a photoelectric converter PD, a transfer transistor M 1 , a reset transistor M 2 , an amplifier transistor M 3 , and a select transistor M 4 .

The photoelectric converter PD may be configured by a photoelectric conversion element such as a photodiode or a photogate. Here, it is assumed that the photoelectric converter PD is formed of a photodiode. The photodiode constituting the photoelectric converter PD has an anode connected to the ground node and a cathode connected to the source of the transfer transistor M 1 . The drain of the transfer transistor M 1 is connected to the source of the reset transistor M 2 and the gate of the amplifier transistor M 3 . A connection node between the drain of the transfer transistor M 1 , the source of the reset transistor M 2 , and the gate of the amplifier transistor M 3 is a so-called floating diffusion portion FD. The drain of the reset transistor M 2 and the drain of the amplifier transistor M 3 are connected to a power supply voltage node (voltage VCC). The source of the amplifier transistor M 3 is connected to the drain of the select transistor M 4 . The source of the select transistor M 4 is connected to the vertical output line Vline (M).

In the case of the pixel configuration illustrated in FIG. 2 , each of the control lines arranged in each row of the pixel unit 4 includes a transfer gate signal line, a reset signal line, and a select signal line. For example, the transfer gate signal line of the N-th row is connected to the gates of the transfer transistors M 1 of the pixels P of the N-th row, and supplies the control signal PTX (N) output from the vertical scanning unit 3 to the gates of the transfer transistors M 1 of the pixels P. The reset signal line of the N-th row is connected to the gates of the reset transistors M 2 of the pixels P of the N-th row, and supplies the control signal PRES (N) output from the vertical scanning unit 3 to the gates of the reset transistors M 2 of the pixels P. The select signal line of the N-th row is connected to the gates of the select transistors M 4 of the pixels P of the N-th row, and supplies the control signal PSEL (N) output from the vertical scanning unit 3 to the gates of the select transistors M 4 of the pixels P. Common control signals are supplied from the vertical scanning unit 3 to the pixels P in the same row.

When each transistor is formed of an n-channel transistor, when a high-level control signal is supplied from the vertical scanning unit 3 , the corresponding transistor is brought into a conductive state, and when a low-level control signal is supplied from the vertical scanning unit 3 , the corresponding transistor is brought into a non-conductive state. Here, the high level corresponds to the logical value “1”, and the low level corresponds to the logical value “0”. Each transistor included in the pixel P may be configured by an n-channel transistor, but may be configured by a p-channel transistor.

The photoelectric converter PD converts (photoelectrically converts) the incident light into an amount of charge corresponding to the amount of light, and accumulates the generated charge. When the transfer transistor M 1 is turned on (conductive state), the charge held by the photoelectric converter PD is transferred to the floating diffusion portion FD. The floating diffusion portion FD includes a capacitance component, holds charge transferred from the photoelectric converter PD, and sets the floating diffusion portion FD to a potential corresponding to the amount of charge by charge-voltage conversion by the capacitance component thereof. The amplifier transistor M 3 has a configuration in which a voltage VCC is supplied to the drain thereof, and a bias current is supplied to the source thereof from a current source (not illustrated) via the vertical output line Vline and the select transistor M 4 , and constitutes a source follower circuit having a gate as an input node. Thus, the amplifier transistor M 3 outputs a signal based on the potential of the floating diffusion portion FD to the vertical output line Vline via the select transistor M 4 . The reset transistor M 2 is turned on to reset the floating diffusion portion FD to a potential corresponding to the power supply voltage. Further, by turning on (conductive state) the transfer transistor M 1 simultaneously with the reset transistor M 2 , the photoelectric converter PD may be reset to a potential corresponding to the voltage VCC. The select transistor M 4 switches the connection between the amplifier transistor M 3 and the vertical output line Vline.

For example, as illustrated in FIG. 3 , the imaging device 10 according to the present embodiment may be configured as a stacked sensor in which a pixel substrate 11 and a signal processing substrate 12 are stacked. For example, the pixel unit 4 of the functional blocks illustrated in FIG. 1 may be disposed on the pixel substrate 11 . In addition, among the functional blocks illustrated in FIG. 1 , for example, the control unit 2 , the vertical scanning unit 3 , the AD conversion unit 5 , the horizontal scanning unit 6 , the data processing unit 7 , and the output unit 8 may be arranged on the signal processing substrate 12 .

The vertical scanning unit 3 is divided into three blocks (vertical scanning units 3 L, 3 C, and 3 R) and is arranged on the signal processing substrate 12 . A processing circuit 13 is arranged between the vertical scanning unit 3 L and the vertical scanning unit 3 C, and a processing circuit 14 is arranged between the vertical scanning unit 3 C and the vertical scanning unit 3 R. The processing circuit 13 is provided with a processing circuit for processing the signals of the pixels P arranged in the first region defined by the columns among the pixels P arranged in the pixel unit 4 . The processing circuit 14 is provided with a processing circuit for processing the signals of the pixels P arranged in the second region defined by the columns among the pixels P arranged in the pixel unit 4 . Here, it is assumed that the range from the first column to the (m/2)-th column is the first region, and the range from the (m/2+1)-th column to the m-th column is the second region. Each of the control unit 2 , the AD conversion unit 5 , the horizontal scanning unit 6 , the data processing unit 7 , and the output unit 8 is divided into two blocks, and is divided into a processing circuit 13 and a processing circuit 14 . The processing circuit 13 and the processing circuit 14 are spaced apart from each other.

›DESCRIPTION OF THE EMBODIMENTS · 3 of 12

As illustrated in FIG. 4 , a signal line to which the control signal PRES is supplied, a signal line to which the control signal PTX is supplied, and a signal line to which the control signal PSEL is supplied are provided in each row of the pixel array constituting the pixel unit 4 . These signal lines are connected to a plurality of pixels P arranged in a corresponding row. For example, the signal lines arranged in the ( 12 i+ 1)-th row are connected to the pixels P (1, 12 i+ 1), . . . , P (j, 12 i+ 1), P (j+1, 12 i +1), P (j+2, 12 i +1), P (m, 12 i+ 1). Each of these signal lines is connected to the vertical scanning unit 3 provided on the signal processing substrate 12 via an electrical connection portion between the pixel substrate 11 and the signal processing substrate 12 . Note that i is an integer of 0 or more in which the row number is in the range of 1 to n. In the first row to the n-th row, the same blocks as the ( 12 i+ 1)-th row to the ( 12 i+ 12)-th row are repeatedly arranged in units of 12 rows.

As illustrated in FIG. 4 , a predetermined number of vertical output lines Vline are provided in each column of the pixel array constituting the pixel unit 4 . In FIG. 4 , it is assumed that eight vertical output lines Vline are provided in each column. For example, eight vertical output lines Vline of vertical output lines Vline 1 ( j ), Vline 2 ( j ), Vline 3 ( j ), Vline 4 ( j ), Vline 5 ( j ), Vline 6 ( j ), Vline 7 ( j ) and Vline 8 ( j ) are provided in the j-th column. The number of vertical output lines Vline provided in each column is not limited to eight, and may be appropriately increased or decreased. Note that j is an integer of 1 or more in which the column number is in the range of 1 to m.

A pixel P in the ( 12 i+ 1)-th row and a pixel P in the ( 12 i+ 3)-th row are connected to the vertical output line Vline 1 in each column. A pixel P in the ( 12 i+ 4)-th row and a pixel P in the ( 12 i+ 6)-th row are connected to the vertical output line Vline 2 in each column. A pixel P in the ( 12 i+ 7)-th row and a pixel P in the ( 12 i+ 9)-th row are connected to the vertical output line Vline 3 in each column. A pixel Pin the ( 12 i+ 10)-th row and a pixel P in the ( 12 i+ 12)-th row are connected to the vertical output line Vline 4 in each column. A pixel P in the ( 12 i+ 5)-th row is connected to the vertical output line Vline 5 in each column. A pixel P in the ( 12 i+ 2)-th row is connected to the vertical output line Vline 6 in each column. A pixel Pin the ( 12 i+ 11)-th row is connected to the vertical output line Vline 7 in each column. A pixel P in the ( 12 i+ 8)-th row is connected to the vertical output line Vline 8 in each column.

Pixel data obtained by adding pixel signals read out from a plurality of pixels P arranged in the vertical direction according to the control signals PRES, PTX, and PSEL supplied from the vertical scanning unit 3 is output to each of the vertical output lines Vline 1 , Vline 2 , Vline 3 , and Vline 4 . That is, pixel data obtained by adding the pixel signal of the pixel P in the ( 12 i+ 1)-th row and the pixel signal of the pixel P in the ( 12 i+ 3)-th row is read out to the vertical output line Vline 1 . Pixel data obtained by adding the pixel signal of the pixel P in the ( 12 i+ 4)-th row and the pixel signal of the pixel P in the ( 12 i+ 6)-th row is read out to the vertical output line Vline 2 . Pixel data obtained by adding the pixel signal of the pixel P in the ( 12 i+ 7)-th row and the pixel signal of the pixel P in the ( 12 i+ 9)-th row is read out to the vertical output line Vline 3 . Pixel data obtained by adding the pixel signal of the pixel P in the ( 12 i+ 10)-th row and the pixel signal of the pixel P in the ( 12 i+ 12)-th row is read out to the vertical output line Vline 4 .

Non-addition pixel data thinned out in the vertical direction according to the control signals PRES, PTX, and PSEL supplied from the vertical scanning unit 3 is output to each of the vertical output lines Vline 5 , Vline 6 , Vline 7 , and Vline 8 . That is, pixel data of the pixel P in the ( 12 i+ 5)-th row is read out to the vertical output line Vline 5 . Pixel data of the pixel P in the ( 12 i+ 2)-th row is read out to the vertical output line Vline 6 . Pixel data of the pixel P in the ( 12 i+ 11)-th row is read out to the vertical output line Vline 7 . Pixel data of the pixel P in the ( 12 i+ 8)-th row is read out to the vertical output line Vline 8 .

That is, the imaging device 10 of the present embodiment is configured to be able to read out two types of pixel data, i.e., pixel data obtained by adding pixel signals read out from a plurality of pixels P arranged in the vertical direction, and non-addition pixel data thinned out in the vertical direction. In other words, the imaging device 10 of the present embodiment is configured to output a plurality of types of signals including at least the first signal and the second signal from each column of the pixel unit 4 . Here, the first signal is pixel data obtained by adding pixel signals read out from a first number (two) of pixels P arranged in the vertical direction. The second signal is pixel data obtained by adding pixel signals read out from a second number (one) of pixels P different from the first number arranged in the vertical direction.

As illustrated in FIG. 5 , each vertical output line Vline is connected to the AD conversion unit 5 provided on the signal processing substrate 12 via an electrical connection portion between the pixel substrate 11 and the signal processing substrate 12 . The AD conversion unit 5 includes a plurality of (m) column circuit groups 51 corresponding to respective columns of the pixel array constituting the pixel unit 4 . Each of the column circuit groups 51 includes a plurality of (eight in the example of FIG. 5 ) column circuits 52 corresponding to the number of vertical output lines Vline arranged in a corresponding column. In other words, the AD conversion unit 5 includes m×8 column circuits 52 for a pixel array of n rows and m columns.

›DESCRIPTION OF THE EMBODIMENTS · 4 of 12

As illustrated in FIG. 5 , the column circuit group 51 of the j-th column includes column circuits 52 ( j, 1), 52 ( j, 2), 52 ( j, 3), 52 ( j, 4), 52 ( j, 5), 52 ( j, 6), 52 ( j, 7), and 52 ( j, 8). The column circuit 52 ( j, 1) is connected to the vertical output line Vline 1 ( j ). The column circuit 52 ( j, 2) is connected to the vertical output line Vline 2 ( j ). The column circuit 52 ( j, 3) is connected to the vertical output line Vline 3 ( j ). The column circuit 52 ( j, 4) is connected to the vertical output line Vline 4 ( j ). The column circuit 52 ( j, 5) is connected to the vertical output line Vline 5 ( j ). The column circuit 52 ( j, 6) is connected to the vertical output line Vline 6 ( j ). The column circuit 52 ( j, 7) is connected to the vertical output line Vline 7 ( j ). The column circuit 52 ( j, 8) is connected to the vertical output line Vline 8 ( j ).

As described above, the AD conversion unit 5 is divided into two and arranged in the processing circuit 13 and the processing circuit 14 . In the present embodiment, half of the m×8 column circuits 52 constituting the AD conversion unit 5 are arranged in the processing circuit 13 , and the remaining half are arranged in the processing circuit 14 . The vertical output lines Vline of the respective columns from the first column to the (m/2)-th column are connected to the column circuit 52 arranged in the processing circuit 13 . The vertical output lines Vline of the respective columns from the (m/2+1)-th column to the m-th column are connected to the column circuit 52 arranged in the processing circuit 14 .

The horizontal scanning unit 6 includes a plurality of (eight in the example of FIG. 5 ) horizontal scanning circuits 61 corresponding to the number of vertical output lines Vline and column circuits 52 arranged in each column of the pixel array constituting the pixel unit 4 . The column circuits 52 belonging to the column circuit group 51 of the same column are connected to different horizontal scanning circuits 61 . For example, as illustrated in FIG. 5 , the horizontal scanning unit 6 includes horizontal scanning circuits 61 ( 1 ), 61 ( 2 ), 61 ( 3 ), 61 ( 4 ), 61 ( 5 ), 61 ( 6 ), 61 ( 7 ), and 61 ( 8 ). The column circuit (j, 1) of the j-th column is connected to the horizontal scanning circuit 61 ( 1 ). The column circuit (j, 2) of the j-th column is connected to the horizontal scanning circuit 61 ( 2 ). The column circuit (j, 3) of the j-th column is connected to the horizontal scanning circuit 61 ( 3 ). The column circuit (j, 4) of the j-th column is connected to the horizontal scanning circuit 61 ( 4 ). The column circuit (j, 5) of the j-th column is connected to the horizontal scanning circuit 61 ( 5 ). The column circuit (j, 6) of the j-th column is connected to the horizontal scanning circuit 61 ( 6 ). The column circuit (j, 7) of the j-th column is connected to the horizontal scanning circuit 61 ( 7 ). The column circuit (j, 8) of the j-th column is connected to the horizontal scanning circuit 61 ( 8 ).

As described above, the horizontal scanning unit 6 is divided into two and arranged in the processing circuit 13 and the processing circuit 14 . The horizontal scanning circuit 61 arranged in the processing circuit 13 sequentially and horizontally transfers digital data of the pixels P from the first column to the (m/2)-th column, and outputs the digital data to the data processing unit 7 of the processing circuit 13 . The horizontal scanning circuit 61 arranged in the processing circuit 14 sequentially horizontally transfers digital data of the pixels P from the (m/2+1)-th column to the m-th column, and outputs the digital data to the data processing unit 7 of the processing circuit 14 .

Next, a configuration example of the processing circuit 13 and the processing circuit 14 in the imaging device 10 according to the present embodiment will be described in more detail with reference to FIG. 6A and FIG. 6B .

The processing circuit 13 includes an AD conversion unit 5 L, a horizontal scanning unit 6 L, a data processing unit 7 L, and an output unit 8 L. The processing circuit 14 includes an AD conversion unit 5 R, a horizontal scanning unit 6 R, a data processing unit 7 R, and an output unit 8 R. The AD conversion units 5 L and 5 R and the horizontal scanning units 6 L and 6 R are as described with reference to FIG. 4 and FIG. 5 .

The data processing unit 7 L may include an input data storage unit 71 L, an operation data storage unit 72 L, an operation data input unit 73 L, and an operation unit 74 L. Each of the input data storage unit 71 L and the operation data storage unit 72 L includes a plurality of memories corresponding to the number of horizontal scanning circuits 61 L included in the horizontal scanning unit 6 L. For example, the input data storage unit 71 L includes eight memories 71 L 1 , 71 L 2 , 71 L 3 , 71 L 4 , 71 L 5 , 71 L 6 , 71 L 7 and 71 L 8 , as illustrated in FIG. 6A . As illustrated in FIG. 6A , the operation data storage unit 72 L includes eight memories 72 L 1 , 72 L 2 , 72 L 3 , 72 L 4 , 72 L 5 , 72 L 6 , 72 L 7 and 72 L 8 .

Similarly, the data processing unit 7 R may include an input data storage unit 71 R, an operation data storage unit 72 R, an operation data input unit 73 R, and an operation unit 74 R. Each of the input data storage unit 71 R and the operation data storage unit 72 R includes a plurality of memories corresponding to the number of horizontal scanning circuits 61 R included in the horizontal scanning unit 6 R. For example, the input data storage unit 71 R includes eight memories 71 R 1 , 71 R 2 , 71 R 3 , 71 R 4 , 71 R 5 , 71 R 6 , 71 R 7 and 71 R 8 , as illustrated in FIG. 6A . As illustrated in FIG. 6A , the operation data storage unit 72 R includes eight memories 72 R 1 , 72 R 2 , 72 R 3 , 72 R 4 , 72 R 5 , 72 R 6 , 72 R 7 and 72 R 8 .

The horizontal scanning circuits 61 ( 1 ) to 61 ( 8 ) of the horizontal scanning unit 6 L are connected to the memories 71 L 1 to 71 L 8 of the input data storage unit 71 L, respectively. The horizontal scanning circuits 61 ( 1 ) to 61 ( 8 ) of the horizontal scanning unit 6 R are connected to the memories 71 R 1 to 71 R 8 of the input data storage unit 71 R, respectively.

›DESCRIPTION OF THE EMBODIMENTS · 5 of 12

The input data storage units 71 L and 71 R and the operation data storage units 72 L and 72 R are connected via the data exchange circuit 9 . Specifically, the memories 71 L 1 to 71 L 4 of the input data storage unit 71 L are connected to the memories 72 L 1 to 72 L 4 of the operation data storage unit 72 L. The memories 71 L 5 to 71 L 8 of the input data storage unit 71 L are connected to the memories 72 R 1 to 72 R 4 of the operation data storage unit 72 R. The memories 71 R 1 to 71 R 4 of the input data storage unit 71 R are connected to the memories 72 L 5 to 72 L 8 of the operation data storage unit 72 L. The memories 71 R 5 to 71 R 8 of the input data storage unit 71 R are connected to the memories 72 R 5 to 72 R 8 of the operation data storage unit 72 R. The data exchange circuit 9 may be configured by, for example, a bus. In this case, for example, data transfer between memories is possible by serial transfer on a pixel-by-pixel basis.

As illustrated in FIG. 6B , the operation data input unit 73 L is connected to the operation unit 74 L. The operation unit 74 L is connected to the output unit 8 L. Similarly, the operation data input unit 73 R is connected to the operation unit 74 R. The operation unit 74 R is connected to the output unit 8 R.

Next, the operation of the processing circuits 13 and 14 will be described with reference to FIG. 6A and FIG. 6B .

The pixel data read out from the vertical output line Vline 1 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 1 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 1 of the input data storage unit 71 L. This data is called data D 1 L.

The pixel data read out from the vertical output line Vline 2 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 2 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 2 of the input data storage unit 71 L. This data is called data D 2 L.

The pixel data read out from the vertical output line Vline 3 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 3 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 3 of the input data storage unit 71 L. This data is called data D 3 L.

The pixel data read out from the vertical output line Vline 4 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 4 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 4 of the input data storage unit 71 L. This data is called data D 4 L.

The pixel data read out from the vertical output line Vline 5 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 5 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 5 of the input data storage unit 71 L. This data is called data DSL.

The pixel data read out from the vertical output line Vline 6 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 6 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 6 of the input data storage unit 71 L. This data is called data D 6 L.

The pixel data read out from the vertical output line Vline 7 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 7 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 7 of the input data storage unit 71 L. This data is called data D 7 L.

The pixel data read out from the vertical output line Vline 8 of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 ( 8 ) of the horizontal scanning unit 6 L and stored in the memory 71 L 8 of the input data storage unit 71 L. This data is called data D 8 L.

Similarly, the pixel data read out from the vertical output line Vline 1 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 1 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 1 of the input data storage unit 71 R. This data is called data D 1 R.

The pixel data read out from the vertical output line Vline 2 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 2 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 2 of the input data storage unit 71 R. This data is called data D 2 R.

The pixel data read out from the vertical output line Vline 3 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 3 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 3 of the input data storage unit 71 R. This data is called data D 3 R.

The pixel data read out from the vertical output line Vline 4 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 4 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 4 of the input data storage unit 71 R. This data is called data D 4 R.

The pixel data read out from the vertical output line Vline 5 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 5 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 5 of the input data storage unit 71 R. This data is called data D 5 R.

The pixel data read out from the vertical output line Vline 6 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 6 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 6 of the input data storage unit 71 R. This data is called data D 6 R.

›DESCRIPTION OF THE EMBODIMENTS · 6 of 12

The pixel data read out from the vertical output line Vline 7 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 7 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 7 of the input data storage unit 71 R. This data is called data D 7 R.

The pixel data read out from the vertical output line Vline 8 of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 ( 8 ) of the horizontal scanning unit 6 R and stored in the memory 71 R 8 of the input data storage unit 71 R. This data is called data D 8 R.

Among the data stored in the input data storage unit 71 L, the data D 1 L to D 4 L stored in the memories 71 L 1 to 71 L 4 are transferred to the memories 72 L 1 to 72 L 4 of the operation data storage unit 72 L, respectively. Among the data stored in the input data storage unit 71 L, the data D 5 L to D 8 L stored in the memories 71 L 5 to 71 L 8 are transferred to the memories 72 R 1 to 72 R 4 of the operation data storage unit 72 R, respectively.

Among the data stored in the input data storage unit 71 R, the data D 1 R to D 4 R stored in the memories 71 R 1 to 71 R 4 are transferred to the memories 72 L 5 to 72 L 8 of the operation data storage unit 72 L, respectively. Among the data stored in the input data storage unit 71 R, the data D 5 R to D 8 R stored in the memories 71 R 5 to 71 R 8 are transferred to the memories 72 R 5 to 72 R 8 of the operation data storage unit 72 R, respectively.

That is, in the present embodiment, the data D 5 L, D 6 L, D 7 L, and D 8 L are transferred from the processing circuit 13 to the processing circuit 14 , and the data D 1 R, D 2 R, D 3 R, and D 4 R are transferred from the processing circuit 14 to the processing circuit 13 .

Thus, the operation data storage unit 72 L stores pixel data (Data D 1 L, D 2 L, D 3 L, D 4 L, D 1 R, D 2 R, D 3 R, D 4 R) from the first column to the m-th column corresponding to the vertical output lines Vline 1 , Vline 2 , Vline 3 , and Vline 4 . The operation data storage unit 72 R stores pixel data (Data D 5 L, D 6 L, D 7 L, D 8 L, D 5 R, D 6 R, D 7 R, D 8 R) from the first column to the m-th column corresponding to the vertical output lines Vline 5 , Vline 6 , Vline 7 , and Vline 8 .

The pixel data corresponding to the vertical output lines Vline 1 , Vline 2 , Vline 3 , and Vline 4 stored in the operation data storage unit 72 L in this manner is pixel data obtained by adding pixels in the vertical direction. On the other hand, the pixel data corresponding to the vertical output lines Vline 5 , Vline 6 , Vline 7 , and Vline 8 , which are stored in the operation data storage unit 72 R in this manner, are non-addition pixel data thinned out in the vertical direction. That is, data corresponding to two different types of pixel readout methods may be separately stored in the operation data storage unit 72 L and the operation data storage unit 72 R.

The data stored in the operation data storage unit 72 L is read out by the operation data input unit 73 L, subjected to operation processing by the operation unit 74 L, and output from the output unit 8 L. Similarly, the data stored in the operation data storage unit 72 R is read out by the operation data input unit 73 R, subjected to operation processing by the operation unit 74 R, and output from the output unit 8 R. Since the operation unit 74 L and the operation unit 74 R are independent from each other, different operation processes may be performed simultaneously in each of the operation units 74 L and 74 R. Thus, all the divided data processing units 7 L and 7 R may be used to separate and simultaneously output data corresponding to two different types of pixel readout methods.

As described above, according to the present embodiment, efficient processing of pixel data may be realized without increasing the circuit scale.

Second Embodiment

An imaging device according to a second embodiment will be described with reference to FIG. 7A and FIG. 7B . FIG. 7A and FIG. 7B are block diagrams illustrating a configuration example of a processing circuit in the imaging device according to the present embodiment. The same components as those of the imaging device according to the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified.

In the first embodiment, data is divided and output according to two types of pixel readout methods of vertical addition and non-addition. In the present embodiment, an example will be described in which pixel data of R/Gr rows and pixel data of Gb/B rows are separated and output in an imaging device having a pixel array of a Bayer arrangement. The R/Gr row is a row in which pixels (R pixels) having red color filters and pixels (G pixels) having green color filters are alternately arranged. The Gb/B row is a row in which G pixels and pixels (B pixels) having blue color filters are alternately arranged. The R/Gr row and the Gb/B row are alternately arranged. For example, in the pixel array of FIG. 1 , the pixel P (1, 1) is an R pixel, the pixel P (2, 1) is a G pixel, the pixel P (1, 2) is a G pixel, and the pixel P (2, 2) is a B pixel. Pixels P in adjacent rows in the same column are pixels of different colors.

The imaging device according to the present embodiment is the same as the imaging device 10 according to the first embodiment except that the data processing units 7 L and 7 R have different configurations. Hereinafter, the imaging device according to the present embodiment will be described focusing on points different from the imaging device according to the first embodiment.

As in the first embodiment, the data processing unit 7 L in the imaging device according to the present embodiment includes an input data storage unit 71 L, an operation data storage unit 72 L, an operation data input unit 73 L, and an operation unit 74 L. As in the first embodiment, the data processing unit 7 R in the imaging device according to the present embodiment includes an input data storage unit 71 R, an operation data storage unit 72 R, an operation data input unit 73 R, and an operation unit 74 R. The data processing units 7 L and 7 R of the present embodiment are different from those of the first embodiment in the connection between the input data storage units 71 L and 71 R and the operation data storage units 72 L and 72 R in the data exchange circuit 9 .

›DESCRIPTION OF THE EMBODIMENTS · 7 of 12

That is, in the present embodiment, the memory 71 L 1 , 71 L 3 , 71 L 5 and 71 L 7 of the input data storage unit 71 L is connected to the memory 72 L 1 , 72 L 3 , 72 L 5 and 72 L 7 of the operation data storage unit 72 L. The memory 71 L 2 , 71 L 4 , 71 L 6 and 71 L 8 of the input data storage unit 71 L is connected to the memory 72 R 1 , 72 R 3 , 72 R 5 and 72 R 7 of the operation data storage unit 72 R. The memory 71 R 1 , 71 R 3 , 71 R 5 and 71 R 7 of the input data storage unit 71 R is connected to the memory 72 L 2 , 72 L 4 , 72 L 6 and 72 L 8 of the operation data storage unit 72 L. The memory 71 R 2 , 71 R 4 , 71 R 6 and 71 R 8 of the input data storage unit 71 R is connected to the memory 72 R 2 , 72 R 4 , 72 R 6 and 72 R 8 of the operation data storage unit 72 R.

Thus, the data D 1 L, D 3 L, D 5 L, and D 7 L stored in the memory 71 L 1 , 71 L 3 , 71 L 5 and 71 L 7 of the input data storage unit 71 L are transferred to the memory 72 L 1 , 72 L 3 , 72 L 5 and 72 L 7 of the operation data storage unit 72 L, respectively. The data D 2 L, D 4 L, D 6 L, and D 8 L stored in the memory 71 L 2 , 71 L 4 , 71 L 6 and 71 L 8 of the input data storage unit 71 L are transferred to the memory 72 R 1 , 72 R 3 , 72 R 5 and 72 R 7 of the operation data storage unit 72 R, respectively.

The data D 1 R, D 3 R, D 5 R, and D 7 R stored in the memory 71 R 1 , 71 R 3 , 71 R 5 and 71 R 7 of the input data storage unit 71 R are transferred to the memory 72 L 2 , 72 L 4 , 72 L 6 and 72 L 8 of the operation data storage unit 72 L, respectively. The data D 2 R, D 4 R, D 6 R, and D 8 R stored in the memory 71 R 2 , 71 R 4 , 71 R 6 and 71 R 8 of the input data storage unit 71 R are transferred to the memory 72 R 2 , 72 R 4 , 72 R 6 and 72 R 8 of the operation data storage unit 72 R, respectively.

That is, in the present embodiment, the data D 2 L, D 4 L, D 6 L, and D 8 L are transferred from the processing circuit 13 to the processing circuit 14 , and the data D 1 R, D 3 R, D 5 R, and D 7 R are transferred from the processing circuit 14 to the processing circuit 13 .

Thus, the operation data storage unit 72 L stores pixel data (Data D 1 L, D 3 L, D 5 L, D 7 L, D 1 R, D 3 R, D 5 R, D 7 R) from the first column to the m-th column corresponding to the vertical output lines Vline 1 , Vline 3 , Vline 5 and Vline 7 . The operation data storage unit 72 R stores pixel data (Data D 2 L, D 4 L, D 6 L, D 8 L, D 2 R, D 4 R, D 6 R, D 8 R) from the first column to the m-th column corresponding to the vertical output lines Vline 2 , Vline 4 , Vline 6 , and Vline 8 .

In the pixel array of FIG. 4 , R/Gr rows are, for example, ( 12 i+ 1)-th row, ( 12 i+ 3)-th row, ( 12 i+ 5)-th row, ( 12 i+ 7)-th row, ( 12 i+ 9)-th row, and ( 12 i+ 11)-th row. The Gb/B row is, for example, ( 12 i+ 2)-th row, ( 12 i+ 4)-th row, ( 12 i+ 6)-th row, ( 12 i+ 8)-th row, ( 12 i+ 10)-th row, and ( 12 i+ 12)-th row. Therefore, the data output to the vertical output lines Vline 1 , Vline 3 , Vline 5 , and Vline 7 are data of R/Gr rows, and the data output to the vertical output lines Vline 2 , Vline 4 , Vline 6 , and Vline 8 are data of Gb/B rows.

By configuring the data processing unit 7 in this way, the data of the R/Gr row may be stored in the operation data storage unit 72 L, and the data of the Gb/B row may be stored in the operation data storage unit 72 R. Thus, the data in the R/Gr row and the data in the Gb/B row may be separated and output simultaneously.

As described above, according to the present embodiment, efficient processing of pixel data may be realized without increasing the circuit scale.

Third Embodiment

An imaging device according to a third embodiment will be described with reference to FIG. 8 to FIG. 11 . FIG. 8 is a circuit diagram illustrating a configuration example of a pixel in the imaging device according to the present embodiment. FIG. 9 is a schematic top view of the pixel in the imaging device according to the present embodiment. FIG. 10 is a diagram illustrating a connection relationship between a pixel unit, a vertical scanning unit, an AD conversion unit, and a horizontal scanning unit in the imaging device according to the present embodiment. FIG. 11 is a block diagram illustrating a configuration example of a processing circuit in the imaging device according to the present embodiment. The same components as those of the imaging device according to the first and second embodiments are denoted by the same reference numerals, and the description thereof will be omitted or simplified.

In the present embodiment, an example is described in which the image generation data and the focus detection data are separated and output simultaneously in the imaging device having the image plane phase difference AF function.

The imaging device 10 according to the present embodiment has the same schematic configuration as that of the imaging device according to the first embodiment except that the circuit configuration of the pixel P is different. Hereinafter, the imaging device according to the present embodiment will be described focusing on points different from the imaging device according to the first embodiment.

As illustrated in FIG. 8 , each of the pixels P in the imaging device 10 according to the present embodiment may include photoelectric converters PDA and PDB, transfer transistors M 1 A and M 1 B, a reset transistor M 2 , an amplifier transistor M 3 , and a select transistor M 4 .

›DESCRIPTION OF THE EMBODIMENTS · 8 of 12

The photoelectric converters PDA and PDB may be configured by photoelectric conversion elements such as photodiodes and photogates. Here, it is assumed that the photoelectric converters PDA and PDB are formed of photodiodes. The photodiode constituting the photoelectric converter PDA has an anode connected to the ground node and a cathode connected to the source of the transfer transistor M 1 A. The photodiode constituting the photoelectric converter PDB has an anode connected to the ground node and a cathode connected to the source of the transfer transistor M 1 B. The drain of the transfer transistor M 1 A and the drain of the transfer transistor M 1 B are connected to the source of the reset transistor M 2 and the gate of the amplifier transistor M 3 . A connection node between the drain of the transfer transistor M 1 A, the drain of the transfer transistor M 1 B, the source of the reset transistor M 2 , and the gate of the amplifier transistor M 3 is a floating diffusion portion FD. The drain of the reset transistor M 2 and the drain of the amplifier transistor M 3 are connected to a power supply voltage node (voltage VCC). The source of the amplifier transistor M 3 is connected to the drain of the select transistor M 4 . The source of the select transistor M 4 is connected to the vertical output line Vline.

In the case of the pixel configuration illustrated in FIG. 8 , each of the control lines arranged in each row of the pixel unit 4 includes a transfer gate signal line connected to the gate of the transfer transistor M 1 A and a transfer gate signal line connected to the gate of the transfer transistor M 1 B. The control line in each row further includes a reset signal line connected to the gate of the reset transistor M 2 and a selection signal line connected to the gate of the select transistor M 4 .

Control signals PTXA (N), PTXB (N), PRES (N), and PSEL (N) are supplied from the vertical scanning unit 3 to the pixels P arranged in the N-th row. The control signal PTXA (N) is a control signal supplied to the gates of the transfer transistors M 1 A of the pixels P of the N-th row. The control signal PTXB (N) is a control signal supplied to the gates of the transfer transistors M 1 B of the pixels P of the N-th row. The control signal PRES (N) is a control signal supplied to the gates of the reset transistors M 2 of the pixels P of the N-th row. The control signal PSEL (N) is a control signal supplied to the gates of the select transistors M 4 of the pixels P of the N-th row. A common control signal is supplied from the vertical scanning unit 3 to the pixels P in the same row.

When each transistor is formed of an n-channel transistor, when a high-level control signal is supplied from the vertical scanning unit 3 , the corresponding transistor is turned on, and when a low-level control signal is supplied from the vertical scanning unit 3 , the corresponding transistor is turned off. Here, the high level corresponds to the logical value “1”, and the low level corresponds to the logical value “0”. Each transistor included in the pixel P may be configured by an n-channel transistor, but may be configured by a p-channel transistor.

The photoelectric converters PDA and PDB convert (photoelectrically convert) incident light into charge of an amount corresponding to the amount of light, and accumulate the generated charge. When the transfer transistor M 1 A is turned on (conductive state), the charge held by the photoelectric converter PDA is transferred to the floating diffusion portion FD. When the transfer transistor M 1 B is turned on (conductive state), the charge held by the photoelectric converter PDB is transferred to the floating diffusion portion FD. The floating diffusion portion FD includes a capacitance component, holds charges transferred from the photoelectric converters PDA and PDB, and sets the potential of the floating diffusion portion FD to a potential corresponding to the amount of charges by charge-voltage conversion by the capacitance component thereof. The amplifier transistor M 3 has a configuration in which a power supply voltage VCC is supplied to the drain thereof, and a bias current is supplied to the source thereof from a current source (not illustrated) via the vertical output line Vline and the select transistor M 4 , and constitutes a source follower circuit having a gate as an input node. Thus, the amplifier transistor M 3 outputs a signal based on the potential of the floating diffusion portion FD to the vertical output line Vline via the select transistor M 4 . The reset transistor M 2 is turned on to reset the floating diffusion portion FD to a potential corresponding to the power supply voltage. Further, by turning on (conductive state) the transfer transistors M 1 A and M 1 B simultaneously with the reset transistor M 2 , the photoelectric converters PDA and PDB may be reset to a potential corresponding to the power supply voltage. The select transistor M 4 switches the connection between the amplifier transistor M 3 and the vertical output line Vline.

Further, the photoelectric converter PDA and the photoelectric converter PDB constituting one pixel P share one microlens ML, for example, as illustrated in FIG. 9 . In other words, the photoelectric converter PDA and the photoelectric converter PDB are configured to receive light that has passed through different pupil regions of light incident on the imaging optical system. With this configuration, the signal based on the charge generated by the photoelectric converter PDA and the signal based on the charge generated by the photoelectric converter PDB may be used as signals for focus detection.

In the pixel P of the present embodiment, the charge of the photoelectric converter PDA and the charge of the photoelectric converter PDB are individually transferred. First, the charge of the photoelectric converter PDA is transferred to the floating diffusion portion FD by the transfer transistor M 1 A, and the signal output from the pixel P is processed for AD conversion by the AD conversion unit 5 . Next, the charge of the photoelectric converter PDB is transferred to the floating diffusion portion FD by the transfer transistor M 1 B, and the signal output from the pixel P is processed for AD conversion by the AD conversion unit 5 . That is, in the imaging device of the present embodiment, two types of data are obtained for one exposure.

›DESCRIPTION OF THE EMBODIMENTS · 9 of 12

As illustrated in FIG. 10 , a signal line to which the control signal PRES is supplied, a signal line to which the control signal PTXA is supplied, a signal line to which the control signal PTXB is supplied, and a signal line to which the control signal PSEL is supplied are provided in each row of the pixel array constituting the pixel unit 4 . These signal lines are connected to a plurality of pixels P arranged in a corresponding row. For example, the signal lines arranged in the i-th row are connected to the pixels P (1, i), . . . , P (j, i), P (j+1, i), . . . , P (m, i). Each of these signal lines is connected to the vertical scanning unit 3 provided on the signal processing substrate 12 via an electrical connection portion between the pixel substrate 11 and the signal processing substrate 12 .

As illustrated in FIG. 10 , one vertical output line Vline is provided in each column of the pixel array constituting the pixel unit 4 . For example, the vertical output line Vline (j) is provided in the j-th column. All pixels P arranged in the j-th column are connected to the vertical output line Vline (j). The vertical output line Vline of each column reads out pixel data from the pixels P of the corresponding column in response to the control signals PRES, PTXA, PTXB, and PSEL supplied from the vertical scanning unit 3 . Each vertical output line Vline is connected to the AD conversion unit 5 provided on the signal processing substrate 12 via an electrical connection portion between the pixel substrate 11 and the signal processing substrate 12 .

The AD conversion unit 5 includes a plurality of (m) column circuits 52 corresponding to respective columns of the pixel array of n rows and m columns constituting the pixel unit 4 . The vertical output line Vline of each column is connected to the column circuit 52 of the corresponding column. For example, the vertical output line Vline (j) of the j-th column is connected to the column circuit 52 ( j ) corresponding to the j-th column. The horizontal scanning unit 6 includes a horizontal scanning circuit 61 . Each column circuit 52 is connected to a horizontal scanning circuit 61 . The horizontal scanning circuit 61 is connected to the data processing unit 7 .

As described above, each of the AD conversion unit 5 , the horizontal scanning unit 6 , the data processing unit 7 , and the output unit 8 is divided into the processing circuit 13 and the processing circuit 14 . The half of the m×8 column circuits 52 constituting the AD conversion unit 5 is arranged in the processing circuit 13 , and the remaining half is arranged in the processing circuit 14 . The vertical output lines Vline of the respective columns from the first column to the (m/2)-th column are connected to the column circuit 52 arranged in the processing circuit 13 . The vertical output lines Vline of the respective columns from the (m/2+1)-th column to the m-th column are connected to the column circuit 52 arranged in the processing circuit 14 . The horizontal scanning circuit 61 arranged in the processing circuit 13 sequentially and horizontally transfers digital data of the pixels P from the first column to the (m/2)-th column, and outputs the digital data to the data processing unit 7 of the processing circuit 13 . The horizontal scanning circuit 61 arranged in the processing circuit 14 sequentially horizontally transfers digital data of the pixels P from the (m/2+1)-th column to the m-th column, and outputs the digital data to the data processing unit 7 of the processing circuit 14 .

In the present embodiment, two types of data are acquired for one exposure of one pixel P. In the readout operation of the pixel P, first, the charge of the photoelectric converter PDA is transferred to the floating diffusion portion FD by the transfer transistor M 1 A, and the signal output from the pixel P is processed for AD conversion by the AD conversion unit 5 . The digital data generated in this way is called data A. Next, the charge of the photoelectric converter PDB is transferred to the floating diffusion portion FD by the transfer transistor M 1 B, and the signal output from the pixel P is processed for AD conversion by the AD conversion unit 5 . The digital data generated in this way is called data B. The data A may be a signal corresponding to the amount of charge generated in the photoelectric converter PDA during a predetermined exposure period. The data B may be a signal corresponding to the amount of charges generated in the photoelectric converters PDA and PDB during a predetermined exposure period.

Next, a configuration example of the processing circuit 13 and the processing circuit 14 in the imaging device according to the present embodiment will be described in more detail with reference to FIG. 11 .

The data processing unit 7 L may include an input data storage unit 71 L, an operation data storage unit 72 L, an operation data input unit 73 L, and an operation unit 74 L. Each of the input data storage unit 71 L and the operation data storage unit 72 L includes a number of memories corresponding to the types of data (two types in this case) output by the pixel P. For example, the input data storage unit 71 L includes two memories 71 L 1 and 71 L 2 . The operation data storage unit 72 L includes two memories 72 L 1 and 72 L 2 .

Similarly, the data processing unit 7 R may include an input data storage unit 71 R, an operation data storage unit 72 R, an operation data input unit 73 R, and an operation unit 74 R. Each of the input data storage unit 71 R and the operation data storage unit 72 R includes a number of memories corresponding to the types of data (two types in this case) output by the pixel P. For example, the input data storage unit 71 R includes two memories 71 R 1 and 71 R 2 . The operation data storage unit 72 R includes two memories 72 R 1 and 72 R 2 .

›DESCRIPTION OF THE EMBODIMENTS · 10 of 12

The horizontal scanning circuit 61 of the horizontal scanning unit 6 L is connected to the memory 71 L 1 and the memory 71 L 2 of the input data storage unit 71 L. The horizontal scanning circuit 61 of the horizontal scanning unit 6 R is connected to the memory 71 R 1 and the memory 71 R 2 of the input data storage unit 71 R.

The memory 71 L 1 of the input data storage unit 71 L is connected to the memory 72 L 1 of the operation data storage unit 72 L. The memory 71 L 2 of the input data storage unit 71 L is connected to the memory 72 R 1 of the operation data storage unit 72 R. The memory 71 R 1 of the input data storage unit 71 R is connected to the memory 72 L 2 of the operation data storage unit 72 L. The memory 71 R 2 of the input data storage unit 71 R is connected to the memory 72 R 2 of the operation data storage unit 72 R.

The operation data input unit 73 L is connected to the operation unit 74 L. The operation unit 74 L is connected to the output unit 8 L. Similarly, the operation data input unit 73 R is connected to the operation unit 74 R. The operation unit 74 R is connected to the output unit 8 R.

Next, the operation of the processing circuits 13 and 14 will be described with reference to FIG. 11 .

The data A read out from the vertical output line Vline of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 of the horizontal scanning unit 6 L and stored in the memory 71 L 1 of the input data storage unit 71 L. This data is called data AL.

The data B read out from the vertical output line Vline of each column from the first column to the (m/2)-th column is sequentially transferred by the horizontal scanning circuit 61 of the horizontal scanning unit 6 L and stored in the memory 71 L 2 of the input data storage unit 71 L. This data is called data BL.

Similarly, the data A read out from the vertical output line Vline of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 of the horizontal scanning unit 6 R and stored in the memory 71 R 1 of the input data storage unit 71 R. This data is called data AR.

The data B read out from the vertical output line Vline of each column from the (m/2+1)-th column to the m-th column is sequentially transferred by the horizontal scanning circuit 61 of the horizontal scanning unit 6 R and stored in the memory 71 R 2 of the input data storage unit 71 R. This data is called data BR.

Among the data stored in the input data storage unit 71 L, the data AL stored in the memory 71 L 1 is transferred to the memory 72 L 1 of the operation data storage unit 72 L. Among the data stored in the input data storage unit 71 L, the data BL stored in the memory 71 L 2 is transferred to the memory 72 R 1 of the operation data storage unit 72 R.

Among the data stored in the input data storage unit 71 R, the data AR stored in the memory 71 R 1 is transferred to the memory 72 L 2 of the operation data storage unit 72 L. Among the data stored in the input data storage unit 71 R, the data BR stored in the memory 71 R 2 is transferred to the memory 72 R 2 of the operation data storage unit 72 R.

That is, in the present embodiment, the data BL is transferred from the processing circuit 13 to the processing circuit 14 , and the data AR is transferred from the processing circuit 14 to the processing circuit 13 .

Thus, the data A (data AL, AR) read out from the pixels P from the first column to the m-th column is stored in the operation data storage unit 72 L. The operation data storage unit 72 R stores data B (data BL, BR) read out from the pixels P in the first column to the m-th column.

The pixel data stored in the operation data storage unit 72 L in this way is the data A (data AL, AR) output from the pixels P in each column. On the other hand, the pixel data stored in the operation data storage unit 72 R in this manner is the data B (data BL, BR) output from the pixels Pin each column. That is, two types of pixel data different from each other may be separately stored in the operation data storage unit 72 L and the operation data storage unit 72 R.

The data stored in the operation data storage unit 72 L is read out by the operation data input unit 73 L, subjected to operation processing by the operation unit 74 L, and output from the output unit 8 L. Similarly, the data stored in the operation data storage unit 72 R is read by the operation data input unit 73 R, subjected to operation processing by the operation unit 74 R, and output from the output unit 8 R. Since the operation unit 74 L and the operation unit 74 R are independent from each other, different operation processes may be performed simultaneously in each of the operation units 74 L and 74 R. Thus, data corresponding to two different types of pixel readout methods may be separated and output simultaneously.

The data B output from the output unit 8 L in this manner may be used as image generation data. The data A and the data B output from the output units 8 L and 8 R in this manner may be used as focus detection data.

As described above, according to the present embodiment, efficient processing of pixel data may be realized without increasing the circuit scale.

Fourth Embodiment

An imaging device according to a fourth embodiment will be described with reference to FIG. 12 to FIG. 14 . The same components as those of the imaging device according to the first to third embodiments are denoted by the same reference numerals, and the description thereof will be omitted or simplified.

In this embodiment, a layout example of each functional block on the signal processing substrate 12 is described. FIG. 12 to FIG. 14 are plan views illustrating layout examples of respective functional blocks on the signal processing substrate of the imaging device according to the present embodiment. The layout example described in this embodiment is applicable to any of the first to third embodiments.

FIG. 12 illustrates an example in which the control unit 2 L, the AD conversion unit 5 L, the horizontal scanning unit 6 L, the data processing unit 7 L, and the output unit 8 L are arranged in the processing circuit 13 , and the control unit 2 R, the AD conversion unit 5 R, the horizontal scanning unit 6 R, the data processing unit 7 R, and the output unit 8 R are arranged in the processing circuit 14 . The control unit 2 L, the AD conversion unit 5 L, the horizontal scanning unit 6 L, the data processing unit 7 L, and the output unit 8 L are arranged in this order in the vertical direction between the vertical scanning unit 3 C and the vertical scanning unit 3 L. Similarly, the control unit 2 R, the AD conversion unit 5 R, the horizontal scanning unit 6 R, the data processing unit 7 R, and the output unit 8 R are arranged in this order along the column direction between the vertical scanning unit 3 C and the vertical scanning unit 3 R.

›DESCRIPTION OF THE EMBODIMENTS · 11 of 12

In this layout, the data stored in the data processing unit 7 L and the data stored in the data processing unit 7 R may be transferred in the horizontal direction as indicated by arrows in FIG. 12 . The data output from the output units 8 L and 8 R may be output from one side of the signal processing substrate 12 along the horizontal direction.

FIG. 13 illustrates an example in which one control unit 2 L or 2 R and one data processing unit 7 L or 7 R are arranged in each of the processing circuit 13 and the processing circuit 14 , and two AD conversion units 5 L or 5 R, two horizontal scanning units 6 L or 6 R, and two output units 8 L or 8 R are arranged in each of the processing circuit 13 and the processing circuit 14 . The AD conversion unit 5 L, the horizontal scanning unit 6 L, the control unit 2 L, the data processing unit 7 L, and the output unit 8 L, the horizontal scanning unit 6 L, and the AD conversion unit 5 L are arranged in this order along the vertical direction between the vertical scanning unit 3 C and the vertical scanning unit 3 L. The output unit 8 L, the control unit 2 L, the data processing unit 7 L, and the output unit 8 L are arranged in this order along the horizontal direction. Similarly, the AD conversion unit 5 R, the horizontal scanning unit 6 R, the control unit 2 R, the data processing unit 7 R, and the output unit 8 R, the horizontal scanning unit 6 R, and the AD conversion unit 5 R are arranged in this order along the vertical direction between the vertical scanning unit 3 C and the vertical scanning unit 3 R. The output unit 8 R, the control unit 2 R, the data processing unit 7 R, and the output unit 8 R are arranged in this order along the horizontal direction.

In this layout, in each of the processing circuit 13 and the processing circuit 14 , two AD conversion units 5 are arranged at two physically separated locations, and the AD-converted data is collected in the data processing unit 7 arranged at the center of each region. The data stored in the data processing unit 7 L and the data stored in the data processing unit 7 R may be transferred in the horizontal direction as indicated by arrows in FIG. 13 . The data output from the output units 8 L and 8 R may be output from two sides along the vertical direction of the signal processing substrate 12 .

FIG. 14 illustrates an example in which one control unit 2 L or 2 R, two AD conversion units 5 L or 5 R, two horizontal scanning units 6 L or 6 R, two data processing units 7 L or 7 R, and two output units 8 L or 8 R are arranged in each of the processing circuit 13 and the processing circuit 14 . The output unit 8 L, the data processing unit 7 L, the horizontal scanning unit 6 L, the AD conversion unit 5 L, the control unit 2 L, the AD conversion unit 5 L, the horizontal scanning unit 6 L, the data processing unit 7 L, and the output unit 8 L are arranged in this order along the vertical direction between the vertical scanning unit 3 C and the vertical scanning unit 3 L. Similarly, the output unit 8 R, the data processing unit 7 R, the horizontal scanning unit 6 R, the AD conversion unit 5 R, the control unit 2 R, the AD conversion unit 5 R, the horizontal scanning unit 6 R, the data processing unit 7 R, and the output unit 8 R are arranged in this order in the vertical direction between the vertical scanning unit 3 C and the vertical scanning unit 3 R.

In this layout, the data stored in the data processing unit 7 L and the data stored in the data processing unit 7 R may be transferred in the horizontal direction as indicated by arrows in FIG. 14 . The data output from the output units 8 L and 8 R may be output from two sides along the horizontal direction of the signal processing substrate 12 .

Fifth Embodiment

An imaging system according to the fifth embodiment will be described with reference to FIG. 15 . FIG. 15 is a block diagram illustrating a schematic configuration of an imaging system according to the present embodiment.

The imaging device 10 described in the first to fourth embodiments is applicable to various imaging systems. Examples of applicable imaging systems include a digital still camera, a digital camcorder, a surveillance camera, a copier, a fax, a cellular phone, an in-vehicle camera, and an observation satellite. A camera module including an optical system such as a lens and an imaging device is also included in the imaging system. FIG. 15 illustrates a block diagram of a digital still camera as an example of them.

The imaging system 200 illustrated in FIG. 15 includes an imaging device 201 , a lens 202 that forms an optical image of an object on the imaging device 201 , an aperture 204 that varies the amount of light passing through the lens 202 , and a barrier 206 that protects the lens 202 . The lens 202 and the aperture 204 are optical systems for focusing light on the imaging device 201 . The imaging device 201 is the imaging device 10 described in any of the first to fourth embodiments, and converts the optical image formed by the lens 202 into image data.

The imaging system 200 also includes a signal processing unit 208 that processes an output signal output from the imaging device 201 . The signal processing unit 208 generates image data from the digital signal output from the imaging device 201 . The signal processing unit 208 may perform various types of correction and compression as necessary to output image data. The imaging device 201 may include an AD conversion unit that generates a digital signal to be processed by the signal processing unit 208 . The AD conversion unit may be formed on a semiconductor layer (semiconductor substrate) on which the photoelectric converter of the imaging device 201 is formed, or may be formed on a semiconductor substrate different from the semiconductor layer on which the photoelectric converter of the imaging device 201 is formed. The signal processing unit 208 may be formed on the same semiconductor substrate as the imaging device 201 .

The imaging system 200 further includes a buffer memory unit 210 that temporarily stores image data, and an external interface unit (external I/F unit) 212 that communicates with an external computer or the like. The imaging system 200 further includes a storage medium 214 such as a semiconductor memory that stores or reads out image data, and a storage medium control interface unit (storage medium control I/F unit) 216 that records or reads out image data on or from the storage medium 214 . The storage medium 214 may be built in the imaging system 200 or may be detachable.

›DESCRIPTION OF THE EMBODIMENTS · 12 of 12

The imaging system 200 further includes a general control/operation unit 218 that controls various operations and the entire digital still camera, and a timing generation unit 220 that outputs various timing signals to the imaging device 201 and the signal processing unit 208 . Here, the timing signal or the like may be input from the outside, and the imaging system 200 may include at least the imaging device 201 and the signal processing unit 208 that processes the output signal output from the imaging device 201 .

The imaging device 201 outputs an imaging signal to the signal processing unit 208 . The signal processing unit 208 performs predetermined signal processing on an imaging signal output from the imaging device 201 , and outputs image data. The signal processing unit 208 generates an image using the imaging signal.

As described above, according to the present embodiment, an imaging system to which the imaging device 10 according to the first to fourth embodiments is applied may be realized.

Sixth Embodiment

An imaging system and a movable object according to the sixth embodiment will be described with reference to FIG. 16A and FIG. 16B . FIG. 16A is a diagram illustrating a configuration of an imaging system according to the present embodiment. FIG. 16B is a diagram illustrating a configuration of a movable object according to the present embodiment.

FIG. 16A illustrates an example of an imaging system related to an in-vehicle camera. The imaging system 300 includes an imaging device 310 . The imaging device 310 is the imaging device 10 according to any one of the first to fourth embodiments. The imaging system 300 includes an image processing unit 312 that performs image processing on a plurality of image data acquired by the imaging device 310 , and a parallax acquisition unit 314 that calculates parallax (phase difference of parallax images) from the plurality of image data acquired by the imaging device 310 . The imaging system 300 also includes a distance acquisition unit 316 that calculates the distance to the object based on the calculated parallax, and a collision determination unit 318 that determines whether there is a possibility of collision based on the calculated distance. Here, the parallax acquisition unit 314 and the distance acquisition unit 316 are an example of a distance information acquisition unit that acquires distance information to the object. That is, the distance information is information related to parallax, defocus amount, distance to the object, and the like. The collision determination unit 318 may determine the possibility of collision using any of the distance information. The distance information acquisition means may be realized by hardware designed exclusively, or may be realized by a software module. It may be realized by FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated circuit), or the like, or may be realized by a combination of these.

The imaging system 300 is connected to the vehicle information acquisition device 320 , and may acquire vehicle information such as a vehicle speed, a yaw rate, and a steering angle. Further, the imaging system 300 is connected to a control ECU 330 , which is a control device that outputs a control signal for generating braking force to the vehicle based on the determination result obtained by the collision determination unit 318 . The imaging system 300 is also connected to an alert device 340 that issues an alert to the driver based on the determination result obtained by the collision determination unit 318 . For example, when the collision possibility is high as the determination result of the collision determination unit 318 , the control ECU 330 performs vehicle control to avoid collision and reduce damage by applying a brake, returning an accelerator, suppressing engine output, or the like. The alert device 340 sounds an alarm such as a sound, displays alert information on a screen of a car navigation system or the like, and provides a warning to the user by applying vibration to a seatbelt or steering.

In the present embodiment, an image of the periphery of the vehicle, for example, the front or the rear is captured by the imaging system 300 . FIG. 16B illustrates an imaging system in the case of capturing an image of the front of the vehicle (imaging range 350 ). The vehicle information acquisition device 320 sends an instruction to the imaging system 300 or the imaging device 310 . With such a configuration, the accuracy of distance measurement may be further improved.

Although an example in which the vehicle is controlled so as not to collide with another vehicle has been described above, the present disclosure is also applicable to a control in which the vehicle is automatically driven following another vehicle, a control in which the vehicle is automatically driven so as not to protrude from a lane, and the like. Further, the imaging system may be applied not only to a vehicle such as a host vehicle, but also to a movable object (mobile device) such as a ship, an aircraft, or an industrial robot. In addition, the present invention can be applied not only to a movable object but also to an apparatus using object recognition in a wide range such as an advanced road traffic system (ITS).

›MODIFIED EMBODIMENTS

The present invention is not limited to the above embodiments, and various modifications are possible.

For example, an example in which a configuration of a part of any embodiment is added to another embodiment or an example in which a configuration of a part of another embodiment is substituted is also an embodiment of the present disclosure.

Further, in the embodiments described above, an application example to a stacked imaging device in which a pixel substrate and a signal processing substrate are stacked is described, but the present disclosure is not necessarily limited to a stacked imaging device, and the present disclosure may be widely applied to an imaging device in which a signal processing circuit is divided into a plurality of regions.

In the above-described embodiment, the signal processing circuit is divided into two areas having different columns, and data is exchanged between these areas, but the manner in which the signal processing circuit is divided is not limited to this. For example, the signal processing circuit may be divided into three or more regions different in corresponding columns, and data may be exchanged between these regions. In addition to the horizontal division, the signal processing circuit may be divided into two or more regions in which corresponding rows are made different.

Although the first to fourth embodiments exemplify a device for the purpose of acquiring an image, that is, an imaging device, an application example of the present disclosure is not necessarily limited to the imaging device. For example, in the case where the present invention is applied to an apparatus mainly targeted for distance measurement as described in the sixth embodiment, it is not always necessary to output an image. In such a case, the device may be said to be a photoelectric conversion device that converts optical information into a predetermined electric signal. The imaging device is one of photoelectric conversion devices.

The imaging systems illustrated in the fifth and sixth embodiments are examples of imaging systems to which the imaging device of the present disclosure may be applied, and the imaging systems to which the imaging device of the present disclosure may be applied are not limited to the configurations illustrated in FIG. 15 and FIG. 16A .

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2020-152620, filed Sep. 11, 2020, which is hereby incorporated by reference herein in its entirety.

Claims as granted

14 claims

Classifications

1 codes

IPC · International Patent Classification

Section H — Electricity

H04N25/78

Claim changes

Soon

Coming soonHow the claims changed between publication and grant

See which claims were amended, added or cancelled during examination, with every added and removed word marked.

AmendedAddedCancelledUnchanged

The published claims of this application are not paired with the granted ones in what we hold.

File wrapper

⤢ drag to zoom

USPTOApplicanthover for detail · click to open

Pendency

1.2 y

424 days filing → grant

Office actions

none on record

Examiner

Mark T Monk

art unit 2696 · TC 2600

Law firm

Law firm · Log in to unlock

Citations: 30 back · 4 forward

See the full prosecution history — every USPTO and applicant action on this file, in order.

Documents

Log in to open the documents of this file: the application as filed, every office action and response, the notice of allowance.

Chain of title

⤢ drag to zoom

Titlehover for detail · click to open

See the full assignment history — every owner this patent has passed through, with recordation dates and reel/frame numbers.