Arbitrated access to resources among multiple devices

In certain data processing applications, multiple devices, such as processor cores and input/output interfaces, may require access to shared resources such as a memory. A resource arbiter can be implemented to arbitrate access to the shared resources. Typically, the arbiter selects a next device to allow to forward a resource request based on a predetermined routine, such as a round-robin selection among all of the devices.

Example embodiments of the present disclosure include a circuit configured to manage and enforce arbitration and quality of service (QOS) among multiple devices accessing a resource, such as a memory. The circuit can include a device interface for receiving requests from a number of devices to access a memory, where each of the requests including a device identifier (ID) identifying the device. A bridge interface forwards the requests to a bridge, where they may be sent to the memory. A per-device counter maintains a resource count allocated to each device, and a bus counter maintains a resource count for forwarding the requests to the bridge via the bus. Further, an arbiter circuit operates to select a next one of the requests to forward to the bridge based on 1) the device identifier of the request, 2) a count of the per-device counter corresponding to the device identifier, and 3) a count of the bus counter.

In further embodiments, the per-device counter may limit a maximum resource count to a first of the plurality of devices relative to a second of the plurality of devices. The arbiter circuit may suspend requests having a given device identifier in response to detecting that the per-device counter for the given device identifier indicates a count that is less than a threshold value. The arbiter circuit may also suspend requests in response to detecting that the bus counter indicates a count that is less than a threshold value. For a given request, the arbiter circuit may suspend the request based on a) an indication of a resource count required by the request, and b) a predetermined threshold for per-request resources.

In still further embodiments, a number of buffers (e.g., FIFOs) may store the requests received from a respective device at the device interface. In response to reaching a threshold storage value, the buffers may generate a backpressure command to limit receipt of further requests to the circuit. The per-device counter may increment in response to receiving an indication that an additional resource associated with a given device identifier is available. Likewise, the bus counter may increment in response to receiving an indication that an additional resource associated with the circuit is available.

In yet further embodiments, a bridge control circuit may operate to control a count of the bus counter. In particular, the bridge control circuit may prevent the return of an initial given count of resources to the bus counter.

The foregoing will be apparent from the following more particular description of example embodiments of the disclosure, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present disclosure.

FIG. 1 is a block diagram illustrating a data processor in which embodiments of the present disclosure may be implemented.

FIG. 2 is a block diagram illustrating a system implementing the data processor of FIG. 1 .

FIG. 3 is a block diagram illustrating an arbiter and bridge connecting a plurality of devices and a memory in one embodiment.

FIGS. 4A-B illustrate example register structures.

FIG. 5 is a flow chart illustrating arbitration of requests in one embodiment.

A description of example embodiments follows.

FIG. 1 is a block diagram illustrating a data processor 100 in an example embodiment. The processor 100 may be implemented as a system-on-chip (SOC) and connected to external devices, resources and communications channels via a printed circuit board (PCB). Alternatively, the processor 100 may be implemented among a number of discrete circuit components connected via a PCB, or may comprise a SOC in combination with one or more discrete circuit components.

The data processor 100 facilitates operations between a number of devices and resources, and arbitrates access to shared resources among the devices. In particular, the processor cores 150 may include one or more data processor cores. In an example embodiment, the processor cores 150 may include a number (e.g., 48) of ARM® processor cores, such as the ARMv8 processor cores. The processor cores 150 are connected, via a coherent memory interconnect (CMI) 135 , to provide shared access to a number of other devices and resources, including the level-2 cache (L2C) and controller 160 (also referred to herein as “L2C”). The L2C further connects to a memory controller 165 for performing memory access operations to an external memory, such as a double data rate synchronous dynamic random-access memory (DDR SDRAM) array. Such a memory (not shown) may alternatively be located on-chip with the data processor 100 . The CMI 135 may also connect to a coherent processor interconnect (CPI) 155 for communication with off-chip devices, such as an additional data processor. An example of one such configuration is described below with reference to FIG. 2 .

The CMI 135 is further connected to an input/output bridge (IOBN) 110 , which provides an interconnect between the processor cores 150 , CPI 155 and L2C 160 and additional devices and resources. In particular, devices 145 A-F connect to the IOBN 110 via input/output interconnects (IOI), IOI 0 155 A and IOI 1 155 B, which may be non-coherent buses (NCBs) including passive and/or arbitrated channels. The devices 145 A-F may include a number of different on-chip devices, such as co-processors, and may include I/O interfaces (e.g., USB, SATA, PCIe, Ethernet) to connect to a number of external or off-chip devices and interfaces. In order to arbitrate resources at the IOBN 110 to the devices 145 A-F, NCB arbiters 140 A-B receive requests from the devices 145 A-F and selectively grant IOBN resources to the devices 145 A-B. Once granted, the devices 145 A-B may communicate with the processor cores 150 , perform a memory access operation to the L2C 160 , or access other components of the data processor 100 .

In order to facilitate shared memory access among several different devices (e.g., the processor cores 150 and devices 145 A-F), the data processor 100 may employ virtualization, whereby a memory interconnect (e.g., CMI 135 and IOBN 110 ) interfaces with the devices using virtual addresses, which are translated to a physical address of the memory. To enable virtualization, a System Memory Management Unit (SMMU) 180 maintains an index of physical and virtual addresses. During a memory access operation where a virtual address is provided, the IOBN 110 forwards the virtual address to the SMMU 180 , which returns a corresponding physical address for accessing the memory (e.g., the L2C 160 or an external memory via the L2C 160 ). The IOBN 110 may translate addresses bi-directionally such that the virtual address is maintained at communications at the device, and the physical address is indicated in operations at the memory. The SMMU 180 may be further configured to support multiple tiers of virtual addresses.

Control status registers (CSRs) 170 include registers for maintaining information about the instructions and operations of the data processor 100 . The CSRs may maintain, for example, status information regarding a number of devices, as well as information about ongoing operations and instructions between devices and/or resources. Devices such as the processor cores 150 and the devices 145 A-B, as well as other requestors 185 and off-chip devices (via the CPI 155 ), may write to and read from the CSRs 170 using a register master logic (RML). To facilitate the multiple requests from several different devices, a master RML (MRML) 120 operates to arbitrate access to the CSRs 170 .

FIG. 2 is a block diagram illustrating a system 200 implementing a plurality of data processors. The system 200 includes data processors 205 A-B, each of which may be configured comparably to the data processor 100 described above with reference to FIG. 1 . The data processors 205 A-B may be linked by a CPI interconnect 255 , which may connect to a respective CPI interface (e.g., 155 in FIG. 1 ) of each data processor 205 A-B. The CPI interconnect 255 may provide shared access to the devices and resources across the data processors 201 A-B. In further embodiments, additional data processors or other devices may be linked to the data processors 205 A-B via the CPI interconnect 255 .

The data processors 205 A-B may be connected to respective memory arrays (e.g., DDR SDRAM) 215 A-B as shown, and/or may be connected to a common memory array. The data processors may be further connected to a number of external devices 245 via a number of devices via respective I/O interfaces (e.g., USB, SATA, PCIe, Ethernet).

Turning back to FIG. 1 , in some embodiments, multiple devices, such as the devices 145 A-F and processor cores 150 , may require access to shared resources such as the L2C 160 . For the devices 145 A-F, the NCB arbiters 140 A-B can be implemented to arbitrate access to the L2C 160 and/or other shared resources. The arbiter 140 A, for example, grants resources to each of the devices 145 A-C for forwarding request to the IOBN 110 via the IOI 0 155 A. To do so, the arbiter 140 A may select a next device to allow to forward a resource request based on a predetermined routine, such as a round-robin selection among all of the devices 145 A-C.

However, each of the devices 145 A-F may utilize a different number of IOBN 110 resources, or may require greater priority over other devices. Further, devices on one bus (e.g., bus IOI 155 A) may utilize a different number of resources, or require greater priority of resources, relative to devices on another bus (e.g., bus IOI 155 B) As a result, without proper arbitration, some devices may receive an insufficient resources to forward requests to the IOBN 110 , while other devices may utilize more resources than necessary.

NCB arbiters 140 A-B, in one embodiment, may be configured to manage and enforce arbitration and quality of service (QOS) to the L2C 160 or other resources by a number of devices. The arbiters 140 A-B receive requests from a number of devices 145 A-F to use resources of a bridge IOBN 110 connecting to the L2C, and maintain a count of bridge resources available on a per-device and per-bus basis. The arbiters 140 A-B operate to select a next one of the requests to grant a bridge resource based on the device originating the request, a count of the per-device resources available, and a count of the resources available to the bus connecting the device to the IOBN 110 .

An example NCB arbiter 140 A configured to provide the aforementioned functions is described below with reference to FIG. 3 .

FIG. 3 is a block diagram illustrating a processing subsystem 300 including a NCB arbiter 140 A and an IOBN 110 connecting a plurality of devices 145 A-C and a memory (L2C 160 ) in one embodiment. The subsystem 300 may include one or more components of the data processor 100 described above with reference to FIG. 1 , or may be incorporated into the data processor. For example, the subsystem 300 may also include processor cores 150 and process access requests from the processor cores 150 as well as from the devices 145 A-C. Further, a second NCB arbiter 140 B may be configured in a manner comparable to the NCB arbiter 140 A as described below.

The devices 145 A-C may forward memory access requests to the L2C 160 via the IOBN 110 , for example to read or write to the L2C 160 . The IOBN 110 may have a limited number of resources (e.g., available slots in a time-division multiplexing (TDM) channel at the CMI 135 ) available for handling such requests. The NCB arbiter 140 A operates to arbitrate these resources among the devices 145 A-C. To do so, the NCB arbiter 140 A may receive requests for IOBN resources from each of the devices 145 A-C via a respective buffer (e.g., FIFO) 355 A-C. An arbiter select circuit 320 may select a next request to grant a resource based on a number of conditions. In particular, the select circuit 320 may reference a device credit counter 375 A and/or a bus credit counter 375 B.

The device credit counter 375 A may maintain a number of per-device counters indicating a count of IOBN resources available to a respective device. The count of IOBN resources may be equivalent to a maximum (or initial) number of resources allocated to a given device, minus a number of in-flight requests associated with the device. To maintain this count, the device credit counter 375 A may decrement a given count when a resource is granted to a corresponding device, and may increment a given count upon receiving an indication from the IOBN 110 that a corresponding request has been completed (e.g., cleared from the IOBN 110 or fulfilled by the L2C 160 ). The register 340 may store programmable values to be used by the device credit counter 375 A as the maximum or initial count values for each device. As a result, the NCB arbiter 140 A may configure a specific number of resources to each device 145 A-C, which may ensure that devices receive a proper number of resources based on priority or other factors, and that the device requests from one device do not prevent another device from receiving a minimum number of resources. Alternatively, the register 340 may be located at the IOBN 110 or another location such as the control and status registers (CSRs) 170 shown in FIG. 1 .

The bus credit counter 375 B may maintain a counter indicating a count of IOBN resources available to all of the devices 145 A-C connecting to the IOBN 110 via a common bus (IOI 0 155 A). The count of IOBN resources may be equivalent to a maximum (or initial) number of resources allocated to the IOI 0 155 A, minus a number of in-flight requests associated with the IOI 0 155 A. To maintain this count, the bus credit counter 375 B may decrement a given count when a resource is granted to device on IOI 0 155 A, and may increment a given count upon receiving an indication from the IOBN 110 that a corresponding request has been completed (e.g., cleared from the IOBN 110 or fulfilled by the L2C 160 ). The register 345 at IOBN 110 may store programmable values to be used by the bus credit counter 375 B as the maximum or initial count values for each device. As a result, the IOBN 110 may configure a specific number of resources allocated to all devices 145 A-C connected to a common bus IOI 0 155 A, which may differ from a number of resources allocated to devices on another bus (e.g., devices 145 D-F on IOI 1 as shown in FIG. 1 ). Such a configuration can provide “bus balancing” among a number of buses, whereby each bus receives a proper number of resources based on priority or other factors, and that the device requests from one bus do not prevent devices at another bus from receiving a minimum number of resources. Alternatively, the register 345 may be located at the NCB arbiter 140 A or another location such as the control and status registers (CSRs) 170 shown in FIG. 1 .

To provide further bus balancing, the IOBN 110 may be configured to refrain from returning an initial number of credits to the bus credit counter 375 B. For example, for the first given number (e.g., 8) of credits used by the devices 145 A-C on a given bus IOI 0 155 A, the IOBN 110 may refrain from returning those credits to the bus credit counter 375 B, while returning all credits following the initial number of credits. This configuration may be applied to some or all of the buses (e.g., IOI 1 155 B). A similar configuration may be applied to the per-device counts for one or more devices 145 A-C.

FIGS. 4A-B illustrate example register structures. As shown in FIG. 4A , an NCB arbiter register 405 (e.g., register 340 in FIG. 3 ) may maintain a distinct credit value for each device. The credit value may be used to set an initial or maximum value at a respective counter of a device credit counter (e.g., counter 375 A). As shown in FIG. 4B , an IOBN register 410 (e.g., register 345 ) may maintain a distinct credit value for each bus (e.g., IOI 0 , IOI 1 155 A-B). The credit value may be used to set an initial or maximum value at a respective counter of a bus credit counter (e.g., counter 375 B).

FIG. 5 is a flow chart illustrating a process 500 arbitrating requests in one embodiment. With reference to FIG. 3 , upon initialization, the NCB arbiter 140 A may set initial credit counts at the device credit counter 375 A and the bus credit counter 375 B based on values at the registers 340 , 345 , respectively ( 505 ). For each request for IOBN resources received at buffers 355 A-C, the NCB arbiter 140 A may parse the request to obtain a corresponding device identifier (ID) ( 510 ). To update the credit counts, the NCB arbiter 140 A may periodically or continuously check for returned credits from the IOBN 110 ( 520 ). If credits are received, then the respective counter(s) are incremented accordingly ( 525 ).

To determine a next request to grant, the NCB arbiter 140 A may first select a next request candidate according to a selection routine (e.g., a round-robin selection) ( 530 ). For the selected candidate, the NCB arbiter 140 A may look up an associated device credit count (at device credit counters 375 A) based on the device ID of the candidate ( 540 ), and may look up an associated bus credit count (at device credit counter 375 B) for the bus (e.g., IOI 155 A) connected to the device ( 550 ). If one of the device credit count or bus credits count does not have a sufficient or threshold number of credits, then the NCB arbiter 140 A may suspend the request an proceed to select a next request candidate ( 530 ). If, however, both the device credit count and the bus credit count have a sufficient number of credits, the NCB arbiter 140 A may grant a resource to the request and decrement the counters associated with the request accordingly ( 560 ). The NCB arbiter may then proceed to repeat the operations of updating the credit counts ( 520 ) and selecting a next request candidate to grant resources ( 530 ).

While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.