Sharding over multi-link data channels

Computing systems and associated networks have greatly revolutionized our world ushering in what is now commonly called the “information age”. Efficient and effective storage of data is thus essential to reading, writing, persisting, and evaluating data. Often a file simply cannot fit onto a single storage device. In that case, the file is sharded (i.e., divided into parts) with each part (or shard) stored on a different storage device. This sharding process also helps to have parallel access to different shards of the file. Thus, fast data transfer enabled by parallel access is also a significant benefit of file sharding. Previous sharding technologies tend to address relatively uniform capacity constraints, such as throughput. For instance, a file might be sharded across similar hard drives. In high performance computing, a file might be sharded across specially designed networks in which throughput is designed to be uniform, regardless of the storage node.

As cloud computing services have developed, so too have storage services. A customer that establishes an account with a storage service may be guaranteed by a customer service agreement to have access to certain quantities of storage and/or certain I/O bandwidths. Of course, storage services hide from view the physical devices that are used to provide the storage on an account. Originally, such storage services were designed to simply store user data, such as documents, photographs, videos, music, and so forth.

The cloud is also able to access massive quantities of processing resources. Thus, complex data mining and analytics technologies have been developed. Such data mining or analytics often use as input massive quantities of data. Accordingly, storage services have become increasingly used as storage for massive amounts of raw data upon which analysis or data mining is performed.

The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced.

At least some embodiments described herein relate to a method for sharding a computing data unit (such as a file) across multiple storage locations (e.g., data centers, clusters, accounts, and so forth). For each of multiple storage locations, the sharding system identifies multi-link traffic constraints in data channeling with the storage location. For instance, for a given data channel leading to and/or from a storage location, there may be multiple links. Each link may have its own constraints (e.g., in throughput and/or storage). A link may be a communication link or may even be a communication endpoint at which storage occurs.

For instance, some links may have constraints that are imposed by physical limits. As an example, some network channels are only capable of transmitting or receiving at certain bit rates. Some links may be limited to logical constraints, such as when policy is enforced on a link. This logical constraint might occur if an application or customer is to share the available physically possible storage capabilities (e.g., storage size and/or throughput) with multiple other applications and/or customers. In that case, to ensure that one application or customer does not impact another, policy enforces sufficient sharing so as to maximize the technical benefit that each application and/or customer receives, and/or to honor technical guarantees provided to the application and/or customer.

Policy may consider the sharding of files in a manner that honors the constraints, thereby avoiding the technical problems (e.g., network inefficiency, impact on other applications, etc.) that can be caused by a service throttling I/O if I/O exceeds certain levels. The policy may also be set so that all portions of the shards may be timely accessed in parallel. When sharding, the policy is applied in order to identify initial placement of the computing data unit across the multiple storage locations. Furthermore, the policy may be consulted to determine which storage location (in the case of multiple copies of the shard existing) to access the shard from and/or which channel to use to access the shard from a given location. Thus, efficient constraint-aware sharding of a computing data unit may be accomplished. Furthermore, the shards may be accessed in a manner that is likewise optimized towards efficient access given the constraints at access time.

The constraints that exist on any given one of the links in the multi-link channel may change over time, perhaps even rapidly. For instance, a link may be inoperative at any moment. The sharding system may keep track of these dynamically changing constraints, thereby responding appropriately to change in data channels and constraints of data links that may be used to access storage locations.

Thus, the sharding system may operate to shard storage locations that do not have uniform constraints, and are outside the control of the sharding system. For instance, the sharding system may shard to storage services. Even so, the sharding system honors constraints, thereby avoiding or mitigating technical penalties (e.g., reduced access speed, adverse impact on other applications, and so forth) that can sometimes apply when attempting to operate at too close to, or exceeding, those constraints.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

In order to describe the manner in which the above-recited and other advantages and features of the invention can be obtained, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:

FIG. 1 illustrates an example computer system in which the principles described herein may be employed;

FIG. 2 illustrates an environment in which a computing data unit is sharded and the various shards are stored at various storage locations in accordance with a sharding policy;

FIG. 3 illustrates a flowchart of a method for sharding a computing data unit across multiple storage locations;

FIG. 4A through 4C illustrate multiple examples of multi-link data channels that may be used to communicate shards to and from various storage locations; and

FIG. 5 illustrates a flowchart of a method for accessing at least one shard of a sharded computing data unit.

At least some embodiments described herein relate to a method for sharding a computing data unit (such as a file) across multiple storage locations (e.g., data centers, clusters, accounts, and so forth). Such storage locations may be local and/or remote from each other. For each of multiple storage locations, the sharding system identifies multi-link traffic constraints in data channeling with the storage location. For instance, for a given data channel leading to and/or from a storage location, there may be multiple links. Each link may have its own traffic constraints (e.g., in the form of a throughput constraint and/or a storage constraint). A link may be a communication link or may even be a communication endpoint at which storage occurs.

For instance, some links may have constraints that are imposed by physical limits. As an example, some network channels are only capable of transmitting or receiving at certain bit rates, and thus this represents a hard limit of the throughput. As another example, a storage device may have a hard limit on the amount of storage that it can physically store.

Some links may be limited by logical constraints (or soft limits), such as when sharing policy is enforced on a link. This logical constraint might occur if an application or customer is to share the available physically possible storage capabilities (e.g., storage size and/or throughput) with multiple other applications and/or customers. In that case, to ensure that one application or customer does not impact another, sharing policy enforces sufficient sharing so as to maximize the technical benefit that each application and/or customer receives, and/or to honor technical guarantees provided to the application and/or customer.

In some cases, if the application or consumer does not honor a logical throughput constraint, that application's or consumer's throughput may be throttled downward. This throttling may seem punitive, as enforcing sharing policy might take some time to allow the experienced throughput to recover even close to the logical throughput constraint. The application might deduce such a logical throughput constraint by monitoring over time when throttling occurs.

Policy may consider the sharding of files (e.g., the number of shards to segment the file into and/or the size of all or each shard) in a manner that honors the constraints, thereby avoiding the technical problems (e.g., network inefficiency, impact on other applications, punitive throttling) that can be caused by a service throttling I/O if I/O exceeds certain levels. The policy may also be set so that all portions of the shards may be timely accessed in parallel, thereby roughly evening out the time required to write (and perhaps subsequently read) the file as a whole.

When sharding, the policy is applied in order to identify initial placement of the computing data unit across the multiple storage locations. Furthermore, the policy may be consulted to determine which storage location (in the case of multiple copies of the shard existing) to access the shard from and/or which channel to use to access the shard from a given location. Multiple copies of a shard may exist (e.g., a copy of a shard may initially be placed in multiple locations). For instance, a shard can be replicated to another storage location. In some cases, after initial placement, the shard itself may be further sharded to multiple other storage locations. The reading operation may be aware of the location of the further sharded shards (or “subshards”), and consider their size, and the traffic constraints to those other storage locations when determining whether to access those subshards. Thus, efficient constraint-aware sharding of a computing data unit may be accomplished. Furthermore, the shards may be accessed in a manner that is likewise optimized towards efficient access given the constraints at access time.

The constraints that exist on any given one of the links in the multi-link channel may change over time, perhaps even rapidly. For instance, a link may be inoperative at any moment. The sharding system may keep track of these dynamically changing constraints, thereby responding appropriately to changes in data channels and constraints of data links that may be used to access storage locations.

Thus, the sharding system may operate to shard to storage locations that do not have uniform constraints, and are outside the control of the sharding system. For instance, the sharding system may shard to storage services. Even so, the sharding system honors constraints, thereby avoiding or mitigating technical penalties (e.g., reduced access speed, adverse impact on other applications, and so forth) that can sometimes apply when attempting to operate at too close to, or exceeding, those constraints.

Because the principles described herein operate in the context of a computing system, a computing system will be described with respect to FIG. 1 . Then, the principles of sharding while honoring constraints in complex data channels (e.g., multi-link data channels) will then be described with respect to FIGS. 2 through 5 .

Computing systems are now increasingly taking a wide variety of forms. Computing systems may, for example, be handheld devices, appliances, laptop computers, desktop computers, mainframes, distributed computing systems, datacenters, or even devices that have not conventionally been considered a computing system, such as wearables (e.g., glasses, watches, bands, and so forth). In this description and in the claims, the term “computing system” is defined broadly as including any device or system (or combination thereof) that includes at least one physical and tangible processor, and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by a processor. The memory may take any form and may depend on the nature and form of the computing system. A computing system may be distributed over a network environment and may include multiple constituent computing systems.

As illustrated in FIG. 1 , in its most basic configuration, a computing system 100 typically includes at least one hardware processing unit 102 and memory 104 . The memory 104 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term “memory” may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computing system is distributed, the processing, memory and/or storage capability may be distributed as well.

The computing system 100 has thereon multiple structures often referred to as an “executable component”. For instance, the memory 104 of the computing system 100 is illustrated as including executable component 106 . The term “executable component” is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods that may be executed on the computing system, whether such an executable component exists in the heap of a computing system, or whether the executable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such structure may be computer-readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term “executable component”.

The term “executable component” is also well understood by one of ordinary skill as including structures that are implemented exclusively or near-exclusively in hardware, such as within a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other specialized circuit. Accordingly, the term “executable component” is a term for a structure that is well understood by those of ordinary skill in the art of computing, whether implemented in software, hardware, or a combination. In this description, the term “component” may also be used. As used in this description and in the case, this term (regardless of whether the term is modified with one or more modifiers) is also intended to be synonymous with the term “executable component” or be specific types of such an “executable component”, and thus also have a structure that is well understood by those of ordinary skill in the art of computing.

In the description that follows, embodiments are described with reference to acts that are performed by one or more computing systems. If such acts are implemented in software, one or more processors (of the associated computing system that performs the act) direct the operation of the computing system in response to having executed computer-executable instructions that constitute an executable component. For example, such computer-executable instructions may be embodied on one or more computer-readable media that form a computer program product. An example of such an operation involves the manipulation of data.

The computer-executable instructions (and the manipulated data) may be stored in the memory 104 of the computing system 100 . Computing system 100 may also contain communication channels 108 that allow the computing system 100 to communicate with other computing systems over, for example, network 110 .

While not all computing systems require a user interface, in some embodiments, the computing system 100 includes a user interface 112 for use in interfacing with a user. The user interface 112 may include output mechanisms 112 A as well as input mechanisms 112 B. The principles described herein are not limited to the precise output mechanisms 112 A or input mechanisms 112 B as such will depend on the nature of the device. However, output mechanisms 112 A might include, for instance, speakers, displays, tactile output, holograms, virtual reality, and so forth. Examples of input mechanisms 112 B might include, for instance, microphones, touchscreens, holograms, virtual reality, cameras, keyboards, mouse of other pointer input, sensors of any type, and so forth.

Embodiments described herein may comprise or utilize a special purpose or general-purpose computing system including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computing system. Computer-readable media that store computer-executable instructions are physical storage media. Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments can comprise at least two distinctly different kinds of computer-readable media: storage media and transmission media.

Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other physical and tangible storage medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computing system.

A “network” is defined as one or more data links that enable the transport of electronic data between computing systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computing system, the computing system properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computing system. Combinations of the above should also be included within the scope of computer-readable media.

Further, upon reaching various computing system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computing system RAM and/or to less volatile storage media at a computing system. Thus, it should be understood that readable media can be included in computing system components that also (or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general purpose computing system, special purpose computing system, or special purpose processing device to perform a certain function or group of functions. Alternatively, or in addition, the computer-executable instructions may configure the computing system to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code.

Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computing system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, datacenters, wearables (such as glasses or watches) and the like. The invention may also be practiced in distributed system environments where local and remote computing systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

Those skilled in the art will also appreciate that the invention may be practiced in a cloud computing environment. Cloud computing environments may be distributed, although this is not required. When distributed, cloud computing environments may be distributed internationally within an organization and/or have components possessed across multiple organizations. In this description and the following claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any of the other numerous advantages that can be obtained from such a model when properly deployed.

For instance, cloud computing is currently employed in the marketplace so as to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. Furthermore, the shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.

A cloud computing model can be composed of various characteristics such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model may also come in the form of various service models such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). The cloud computing model may also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud computing environment” is an environment in which cloud computing is employed.

FIG. 2 illustrates an environment 200 in which sharding of a computing data unit occurs. The computing data unit 201 is a unit of data that is identified by a computing system. For instance, a computing data unit 201 might be a file, in which case, the file is recognized by a file system of a computing system (such as the computing system 100 of FIG. 1 ). The computing data unit 201 might be a table in which case the table is recognized by a database application running on the computing system. Alternatively, the computing data unit may be a collection of other computing data units. For instance, the computing data unit might be a folder containing one or more files or subfolders. The computing data unit might also be a collection of related tables, or perhaps even an entire relational database. The computing data unit might also be unstructured data that is identified as a single unit to the computing system (such as is often the case in big data systems).

The computing data unit 201 is provided to a sharding component 202 , which operates to perform sharding in accordance with the principles described herein. This is all represented by the multi-headed arrow 221 of FIG. 2 . For instance, sharding of the computing data unit 201 might be performed in accordance with the method 300 of FIG. 3 , which illustrates a flowchart of a method 300 for sharding a computing data unit across multiple storage locations. The sharding component 202 might be, for instance, an example of the executable component 106 of FIG. 1 , when run on the computing system 100 .

The sharding component 202 shards the computing data unit 201 into shards which are referred to collectively as “shards 210 ”. In the illustrated embodiment, the computing data unit 201 is split into shards 210 A through 210 D as also represented by the arrow 221 . The ellipsis 210 E represents that the computing data unit 201 may be split into any number of shards. The shards may each be the same size or they may be different sizes. In accordance with the principles described herein, the computing data unit 201 is sharded in accordance with a policy 203 .

FIG. 2 also illustrates storage locations 211 that may be used in order to store the various shards. In the illustrated embodiment, there are six storage locations 211 A through 211 F illustrated and that may be selected to store a shard of the computing data unit. The ellipses 211 G represents that the principles described herein operate regardless of the number of possible storage locations to which the shards may be stored. Each of the storage locations 211 has a corresponding data channel 212 over which the shard may be communicated. For instance, data channels 212 A through 212 F may be used to read the shard from and/or write the shard to respective storage locations 211 A through 211 F. Again, the ellipses 212 G represents that that there may be as many data channels as there are storage locations.

The storage locations 211 are illustrated as having different shapes, which symbolizes that the nature of the storage locations 211 may be different. For instance, the storage locations might be a logical storage location (such as a storage account on a storage service), or may be a physical storage location (such as a datacenter, or portion thereof, such as a storage node or cluster of storage nodes). For instance, storage locations 211 A and 211 B are illustrated as circles. Perhaps this represents that the storage locations 211 A and 211 B are storage accounts in a storage service of a cloud computing environment. The storage location 211 C is illustrated as being a triangle, perhaps representing that this storage location 211 C is a different kind of storage location (such as perhaps a Storage Area Network). The storage locations 211 D, 211 E, and 211 F, are octagonal, representing perhaps yet another type of storage location such as a datacenter.

FIG. 3 illustrates a flowchart of a method 300 for sharding a computing data unit across multiple storage locations. The method 300 may be performed by, for instance, the sharding component 202 of FIG. 2 , in the environment 200 of FIG. 2 . Accordingly, the method 300 of FIG. 3 will be described with frequent reference to the environment 200 of FIG. 2 .

For each of multiple storage locations, multi-link traffic constraints are identified in data channeling with the storage location (act 301 ). For instance, in FIG. 2 , each of the data channels 212 may have multiple links, each link having its own traffic constraints. Thus, while the data channels 212 are symbolically represented as simple tubes, they each represent complex data channels in which there are multiple links. In some embodiments, there may be an opportunity to create new storage locations. For instance, the sharding component may create new storage accounts at a storage service. In that case, each newly created storage account may be considered as a storage location to which shards may be stored. Thus, when considering the traffic constraints of each storage location, the traffic constraints of existing, and potentially new, storage locations may be considered.

FIGS. 4A through 4C illustrate various examples of multi-link data channels. For instance, FIG. 4A illustrates two serial links 411 A and 412 A. Link 411 A connects node 401 A to node 402 A. Link 412 A connects node 402 A and node 403 A. An example of a constraint that might be applied to each link includes a throughput constraint in terms of bits per second. For instance, link 411 A might have a higher throughput constraint than link 412 A. This might mean that the multi-link channel 400 A may be considered to have a constraint equal to its lowest bandwidth link (i.e., the throughput constraint of link 412 A).

The link constraints might alternatively be different in forms. For instance, link 411 A might include a throughput constraint, whereas link 412 A might include a storage capacity constraint (also referred to herein as a “storage constraint”). In that case, the multi-link constraint may be a compound constraint having multiple constraint dimensions including bits per second (for throughput constraints) and/or bytes (for storage constraints). The links might be logical or physical. For instance, link 411 A might be a network link that is dedicated for use by the sharding component 202 , in which case the throughput might be limited only by the physical capacity of the network itself, thus being a hard throughput constraint. On the other hand, the link 411 A might include a throttling constraint in which the sharing policy of a service limits the utilization of the link 411 A to a specified threshold, and throttles attempts to use more than the specified threshold. As another example of a logical constraint, the link 412 A might be a storage service account, in which case perhaps the storage capacity (and/or the bandwidth) is artificially constrained so as to prevent collision with storage of other tenants of the service.

FIGS. 4B and 4C simply illustrate that the multi-link channels may take any of a variety of forms. For instance, FIG. 4B shows a multi-link channel in which there is a link 411 B that couples the nodes 401 B and 402 B, a link 412 B that couples the nodes 401 B and 403 B, a link 413 B that couples the nodes 402 B and 404 B, and a link 414 B that couples the nodes 403 B and 404 B. Here, there are multiple constituent channels that lead to the same storage location 404 B. These constituent channels might each be selectable for storage to the storage location 404 B. In that case, when selecting to store a shard over the data channel 400 B, the policy may also be used to select the data channels to use to store a shard to the selected storage location. On the other hand, the channel may use each of the two channels to write and read the shard in parallel, for higher throughputs. FIG. 4C illustrates a series of three links 411 C, 412 C and 413 C that lead from a node 401 C to a node 404 C (via nodes 402 C and 403 C).

The constraint of any given link in a multi-link data channel may change over time, and perhaps frequently change. For instance, the availability of some links may be quite transient, and depend on a variety of dynamically changing factors, such as the utilization of the link by other applications, weather conditions, maintenance, and so forth. The act 301 encompasses tracking any dynamically changing constraints as circumstances change. Furthermore, the act 301 may also encompass historical constraints (such as the average of a particular constraint).

The constraints of the multi-link data channel (e.g., the current constraint and/or the historical constraint) are then compared with the sharding policy to determine sharding (e.g., shard size and/or number of shards) and initial placement of shards of the computing data unit across multiple storage locations (act 302 ). The sharding policy honors traffic constraints in data channeling with multiple storage locations. As an example, suppose that a first multi-link data channel has no constraints, but has a bandwidth that averages twice that over a second multi-link data channel that has a storage constraint of one Terabyte (TB). In that case, a shard that is two TB might be stored over the first multi-link data channel, and a shard that is one TB might be stored over the second multi-link data channel. Both shards would then be written in parallel in about the same amount of time. As apparent from this description, the policy may be used to determine how the computing data unit is to be sharded, as well as where those shards are to be stored. When determining initial placement, a single shard may be replicated in multiple storage locations, although not required.

The shards are then created according to the determined sharding (act 303 ) and caused to be stored across the multiple locations in accordance with the determined initial placement (act 304 ). For instance, in the illustrated example of FIG. 2 , shard 210 A is initially stored to the storage location 211 A using the data channel 212 A as represented by arrow 222 A, shard 210 B is initially stored to the storage location 211 B using the data channel 212 B as represented by arrow 222 B, shard 210 C is initially stored to the storage location 211 C using the data channel 212 C as represented by arrow 222 C, and shard 210 D is initially stored to the storage location 211 D using the data channel 212 D as represented by arrow 222 D. As previously mentioned, this sharding and initial placement may be made such that writing and/or reading may be done in parallel for each shard, so that the write or read is completed roughly at the same time for each shard. Furthermore, the writing and reading process may be adaptive with proper restraints, so as to avoid any throttling performed by sharing policy, and thereby avoiding punitive throttling.

Now that the shards have been written, a number of things may happen to each shard. As one example, the shard may be replicated from one storage location to another. For instance, in FIG. 2 , as represented by arrow 223 A, the shard 210 A has been replicated from storage location 211 A to storage location 211 C. The sharding component 202 thus knows that there are two storage locations 211 A and 211 C from which the shard 210 A may be read. As another example, the shard 210 D has been replicated from storage location 211 D to both storage locations 211 E and 211 F. Thus, the sharding component 202 knows that there are now three storage locations 211 D, 211 E, and 211 F from which the shard 210 D may be read. Examples of other things that may happen to a copy of a shard is that the shard may perhaps become corrupted, or the storage location in which a shard is located may no longer be available. Furthermore, a shard may be replicated via further sharding of the shard (e.g., into subshards) that are then stored across multiple storage locations. The sharding component 202 may keep track of any information that represents the locations from which the shards or subshards can be accessed, or can no longer be accessed.

FIG. 5 illustrates a flowchart of a method 500 for accessing at least one shard of a sharded computing data unit. As an example, suppose that the sharding component 202 is to now access each of the shards 210 A, 210 B, 210 C and 210 D. For any shards for which there is only one data channel that can be used to access the shard, those shards may simply be obtained over that data channel. For instance, as represented by arrow 224 B, there is but one data channel 212 B that may be used to access the shard 210 B. Furthermore, as represented by arrow 224 C, there is but one data channel 212 C that may be used to access the shard 210 C. However, the method 500 is performed for any shard for which there are multiple options for acquiring the shard. For instance, shard 210 A might be acquired over data channel 212 A or 212 C; and shard 210 D may be acquired over data channel 212 D, 212 E or 212 F.

First, based on application of the policy (e.g., policy 203 ) to the multi-link traffic constraints of each of the available data channels, the sharding component 202 selects a data channel to use to access the shard (act 501 ). For instance, suppose that data channel 212 A remains the best channel to use to access the shard 210 A, despite that the shard 210 A can also be accessed now over data channel 212 C. Further suppose that data channel 212 E is now the best channel to use to access the shard 210 D, despite the shard 210 D still available over the data channel 212 D, and despite also newly being available over the data channel 212 F. The determining of which channel is the best depends on the sharding policy 203 and also is based on the various constraints of the links leading to and from the storage location. As previously mentioned such constraints may include self-restraining constraints in throughput that avoid potentially punitive throttling by sharing policy.

In response, the sharding component 202 uses the selected data channel to access the corresponding shard (act 502 ). For instance, as represented by arrow 224 A, the sharding component 202 accesses the shard from the storage location 211 A over the data channel 212 A. Also, as represented by arrow 224 D, the sharding component 202 accesses the shard 210 D from the storage location 211 E over the data channel 212 E. Again, the reading process may be adaptive with proper restraints, so as to avoid any throttling performed by the sharing policy, and to thereby avoiding punitive throttling.

Accordingly, the principles described herein intelligently select a best initial placement of shards of a computing data unit given the then existing and/or historical constraints of the various data channels leading to competing storage locations for the shard. Such constraints may be dynamically changing constraints, and may also factor in a component of history of the constraint over time. Also, when reading a shard, the policy allows for access of the shard from an appropriate storage location over an appropriate channel, given the then current constraints of that channel.

The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.