Developing Cloud Applications with Windows Azure Storage: Blobs

Continuation tokens and blobs

A pagination mechanism for data that can be presented in a tabular format is often required because humans generally cannot digest more than a page of information at one time. Also, you want to ensure that the data being requested is appropriate for the query and not a mistake, because you are dealing with potentially massive databases in the cloud. It can take a lot of resources to compute and transmit billions of rows of data over the wire. Windows Azure must also take into account its own scalability; a large query can block or significantly impede many smaller queries, because those smaller queries may have to wait or compete for resources. Continuation tokens, as introduced in Chapter 4, allow Windows Azure storage to return a smaller subset of your data. A continuation token imposes upon you, however, to ask for subsequent pages of your data by passing you back a continuation token that you must then use to call back to retrieve subsequent pages of your query. Windows Azure storage refers to these pages of data as segments.

Blobs are not tabular data, so you do not have to anticipate continuation tokens when retrieving blobs; however, some of the API does return tabular data such as retrieving a list of blobs stored in a container. Because a container can have an unlimited number of entries, you should always anticipate that you may receive a continuation token back from any request that you make by calling an xxx-Segmented retrieval method.

Console.Clear();
CloudBlobClient client = account.CreateCloudBlobClient();

// ListContainers/BlobsSegmented return up to 5,000 items
const Int32 desiredItems = 5000 + 1000;  // Try to read more than the max

// Create a bunch of blobs (this takes a very long time):
CloudBlobContainer container = client.GetContainerReference("manyblobs");
container.EnsureExists();
for (Int32 i = 0; i < desiredItems; i++)
    container.GetBlockBlobReference(String.Format("{0:00000}", i)).UploadText("");

// Enumerate containers & blobs in segments
for (ContainerResultSegment crs = null; crs.HasMore(); ) {
    crs = client.ListContainersSegmented(
      null, ContainerListingDetails.None, desiredItems, crs.SafeContinuationToken());
    foreach (var c in crs.Results) {
        Console.WriteLine("Container: " + c.Uri);
        for (BlobResultSegment brs = null; brs.HasMore(); ) {
            brs = c.ListBlobsSegmented(null, true, BlobListingDetails.None,
                desiredItems, brs.SafeContinuationToken(), null, null);
            foreach (var b in brs.Results) Console.WriteLine("   Blob: " + b.Uri);
        }
    }
}

In the preceding code, the value of the ContinuationToken property of the ResultSegment will be either a continuation token or null when there are no more results to read. The following extension method, which is included in the Wintellect Azure Power Library, was used to simplify the programming model. It does this by supplying a Boolean indicator that controls exiting the for-loop when no more result segments are available.

public static BlobContinuationToken SafeContinuationToken(
          this ContainerResultSegment crs) {
   return (crs == null) ? null : crs.ContinuationToken;
}Blob request options

As used in the preceding code examples, instances of BlobRequestOptions may be passed as arguments to blob operations to augment the behavior of a request. The BlobRequestOptions class is an assortment of loosely related properties that are applicable during different kinds of operations. Only Timeout and RetryPolicy are used by all data access storage requests. Table 5-6 describes the properties of BlobRequestOptions.

Table 5-6 BlobRequestOptions properties and their impact on requests