A page at a time (Page Blobs–Year of Azure Week 7)

Going to make this quick. I’m sitting in SeaTac airport in Seattle enjoying the free wifi and need to knock this out quickly as we’re going to start boarding in about 10-15 minutes. I’ve been here in Seattle doing some azure training and working with a client that’s trying to move to Azure when my topic for this week fell in my lap.

One of the attendees at my training wanted to see an example of doing page blobs. I poked around and couldn’t find one that I liked to I decided I’d come up with one. Then, later in the afternoon we had a discussion about an aspect of their client and the idea of the random access abilities of page blobs came to mind. So while I haven’t had a chance to fully prove our my idea yet, I do want to share the first part of it with you.

The sample below focus’s on how to take a stream, divide it into chunks, and write those to an Azure Storage page blob. Now, in the same I keep each write to storage at 512 bytes (the size of a page), but you could use any multiple of 512. I just wanted to be able to demonstrate the chunk/reassemble process.

We start off by setting up the account, and creating a file stream that we’ll write to Azure blob storage:

MemoryStream streams = new MemoryStream();// create storage account
var account = CloudStorageAccount.DevelopmentStorageAccount;
// create blob client
CloudBlobClient blobStorage = account.CreateCloudBlobClient();CloudBlobContainer container = blobStorage.GetContainerReference(“guestbookpics”);
container.CreateIfNotExist(); // adding this for safetystring uniqueBlobName = string.Format(“image_{0}.jpg”, Guid.NewGuid().ToString());

System.Drawing.Image imgs = System.Drawing.Image.FromFile(“waLogo.jpg”);

imgs.Save(streams, ImageFormat.Jpeg);

You may remember this code from the block blob samples I did a month or two back.

Next up, I need to create the page blob:

CloudPageBlob pageBlob = container.GetPageBlobReference(uniqueBlobName);
pageBlob.Properties.ContentType = “image\\jpeg”;
pageBlob.Metadata.Add(“size”, streams.Length.ToString());
pageBlob.Create(23552);

Notice that I’m setting it to a fixed size. This isn’t ideal, but in my case I know exactly what size the file I’m uploading is and this is about twice what I need. We’ll get to why I’ve done that later. The important part is that the size MUST be a multiple of 512. No partial pages allowed!

And finally, we write start reading my file stream into a byte array buffer, convert that buffer into a memory stream (I know there’s got to be a way to avoid this but I was in a hurry to write the code for this update), and writing each “page” to the page blob.

streams.Seek(0, SeekOrigin.Begin);
byte[] streambuffer = new byte[512];int numBytesToRead = (int)streams.Length;
int numBytesRead = 0;
while (numBytesToRead > 0)
{
// Read may return anything from 0 to 10.
int n = streams.Read(streambuffer, 0, streambuffer.Length);
// The end of the file is reached.
if (n == 0)
break;MemoryStream theMemStream = new MemoryStream();
theMemStream.Write(streambuffer, 0, streambuffer.Length);
theMemStream.Position = 0;
pageBlob.WritePages(theMemStream, numBytesRead);numBytesRead += n;
numBytesToRead -= n;
}

Simple enough, and it works pretty well to boot! You’ll also notice that I’m doing this one 512 byte page at a time. This is just for demonstration purposes as the maximum size you can write (based on the REST API documentation) is 4mb. But as part of my larger experiment, the one page at a time method means I can use smaller sample files. Smile

The one piece we’re missing however is the ability to shrink the page blob down to the actual minimum size I need. For that, we’re going to use the code snippet below:

Uri requestUri = pageBlob.Uri;
if (blobStorage.Credentials.NeedsTransformUri)
requestUri = new Uri(blobStorage.Credentials.TransformUri(requestUri.ToString()));HttpWebRequest request = BlobRequest.SetProperties(requestUri, 200,
pageBlob.Properties, null, 12288
);blobStorage.Credentials.SignRequest(request);
using (WebResponse response = request.GetResponse())
{
// call succeeded
};

You’ll notice this is being done via a REST request directly to blob storage, resizing a blob isn’t supported via the storage client. I also need to give credit for this last snippet to the Azure Storage Team.

As I mentioned, I’m in a hurry and wanted to get this out before boarding. So you’ll need to wait until next week to see why I’m playing with this and hopefully the potential may excite you. Until then, I’ll try to refine the code a bit and get the entire solution posted online for you.

Until next time!

About these ads

4 Responses to A page at a time (Page Blobs–Year of Azure Week 7)

  1. Pingback: Windows Azure and Cloud Computing Posts for 8/18/2011+ - Windows Azure Blog

  2. Pingback: Something about Development Technology :-)

  3. Mergim says:

    Hi Brent and thanks for a great post. Very useful.

    I though I’d share my take on this and ask a quick question.

    I converted your example to upload a plain text file instead of an image. All worked well except one gotcha. Some of the lines, from the end of the text, were not being written to the blob storage. After debugging I noticed that if the last bytes read were not equal to 512 (or whatever page size is set) it will just skip them. And if a text file had say 600 bytes, then the first 512 will get written and the rest 88 will not. If we were to change the page to 1024, then the whole 600 will get written, and have the zero fillers in it (as expected). However, if the file is several hundreds of megs, then the problem persists, i.e., due to size limits to 4 megs. I wrote a small filler code as a way around it (see below), but I was just wondering if you recommend a better way, or this is it? Thanks again,

    /*If we were to add the following code just after where we check if n==0, this seems to solve the problem, where all the text gets written */
    if ((n % 512) != 0)
    {
    //we have read data that is not equal to 512 in size
    for (int i = n; i < streambuffer.Length; i++)
    {
    streambuffer[i] = 0x20;
    }
    }

    *I surmise the same problem exists with the image upload.

    • Brent says:

      That’s actually a great catch. What you’re seeing is the page/blocks used by Azure Storage Blobs. With block blobs, you can vary the size of the block up to 4mb (with a maximum size of 50k blocks or 200gb whichever comes first). So you pick a block size that makes sense for your file and can fill with zeros if you so choose. Just be aware of the size when reading it back out. I do something similar in my posts on page blobs. So filling out the blocks as you describe isn’t a bay idea but depending on the file type may not always be necessary.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,149 other followers

%d bloggers like this: