Azure Storage – Hands on with Queues, Part 1

This is what I get for wanting to understand the basics. I spent about 4 hours reverse engineering the StorageClient sample project and reconciling it against the Azure Storage Queue API. I do want to give a salute to the Azure team’s sample project. Its a great example and I have little doubt that many shops will simply adopt it as their standard client. And while I understand the need to promote the platform agnostic nature of consuming storage via its REST api, I am still a bit disappointed we’re not hearing more about an “official” .NET library for access.

Before I dive in there’s a couple things I’d like to make perfectly clear. First off, I don’t have much practical experience with doing REST based calls. I’ve been fortunate enough that most of the time I have someone on the team that really enjoys doing those types of things so I’m able to dodge it. Secondly, the approach I’m going to show you is a bit more complex then just building the request via a stringbuilder but also not as flexible as the StorageClient sample project.

I’ll also admit that my example is going to be very crude, and extremely basic. Just please keep in mind that the goal for this isn’t to enforce/promote best practices but to give you a straightforward example of making these rest based calls.

Ok, on to business. The process of creating a rest call is actually pretty straight forward:

  1. Construct the URI
  2. build the appropriate HTTPWebRequest
  3. digitally sign the request using HMACSHA256 (Message Authentication Code, Secure Hash Algorithm)
  4. call the HTTPWebRequest.GetResponse method
  5. close the response and release resources.

Creating the Solution

Start by firing up Visual Studio and creating a new “Web and Worker Cloud Service” project. I’ve named mine “QueueDemo”. And because I’m going to want to access my Azure Storage classes from both the Web and Worker roles, I’ll add a class library called “StorageDemo”. If you use the “Class Library” template, we’ll want to also add the following references: Microsoft.ServiceHosting.ServiceRuntime (Windows Azure Service Hosting namespace), and System.DataServices.Client (for accessing ADO.NET Data Services). Lastly, rename the default class to “AzureQueue”.

At this point, I’m not 100% certain we need the ServiceHosting reference, but I’m adding it because we are doing an Azure application and the StorageClient sample uses it. Later I’ll play around and see if I really need it. I also left out a reference to System.Configuration. While best practices tell me to put my account information into a configuration file, I want to do a seperate blog post at a later date on configuration options in Azure, so for the moment I’m just going to hard-code my credentials. Also, our sample project will only work against development storage for now. We’ll switch this later.

Now, enhance the AzureQueue class in our web role project as seen below:

    public class AzureQueue: Object
    {
        public static AzureQueue Create()
        {
        }
    }

We’re going to start our hands on journey with creating a queue. We’ll extend this initial example over several more articles in the coming weeks.

Generate the URI

Regardless of the operation we’re performing, we need to know the URI that we’ll be using. Using the host, our account name, and a handful of optional parameters we can create two types of URI’s. For local development storage we’ll use a “path style” URI. Hosted storage uses a “host style” URI. The primary difference between the two is where the account name will end up.

Path Style (development): http://127.0.0.1:10001/<accountname>/<queuename&gt;

Host Style: http://<accountname&gt;.queue.core.windows.net/<queuename>

There’s also the option of appending query string parameters to these URI’s. The StorageClient sample project includes a “timeout” parameter for the queue timeout. We’ll leave that one off our example for the moment.

To handle this, lets add a private string to our AzureQueue class to store the proper value for our example. Mine will look like this:

	private static string myURI = "http://127.0.0.1:10001/";

Note that I’m leaving both the account name and queue name off so I can append them appropriately for each queue I need to create. I made it static just to keep things easy for me when I’m using this value later.

We’ll also add two private static values that hold our account name and key. Since we’re access development storage, these are the same for EVERYONE. As I mentioned earlier, we’ll discuss putting these values into a configuration file and accessing them from there in another article.

Next up, we’re going to add a private string to hold my QueueName, a public property that exposes it, and finally we need to enhance the class constructor so that we assign the parameter to my private value. We’re also going to add a property that generates our path style URI for us using the account and queue names.

The updated class looks like this:

    public class AzureQueue: Object
    {
        private static string myURI = "http://127.0.0.1:10001/";
        private static string myAccountName = "devstoreaccount1"; // same for everyone
        private static string myAccountKey = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==";
        private string myQueueName = string.Empty;

        public string Name
        {
            get { return myQueueName.Trim(); }
        }

        public Uri URI
        {
            get { return new Uri(myURI + myAccountName + "/" + Name.Trim()); }
        }

        public AzureQueue(string QueueName): base()
        {
            myQueueName = QueueName.Trim();
        }

        public static AzureQueue Create(string QueueName)
        {
        }
    }

Create the HTTPWebRequest

That takes care of setting the stage for us to execute our request. So we’ll start filling in the Create method we just added as this is where the actual execution of our REST call will happen. We’ll start this by creating our HTTPWebRequest object as follows:

  • create an instance of the System.Net.HttpWebRequest class using our generated/composite URI property
  • set the Timeout and ReadWriteTimeout property to our Timeout in Milliseconds
  • set the appropriate http method (put, get, etc… more in a few on this)
  • set the content length (which is zero for this operation)
  • add a header for the date/time

For other Azure Storage requests, we may also have some request headers that may need to be set. But for this initial example, this is all we need.

Now about the http method. Thes HTTP methods help control the type of operation we’re performing on containers, or for this example, our queue. You can find all these in the MSDN API reference, but here’s a short list of methods for queues:

PUT

creates a queue or sets queue metadata

GET

used to retrieve a list of queues, a message from a queue, or peek at a message

DELETE

removes a queue, a message, or clears all messages from a queue

POST

puts a message on a queue

GET/HEAD

retrieves meta data about a queue
 
The exact operation performed will depend on the URI used and in most cases additional parameters and/or a request body. Fortunately, creating a queue requires none of these.
 
Here’s my updated create method with our HttpWebRequest object’s initial setup:
   public static AzureQueue Create(string QueueName)
        {
            bool result = false;
            AzureQueue tmpQueue = new AzureQueue(QueueName); // create base object

            // create our web request
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(tmpQueue.URI); // create queue
            request.Timeout = (int)TimeSpan.FromSeconds(30).TotalMilliseconds; // 30 second timeout
            request.ReadWriteTimeout = request.Timeout; // same as other value
            request.Method = "PUT"; // we want to create a queue
            request.ContentLength = 0;
            request.Headers.Add("x-ms-date", DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture)); // always use UTC date/time

            // return the result
            return (result?tmpQueue: null);
        }
    }

We could do a few extra operations before we call this, such as making sure the queue doesn’t already exist in storage. But you’ll see later that this isn’t always necessary.

Update May 13th: I’d like to point out that in the development storage, queue names can be a mix of upper and lower case. However, when accessing hosted azure storage, upper case is not allowed. Be sure to adhere to the API rules for queue names if you plan to deploy.

Oh, and don’t forget to add some using clauses for the System.Net and System.Globalization namespaces. 🙂 Helps avoid nasty red lines when compiling.
 

Sign our request on the dotted line

Ok, we have our URI, we’ve built a request object, but because we’re being nice and secure, we need to make sure we sign our request using MACSHA (Message Authentication Code, Secure Hash Algorithm – yeah, I looked it up). This occurs in two stages, canonizing the request, and computing the digital signature.

If you look at the SampleStorage project, they have some excellent bits of code written that will handle the canonization for us. You can find this looking at the MessageCanonicalizer class found in the Authentication.cs file. If you like, you can just grab that code and implement it to do the work for you. However, to be more in keeping with the intent of this article, we’re going to grab a string that is shown in the Authentication Schemes API reference found on MSDN.

StringToSign = VERB + “\n” + 
      Content-MD5 + "\n" +
      Content-Type + "\n" +
      Date + "\n" +
      CanonicalizedHeaders + "\n" +
      CanonicalizedResource;

The resulting code will be inserted into our Create method right above the return. Here’s what I ended up with:

            // create the canonized string we're going to sign
            string StringToSign = request.Method + "\n" +              // "VERB", aka the http method
                String.Empty + "\n" +                                  // "Content-MD5", we not using MD5 on this request
                request.ContentType + "\n" +                           // "Content-Type"
                String.Empty + "\n" +                   	       // "Date", this is a legacy value and not really needed
                "x-ms-date:" + request.Headers["x-ms-date"] + "\n" +   // "CanonicalizedHeaders", we only have one, so we'll do it manually
                "/" + myAccountName + tmpQueue.URI.AbsolutePath;       // "CanonicalizedResources", just storage account name and path for this request

With the canonized string created, we next need to generate the MACSHA hash and attach that calculated signature to our message. This is pretty straight forward as seen below:

            // compute the MACSHA signature
            byte[] KeyAsByteArray = Convert.FromBase64String(myAccountKey);
            byte[] dataToMAC = System.Text.Encoding.UTF8.GetBytes(StringToSign);
            string computedBase64Signature = string.Empty;

            using (HMACSHA256 hmacsha1 = new HMACSHA256(KeyAsByteArray))
            {
                computedBase64Signature = System.Convert.ToBase64String(hmacsha1.ComputeHash(dataToMAC));
            }

            // add the signature to the request
            string AuthorizationHeader = string.Format(CultureInfo.InvariantCulture, "SharedKey {0}:{1}",
                myAccountName, computedBase64Signature);
            request.Headers.Add("Authorization", AuthorizationHeader);

To review this, we convert the account key private static variable and our canonized string to byte arrays. Using the keys byte array as for our hash seed, we compute the hash on the canonized string. We then add this signature to the request as a header named “Authorization”. The Queue service will then compute the hash just as we have and verify that the two values match before accepting our request.

Execute the HTTPWebRequest and parse the response

So we’ve created our request and created its digital signature. Now all that remains is to execute it and check our response. To do this, I’m going to borrow the code used in the StorageClient sample almost verbatim.

            // execute our request and process the response 
            // (taken almost word for word from the StorageClient sample project)
            try
            {
                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    if (response.StatusCode == HttpStatusCode.Created)
                    {
                        // as an addition we could parse the result and retrieve
                        // queue properties at this point
                        result = true;
                    }
                    else if (response.StatusCode == HttpStatusCode.NoContent)
                        result = true;
                    else
                        result = false;
                    response.Close();
                }
            }
            catch (WebException we)
            {
                if (we.Response != null &&
                    ((HttpWebResponse)we.Response).StatusCode == HttpStatusCode.Conflict)
                {
                    result = true;
                }
                else
                    result = false;
            }

And there we have it, all the code necessary to generate our “create queue” request against the Azure Storage service. 🙂 I did mention above that we didn’t have to check for the pre-existance of the queue. If you look at our example, there are several options that return true. While its more obvious in the original StorageClient example, one of these is the result of a queue already existing. We could just have easily handled this differently. So the final implementation is up to you.

Putting it all to work

Of course all this code is worthless without some function to access it. I’ve already been wordy enough so I’ll just explain this the quick way… Go to the web role project and add 3 controls to the default.aspx page: a text box, a button, and a label. Name and position them all as you see fit. Wire up the click event of the button and inside that handler create an instance of our Azure Queue using the value entered into the text box for the queue name. Here’s the sample code I used:

            AzureQueue tmpQueue = AzureQueue.Create(tbQueueName.Text.Trim());
            if (tmpQueue != null)
                lblResult.Text = "Queue Created";
            else
                lblResult.Text = "Creation Failed";

If you’ve done everything correctly, you should be a confirmation message. If not, good luck troubleshooting the error. During my initial test run I kept getting a “Forbidden” response from the Azure Storage Services. Turned out it was due to two errors I had made, one on the URI and one on the canonized signature. Took about an hour to work out so I’m hoping over time I get more adept at troubleshooting these types of problems. I can also see that unless things change, its going to be difficult to support these. Ug!

Next time, our intrepid code monkey ventures deeper into the Storage Jungle

I had wanted to cover a bit more but this has already drug on long enough. So I’m going to delay 3 topics until my next post. So tune back in again in about a week and I’ll cover getting a list of the available queues, as well as putting and getting queue messages. All these will be based on the work I’ve started here, so I’ve uploaded a copy of my AzureQueue.cs file for you to check out.

Till then!

Advertisements

6 Responses to Azure Storage – Hands on with Queues, Part 1

  1. Brent says:

    I\’ve added a small update to this article to reflect a difference I uncovered between development and hosted stroage. Namely that while development storage supports the use of upper case letters in queue names, hosted storage does not.

  2. Ronald says:

    this is awesome! I\’ve got 2 main concerns:1. static create method. makes it alot readable i guess, however at the expense of making it harder for creating a mock for it (in the context of BDD)2. Do you worry about racing conditions? i kinda wish that I could create the queue upon first request rather than onApplicationStart. And in that case there might be 2 requests coming in almost at the same time, and will cause 2 requests for creating queue. the first would succeed and the second might throw an error? do you feel the need to introduce locking as I do when calling create()?

  3. Brent says:

    Ronald, both good questions.1) I took the static method approach because is a lazy man\’s implementation of a object factory pattern. I personally like that pattern but didn\’t want to bog down my post by creating a full factory. The real goal of the article was to strip away all the layers that are in the StorageClient sample application so that you could easily create your own solution. :)2) I\’m actually not worried about that. Given that the actual creation of the queue happens within the Azure Storage "engine", any type of cross-thread locking I were to put in place would occur two far away from the actual creation of a queue to do much good. I could make it request safe, but what if I have a web role and a worker role that both try to ensure that the queue is there at the same instance. Ultimate Azure storage needs to be the authority. In that case, the 2nd item will return a "conflict" response that my sample traps for that simply tells me that the queue already existed. For your specific implementation, you can handle it differently if you like.

  4. Ronald says:

    Hi Brent, I agree that Azure storage needs to be the authority. However, I still think it\’s not acceptable for the 2nd item to return a \’conflict\’ response. I put into more thoughts into what you build, and I guess what I have a better understanding to what I\’m trying to achieve:1. create() needs to return a newly created queue if doesn\’t exist, 2. but return an existing queue if it already exist. 3. It also needs to have a threadsafe mechanism to check whether queue has already been created or notOtherwise, let\’s imagine how your class is going to be used without them:1 public void run(string msg)2 {3 // we can put this in a static variable but will introduce more complexities4 // in multiple threads. so let\’s not do that for now.5 AzureQueue q = AzureQueue.Get("testQueue");6 7 if (!isQueueFound(q))8 {9 AzureQueue.Create("testQueue");10 }11 12 q.AddToQueue(msg);13}the method is ran by 2 different users nearly at the same time.user 1 hits line 9. azure storage is being createduser 2 hits line 5. azure storage is not ready yet return QueueNotFound(404)user 1 hits line 12. azure storage is created. successfully add msg to queueuser 2 hits line 9. azure storage creation is declined. QueueAlreadyExist(404)user 2 hits line 12. nullreference exception

  5. Brent says:

    IMHO You\’ve made a solution that\’s more difficult then it needs to be and subsequently creates the very situation you\’re trying to avoid. You also still haven\’t addressed that fact that even if one application has thread safe management of the queue, what happens when two applications are both trying to access the queue service?A call to the create method I spec\’d out will return an object regardless of it it had to create it or it already existed. Your test of \’does it exist\’ does not add any value given this scenario. If you were trying to save the create call (when the queue exists), you\’ve only replaced it with a different call that will have a potentially larger return result.simply handle the \’already exists\’ scenario and don\’t focus on trying to check for its existance.

  6. Pingback: Writing to an Azure Storage Queue from a Micro Framework device | devMobile's blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: