Azure Storage – Hands on with Queues, Part 2

How’s this for commitment, its 6am on a Saturday morning and I’m sitting here watching movies on my 360 and putting together part 2 in this series of articles on Queues. I’m here for ya, have no doubt about that.

In part 1 we covered the basics of a queue storage REST call. We constructed our URI, built an HTTPWebRequest, and then digitally signed the request. Lastly, we executed the request and created a queue. In this installment, we’re going to extend this by creating 3 additional requests: get a list of queues, put a message into a queue, and read/peek at messages.

If you missed part 1, please go back and check it out. In addition to the information on generating our initial request, that post also includes all the disclaimers about the approach I’m taking. In short, I’m no REST expert and I’m intentionally keeping the implementation simple so I can focus on the mechanics of making the low level call.

Adjustments to our current AzureQueue class

Before we begin, I want to make some changes to the class we started last time. Last time we imbedded three functions that we’ll want to reuse in all our new methods. So I’m breaking those out into their own methods with some minor modifications. What can I say, my love of reuse stems from a deep seated laziness. Separating out this common code will also help highlight the differences between the various calls we’re making.

First up is a method that will create the base HttpWebRequest object and set the initial properties. It will accept two parameters, the URI we’re calling and the method of the Request (get, post, put, etc). We’re making this and the other methods static because they don’t need any instanced values of the parent class and so that they can easily be called from any other static class methods.

        private static HttpWebRequest getBaseRequest(string _URI, string _Operation)
        {
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(_URI); // create queue
            request.Timeout = (int)TimeSpan.FromSeconds(30).TotalMilliseconds; // 30 second timeout
            request.ReadWriteTimeout = request.Timeout; // same as other value
            request.Method = _Operation; // we want to create a queue
            request.ContentLength = 0;
            request.Headers.Add("x-ms-date", DateTime.UtcNow.ToString("R", CultureInfo.InvariantCulture)); // always use UTC date/time

            return request;
        }

Next up is a method for creating the basic canonized string. I’m adjusting this slightly so that the string is created from a passed in HttpWebRequest object.

        private static string getBaseCanonizedString(HttpWebRequest _Request)
        {
            string StringToSign = _Request.Method + "\n" +             // "VERB", aka the http method
                String.Empty + "\n" +                                  // "Content-MD5", we not using MD5 on this request
                _Request.ContentType + "\n" +                          // "Content-Type"
                String.Empty + "\n" +                                  // "Date", we know this is here because we added it
                "x-ms-date:" + _Request.Headers["x-ms-date"] + "\n" +  // "CanonicalizedHeaders", we only have one, so we'll do it manually
                "/" + myAccountName + _Request.RequestUri.AbsolutePath;// "CanonicalizedResources", just storage account name and path for this request

            return StringToSign;
        }

And lastly, I want a method that generates the signature and attaches it to the request object. We’ll pass in the HttpWebRequest and string objects. Since the HttpWebRequest is a class and not a simple type, its by referrence so any changes we’re making will be available outside the method. This chunk is taken exactly from the original one with the exception of some minor variable renaming.

        private static void SignRequest(HttpWebRequest _request, string _CanonizedString)
        {
            // compute the MACSHA signature
            byte[] KeyAsByteArray = Convert.FromBase64String(myAccountKey);
            byte[] dataToMAC = System.Text.Encoding.UTF8.GetBytes(_CanonizedString);
            string computedBase64Signature = string.Empty;

            using (HMACSHA256 hmacsha1 = new HMACSHA256(KeyAsByteArray))
            {
                computedBase64Signature = System.Convert.ToBase64String(hmacsha1.ComputeHash(dataToMAC));
            }

            // add the signature to the request
            string AuthorizationHeader = string.Format(CultureInfo.InvariantCulture, "SharedKey {0}:{1}",
                myAccountName, computedBase64Signature);
            _request.Headers.Add("Authorization", AuthorizationHeader);
        }

Now we just have to make some minor adjustments to only Create method and we’re all ready to go. Here’s a snip-it of the updated method.

        public static AzureQueue Create(string QueueName)
        {
            bool result = false;
            AzureQueue tmpQueue = new AzureQueue(QueueName); // create base object

            // create base request object and modify
            HttpWebRequest request = AzureQueue.getBaseRequest(tmpQueue.URI.ToString(), "PUT"); // create queue

            // create the canonized string we're going to sign
            string StringToSign = AzureQueue.getBaseCanonizedString(request);

            // sign request
            SignRequest(request, StringToSign);

Now we’re ready to get to work on expanding our queue functions.

Getting a list of available queues

I don’t know about you, but I always like to be able to get a list of queues that are available. So we’ll start expanding our queue storage functionality here. There is only a few differences in this request over the one we created previously:

  • our URI does not include the queue name, we’re address our request to the “queue manager”
  • the request operation will be a “GET” instead of a “POST”
  • we need a query string to tell the queue manager what action we’re performing
  • we need to read the response stream to get our list of queues

The URI of our last request was a path style URI that looked like “http://127.0.0.1:10001/<accountname>/<queuename>”. Since we’re still accessing development storage, we’ll continue to use a path style URI, but this time we’re accessing the account and not a specific queue. We also want to add a couple parameters that help describe the request we’re making. Our new URI is “http://127.0.0.1:10001/<accountname>/?comp=list&maxresults=10”. The most important part here is the “comp” parameter. This tells the queue account manager that we’re wanting a list of queues. Using operational parameters such as “maxresults”, we can scope what we’re getting back the service. If you refer to the Azure "list queues" API on MSDN you’ll get a list of the other parameters.

So here’s the start of our new method for getting a list of queues. We’ll use the new URI and we’ll also change our operation from a “PUT” to a “GET”.

        public static DataSet getQueueList()
        {
            DataSet result = null;
            string tmpURI = myURI + myAccountName + "/?comp=list&maxresults=10";

            // create base request object and modify
            HttpWebRequest request = AzureQueue.getBaseRequest(tmpURI, "GET"); // create queue

            return result;
        }

Next we’ll create the canonized string and digitally sign our base request. Note how unlike our create queue example, we want to include the “?comp=list” query parameter as part of the canonized string.

            // create the canonized string we're going to sign
            string StringToSign = AzureQueue.getBaseCanonizedString(request);
            StringToSign += "?comp=list"; // required by the signature

            // sign request
            SignRequest(request, StringToSign);

Request is created, signed, and we’re ready to fire it off. Time for another quick disclaimer. The response to this request is an XML document. I have enough I want to cover in this post that I’m not going to walk you through its form. If you really want to see the schema of the request, you can find it on MSDN or just alter the code I’m giving you to save the doc to a string so you can peek at it.

We execute our request and make sure we get an “OK” response back. When inserting or creating something, this is all we needed to do. But now, we have to get a response stream and use that to read the xml result into a dataset object that is returned from the method.

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    if (response.StatusCode == HttpStatusCode.OK)
                    {
                        using (Stream stream = response.GetResponseStream())
                        {
                            result = new DataSet();
                            result.ReadXml(stream);
                            stream.Close();
                        }
                    }
                    else
                        result = null;
                    response.Close();
                }

Implementing our new method is deceptively simple. What I did is to put a dropdownlist control on a web page and then populate it using the results of our new method. Here’s what that looked like:

                DataSet tmpQueueList = AzureQueue.getQueueList();
                if ((tmpQueueList != null) && (tmpQueueList.Tables.Count >= 3))
                {
                    ddlQueues.DataTextField = "QueueName";
                    ddlQueues.DataSource = tmpQueueList.Tables[2]; // our list is the 3rd table
                    ddlQueues.DataBind();
                }

This is going to come in handy for our next two calls where we start putting messages onto and reading them from queues. So keep it handy.

Putting a message onto a queue

This call will be almost the opposite of the last one. We’ll construct a message payload and using a stream, feed it up to the service. Here’s the highlights:

  • our URI uses the queue name and the path “messages”
  • the request operation will be a “POST”
  • we’ll create the outbound message payload
  • send the message to the queue using the request stream

First we’ll create our message payload. We do this first because we’ll need its length when we create the request object. This snip-it is the construction of our message payload. We have to wrap the actual payload in a couple of XML tags and I’m just inserting a string into the middle of it that for our example. We’re also making it byte array because it should be UTF8 encoded and because when we stream it up to the service we’ll need it as an array anyways.

            // transform message payload
            System.Text.UTF8Encoding enc = new UTF8Encoding(false);

byte[] baMessage = enc.GetBytes("<QueueMessage><MessageText>" + _message.Trim() +

"</MessageText></QueueMessage>");

 

 

 

Next we’ll create our request object and sign the request as follows:

            // create base request object and modify
            HttpWebRequest request = AzureQueue.getBaseRequest(this.URI + "/messages", "POST"); // create queue
            request.ContentLength = baMessage.Length;

            // create the canonized string we're going to sign
            string StringToSign = AzureQueue.getBaseCanonizedString(request);

            // sign request
            SignRequest(request, StringToSign);

There are two things I want to call out in this. First off is our URI, we’re going to use the queues base uri and then add on the “/messages” part so that the service knows were after the queue’s contents. Next up, you’ll notice we’ modify the request after it is created, using the length of the byte array that contains our message payload.

Lastly, we’re going to execute our request…

                // send request
                using (Stream requestStream = request.GetRequestStream())
                {
                    requestStream.Write(baMessage, 0, baMessage.Length);
                    requestStream.Close();
                    using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                    {
                        result = (response.StatusCode == HttpStatusCode.Created);

                        response.Close();
                    }
                }

So using our request object, we get a stream, write our message to it, then execute the request. We need to check the return code to make sure the message posted properly. Not much to it really. You can optionally add in a time to life for the request (in seconds), but again, we’re just trying to keep these examples simple.

Reading and Peeking at messages in the queue

Now is where things start to get kind of fun. At least for a geek like me. This is also the first of these calls I got to work the first time! It was extremely satisfying to not have another of those “Forbidden” web exceptions.

Here’s the highlights of our new “read messages” method:

  • our URI uses the queue name and the path “messages”
  • the request operation will be a “GET”
  • use query string parameters to describe the number of messages and if we’re only “peeking”
  • read the response stream to get our messages

Getting or peeking at messages will combine several of the techniques we’ve used previously. We’ll use the same URI we used last time, put a couple query parameters on it, and finally execute our request with the response stream load a dataset object.  Lets start by preparing our URI with its query parameters.


            string tmpQueryParms = String.Empty;
            tmpQueryParms = "numofmessages=" + _NumOfMessages.ToString(CultureInfo.InvariantCulture);
            if (_peek)
            {
                if (tmpQueryParms.Length > 0)
                    tmpQueryParms += "&";
                tmpQueryParms += "peekonly=" + _peek.ToString(CultureInfo.InvariantCulture);
            }

            string tmpURI = this.URI + "/messages?" + tmpQueryParms;

The base URI is the same as in our last request. But we’re adding two parameters to it. First off, the “numofmessages” will tell us how many we want to retrieve. Next, we have an optional parameter “peekonly” which if the boolean variable I’m checking is set to true, will cause us to only peek at the messages. Time for a quick tangent…

For those reading this that may not be familiar with the concept of message queues, a queue is intended as a FIFO (first in, first out) method of sending message between two processes. Think of it as an early form of service bus. One process puts a message into a queue and one or more will pull it back out. The pulling process has the option of reading the message (making it unavailable to any other process for a given time), or just peeking at it. A “peek” doesn’t remove the message’s visibility from the queue.

Ok, back to task. After creating our URI, we’ll create the request and sign it just as we have before. We’ve done this several times already so I won’t bore you with the code yet again. We don’t need to worry about inserting any of our query string parameters like we did when we got the queue list, so creating our message signature is also straightforward. After those steps are performed, all that’s left is to execute our response.

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    if (response.StatusCode == HttpStatusCode.OK)
                    {
                        using (Stream stream = response.GetResponseStream())
                        {
                            result = new DataSet();
                            result.ReadXml(stream);
                            stream.Close();
                        }
                    }
                    else
                        result = null;

                    response.Close();
                }

Yes, you’re seeing this right. This is exactly like we did when we got the list of queues earlier. And like that example, I’m going to bind the result set to a control so I can see the results. It allows me to display the results easily. Here’s our implementation example:

            AzureQueue tmpQueue = new AzureQueue(ddlQueues.SelectedValue);
            DataSet tmpMessageList = tmpQueue.Get(10, cbPeekOnly.Checked);

            if ((tmpMessageList != null) && (tmpMessageList.Tables.Count > 0))
            {
                gvMessages.DataSource = tmpMessageList;
            }
            else
                gvMessages.DataSource = null;

In my case, I have a button that will read the queue specified by a dropdownlist control and return up to 10 messages from that queue. I’ve also put in a checkbox that I can use to determine if I’m going to peek at the messages or actually read them. Finally, I find the messages to a grid view control to display the results. The columns that are returned will vary depending on if we’re peeking or actually reading. The reason for these changes will be apparent to you when we get to part 3 of this series.

In our next episode, processing the messages

Well that’s it for this time. I already have the code snip-its for my next post completed so I just need to get the blob post written. In my next article I’m going to finally dig into processing these messages and removing them from the queue. I’ll even make a slight detour and show you how to check our queue depth.

Meanwhile, I’ve again uploaded the new version of our AzureQueue class. I know I skipped around the code a bit so I encourage you to check it out and play around with your own implementation of it. Especially look at the results from peeking versus reading the messages from queues.

Till next time!

PS – I would really like to hear from anyone that’s reading this blog. Feedback on the content or style is always appreciated.

Advertisements

3 Responses to Azure Storage – Hands on with Queues, Part 2

  1. Stacey says:

    Great job taking this down to a beginner level, you should write a book 🙂

  2. Adrian says:

    How is this working for a Cloud account? I am more concerned about string to sign for 2009-09-19 version for ListQueues. Have you tried that?

  3. Brent says:

    I haven\’t revisited this with the lastest release of Azure yet Adrian. But I\’ll add it to my list.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: