Azure Storage – Hands on with Queues, Part 3

I started drafting this when my son’s little league practice was cancelled due to rain. I loaded him down with horrible chores and put my daughter on laundry detail all so I could get this next update out. Family first? Not for this committed techno-geek. Its about the blog and sharing my excitement with you.

Just don’t tell my wife. ok?

In part 1 of this series we covered the basics of making a REST based call to Azure storage. In part 2 we played with variations of our first REST call to perform several other functions. In this final part I’m going to wrap things up and by showing two final variations and we’ll create a web role queue sample project.

The same disclaimers I’ve stated in previous articles apply. I’m no REST expert and doing my best to keep these samples as simple as possible. So please don’t expect best practices or elegant solutions to simple problems.

Deleting Messages

If you did as I suggested in part 2 of this series, you played around a bit with peeking versus reading messages from queues. And if you did that, you noticed what when you read a message from a queue, you get back two nodes/columns that are not present when peaking and that you didn’t set when you put the message onto the queue: PopReceipt and MessageId

When a message is read from the queue, that message will be hidden from any other processes reading the queue until the TimeNextVisible timestamp has passed. By default, this value will be 30 seconds from when the message was read. If you look at the Azure Queue API on MSDN you will see we could have added an optional query string parameter, visibilitytimeout,  to our example that would have allowed us adjust this period from 0 seconds to 2 hours. Feel free to go back and add this in, it all depends on how fast you can copy/paste when we get to testing our new method.

The next value, PopReceipt, is even more important. When we process a queue message, we will need to remove it from the queue permanently. To perform this operation we’ll need the PopReceipt as well as the MessageId values for the message to be removed.  This sets us up for our next new method, the Delete method.

I received some positive feedback on the bulleted lists I used last time to describe the variations of the methods so I’m sticking with it for now. Here’s the highlights for our new “delete messages” method:

  • our method will accept two string parameters: the PopReceipt and MessageId
  • the request operation will be a “DELETE”
  • the URI will use the queue name and the path “/messages/<MessageId>”
  • the PopReceipt will be specified via a query string parameter
  • we’ll look for two possible success status codes from the resulting HttpWebResponse

I want to start by discussing our URI. As you’ll notice we’re addressing our request to the queue, its messages, and finally one message in particular. The final part of this comes from the passed in MessageId. You would have thought that the message id would have been a query string value, this would have made sense to me as well. However, we need to remember that we’re addressing our request to a particular spot in storage. So setting our address to the specific message makes sense.

We then add on our PopReceipt as a query string value. The result of these two operations in our AzureQueue class looks like this:

            string tmpQueryParms = String.Empty;
            tmpQueryParms = "?popreceipt=" + _Receipt.Trim();

            string tmpURI = this.URI + "/messages/" + _MessageID.Trim() + tmpQueryParms;

Of course, _Receipt and _MessageID are the parameters passed into our method.

Next up, we’ll create the request object and sign it. Same operations we’ve done several times before, except that this time we’re using the DELETE http request operation.

Now all that’s left is to execute our request and check the response:

        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            if (response.StatusCode == HttpStatusCode.NoContent)
                result = true; // successfully deleted
            else if (response.StatusCode == HttpStatusCode.NotFound)
                result = true; // message was already deleted
            else
                result = false; // no such message found or other error
        }

Note that we have two positive results that can result. If the response code was “NoContent”, then the message was still there and it was successfully deleted. If the result was a “NotFound”, then the message had been removed before we made our request. If this is the case, we may actually want code in place to handle things if the message was processed multiple times.

This is where things can get interesting with the interaction between multiple subscribers (processes reading from a queue). If two processes both read a message, each will get a different PopReceipt value. Each time a new PopReceipt is generated, the old one expires. So if you are unable to delete a message, this means that another process has read the message (or that maybe someone deleted all the message from the queue). Regardless, it will be up to you to decide how best to address the situation. I can only recommend you plan carefully.

Queue Depth

Anyone that has worked with queues previously knows that there is one very important indicator of system health, queue depth. Queue depth is simply the number of messages that exist within a queue at any given time. If the number climbs above a given threshold, this could indicate that there is a performance problem in the queue subscribers or that more subscribers are needed. If the queue remains nearly empty, it may be that you have more queue subscribers than are required.  In either case, you’re going to want to be able to poll your queues to see what their depths are so you can act accordingly.

To that end, we’re going to create a new method for our AzureQueue class, GetQueueDepth. Here are the highlights:

  • the URI will use the queue name only
  • the request operation will be a “GET”
  • we’ll use a query string parameter to tell the queue what operation we’re performing
  • we’ll read a response header to get the value

Most of these we’ve done before, so I’ll just post my snip-it setting up the request:

            string tmpURI = this.URI + "?comp=metadata";

            // create base request object and modify
            HttpWebRequest request = AzureQueue.getBaseRequest(tmpURI, "GET"); 

            // create the canonized string we're going to sign
            string StringToSign = AzureQueue.getBaseCanonizedString(request);
            StringToSign += "?comp=metadata"; // required by the signature

            // sign request
            SignRequest(request, StringToSign);

So we take the queue object’s URI which already contains our path style URI with the account name and queue name and append our query string parameter on it. In this case we’re tell the queue we want its meta data. When creating the canonized string for our signature, we need to make sure we include this parameter in the string. We last did this when we were retrieving a list of the queues in our account (which also used the “comp” parameter). The “comp” parameter means we’re requesting a component of the targeted resource (in this case the queue) and according to the Azure Storage Authentication document on MSDN, we need to make sure we include it. 

Next up we simply execute our request, check the response status, and get our header from it:

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    if (response.StatusCode == HttpStatusCode.OK)
                    {

result = Convert.ToInt32(response.Headers["x-ms-approximate-messages-count"],

CultureInfo.InvariantCulture);

                    }

                    response.Close();
                }

I suppose I should do a couple checks to make sure the header is there. But I did test against an empty queue and it comes back with zero. So in a nuthsell there we have it. 🙂

Where to next, a sample implementation

So at this point, there are still other operations we could explore: delete queues, setting queue metadata, clearing queues. There are also other options within the operations we’ve explored that we could get into. But instead I figured I’d toss in a quick sample implementation that you can use to play with queues a bit.

Our sample application is going to be rigged to put messages into a queue at given intervals, getting messages back out of queues, and peeking at the contents. We’ll use a couple timer controls, a gridview, and a couple dropdownlists and checkboxes. The implementation will be pretty simple as well:

  • create the queue on page load and get an initial list of messages
  • start inserting messages into the queue at 5 second intervals, update message list each time
  • allow the user to initiate when messages start being removed from the queue
  • allow the user to adjust the intervals at which messages are put into or gotten off the queue

So here’s the snip-its…

On Page_Load

     if (myQueue == null)
     {
         myQueue = AzureQueue.Create("tstQueue");
         getMessages();
     }

The variable myQueue is a private class instance of our AzureQueue object. We use the static Create method of AzureQueue to get an instance of the object, and finally we call a new private method I’m going to create that will get up to 32 messages (the maximum allowed by a single get operation) from the queue. We could check to make sure an instance of the object was created, but as we’ve already stated, I’m here to demonstrate queue access, not best practices. 🙂 Laziness is only laziness if you’re doing it to avoid expending effort. I have a reason for taking a shortcut!

Here’s the getMessages method:

        protected void getMessages()
        {
            DataSet tmpMessageList = myQueue.Get(32, true);

            if ((tmpMessageList != null) && (tmpMessageList.Tables.Count > 0))
            {
                gvMessages.DataSource = tmpMessageList;
            }
            else
                gvMessages.DataSource = null;
            gvMessages.DataBind();
        }

I’m hopeful you’ll recognize this from our implementation in part 2 of this series. We’re peeking at the queue, and getting up to 32 messages. I then bind them to a gridview control. Course we won’t have anything to display unless we simulate messages being put on the queue.

        protected void TimerPut_Tick(object sender, EventArgs e)
        {
            myQueue.Put("Msg Inserted at: " + DateTime.Now .ToLongTimeString());

            // update list of messages
            getMessages();
        }

I don’t think I need to explain that this is tick event of our put timer. It inserts a message into our queue at a given interval. Then we just update the dataset that our gridview is bound too.

This leaves only one final operation, reading (or popping in queue terms) messages off the queue. We’ll do a non-peek get for 1 message, and get its MessageId and PopReceipt. Then we’ll turn around and issue our Delete request. Finally, we have to again update our list of messages for display:

        protected void TimerGet_Tick(object sender, EventArgs e)
        {
            DataSet tmpMessageList = myQueue.Get(1, false);
            if ((tmpMessageList != null) && (tmpMessageList.Tables.Count > 0))
            {
                myQueue.Delete(
                    tmpMessageList.Tables[0].Rows[0]["MessageId"].ToString(), 
                    tmpMessageList.Tables[0].Rows[0]["PopReceipt"].ToString());
            }

            // update list of messages
            getMessages();
        }
And that’s all the key items to our sample implementation. There’s a few other events that need hooked up, mainly for enabling/disabling our timers and setting their frequency. But I’m certain those need no explanation. I will close wish a quick look at my implementation.

queue_sample

What will happen as this runs is that the top message will “pop” off, and a new message will appear at the bottom. You can see the impact of the queue depth by enabling/disabling inserts or reads as well as adjusting the frequency. Admittedly this isn’t a very practical example (unless you need a database style “stack” collection), but it does a fair job of illustrating the usage of a queue.

You are the queue master, you have mastered the queues!

Well, this concludes my series on accessing queues. I want to again salute the “Red Dog” team and their StorageClient sample project. Its a very robust implementation and some great thought went into it. I hope that what I’ve given you here is a simple, straightforward set of examples for creating your own queue access implementation. Perhaps it will be a robust solution like the StorageClient. Or maybe you’ll go for something more lightweight.

In either case, thanks for taking the time to read these postings. I’ve uploaded a zip file containing my solution so you can pull it down and try running it yourself.

Till next time and thanks for stopping by!  Now how is that Windows 7 RC downloading going….. oh wait, its family time! 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: