Partial Service Upgrades

So I as working on an answer for a stack overflow question yesterday and realized it was a topic that I hadn’t put down in my blog yet. So rather than just answer the question, I figured I’d blog about it here so I could include some screen shots and further explanation. The question was essentially how can I control the deployment of individual parts of my service.

So for this, I create a simple Windows Azure service with a Web Role, and a Worker Role. Its already up and hosted when we start this.

NOTE: this post only details doing this via the windows.azure.com portal. We’ll leave doing I programmatically via the management API for another day.

Upgrading a Single Role

This is actually surprising simple. I open up the service, and select the role (not instances) I want to upgrade. Then we can right click and select upgrade, or click on the “upgrade” button on the toolbar.

Either option will launch the “Upgrade Deployment” diaglog box. If you look at this box (and presuming you have the role selected, you’ll notice that in the box, the “Role to Upgrade” option will list the role you had selected. If you didn’t properly select the role, this may list “All”.

Take a peek at the highlighted section of the following screen shot for an example of how this should look.

Note: while creating this post, I did receive an unhandled exception message from the silverlight portal. This has been reported to MSFT and I’ll update his when I get a response.

Manual Upgrades

I’ve run out of time today, but next time I’d like to cover doing a manual upgrade. Of course, I still have two posts in my PHP series I need to finish. So we’ll see which of these I eventually get back around to first.

Until next time!

A Custom High-Availability Cache Solution

For a project I’m working on, we need a simple, easy to manage session state service. The solution needs to be highly available, low latency, but not persistent. Our session caches will also be fairly small in size (< 5mb per user). But given that our projected high end user load could be somewhere in the realm of 10,000-25,000 simultaneous users (not overly large by some standards), we have serious concerns about the quota limits that are present in todays version of the Windows Azure Caching Service.

Now we looked around, Memcached, ehCache, MonboDB, nCache to name some. And while they all did various things we needed, there were also various pros and cons. Memcached didn’t have the high availability we wanted (unless you jumped through some hoops). MongoDB has high availability, but raised issues about deployment and management. ehCache and nCache well…. more of the same. Add to them all that anything that had a open source license would have to be vetted by the client’s legal team before we could use it (a process that can be time consuming for any organization).

So I spent some time coming up with something I thought we could reasonably implement.

The approach

I started by looking at how I would handle the high availability. Taking a note from Azure Storage, I decided that when a session is started, we would assign that session to a partition. And that partitions would be assigned to nodes by a controller with a single node potentially handling multiple partitions (maybe primary for one and secondary for another, all depending on overall capacity levels).

The cache nodes would be Windows Azure worker roles, running on internal endpoints (to achieve low latency). Within the cache nodes will be three processes, a controller process, the session service process, and finally the replication process.

The important one here is the controller process. Since the controller process will attempt to run in all the cache nodes (aka role instances), we’re going to use a blob to control which one actually acts as the controller. The process will attempt to lock a blob via a lease, and if successful will write its name into that blob container. It will then load the current partition/node mappings from a simple Azure Storage table (I don’t predict us having more then a couple dozen nodes in a single cache) and verify that all the nodes are still alive.  It then begins a regular process of polling the nodes via their internal endpoints to check on their capacity.

The controller also then manages the nodes by tracking when they fall in and out of service, and determining which nodes handle which partitions. If a node in a partition fails, it will assign that a new node to that partition, and make sure that the node is in different fault and upgrade domains from the current node. Internally, the two nodes in a partition will then replicate data from the primary to the secondary.

Now there will also be a hook in the role instances so that the RoleEnvironment Changing ad Changed events will alert the controller process that it may need to rescan. This could be a response to the controller being torn down (in which case the other instances will determine a new controller) or some node being torn down so the controller needs to reassign their the partitions that were assigned to those nodes to new nodes.

This approach should allow us to remain online without interruptions for our end users even while we’re in the middle of a service upgrade. Which is exactly what we’re trying to achieve.

Walkthrough of a session lifetime

So here’s how we see this happening…

1. The service starts up, and the cache role instances identify the controller.
2. The controller attempts to load any saved partition data and validate it (scanning the service topology)
3. The consuming tier, checks the blob container to get the instance ID of the controller, and asks if for a new session ID (and its resulting partition and node instance ID)
4. The controller determines if there is room in an existing partition or creates a new partition.
5. If a new partition needs to be created, it locates two new nodes (in separate domains) and notifies them of the assignment, then returns the primary node to the requestor.
6. If a node falls out (crashes, is being rebooted), the session requestor would get a failure message, and goes back to the controller for a new node for that partition.
7. The controller provides the name of the previously identified secondary node (which is of course now the primary), and also takes on the process of locating a new node.
8. The new secondary node will contact the primary node to begin replicate its state. The new primary will start sending state event change messages to the secondary.
9. If the controller drops (crash/recycle), the other nodes will attempt to become the controller by leasing the blob. Once established as a controller, it will start over at step 2.

Limits

So this approach does have some cons. We do have to write our own synchronization process, and session providers. We also have to have our own aging mechanism to get rid of old session data. However, its my believe that these shouldn’t be horrible to create so its something we can easily overcome.

The biggest limitation here is that because we’re going to be managing the in-memory cache ourselves, we might have to get a bit tricky (multi-gigabyte collections in memory) and we’re going to need to pay close attention to maximum session size (which we believe we can do).

Now admittedly, we’re hoping all this is temporary. There’s been mentions publically that there’s more coming to the Windows Azure Cache service. And we hope that we can at that time, swap out our custom session provider for one that’s built to leverage whatever the vNext of Azure Caching becomes.

So while not ideal, I think this will meet our needs and do so in a way that’s not going to require months of development. But if you disagree, I’d encourage you to sound off via the site comments and let me know your thoughts. .

Windows Azure & PHP (for nubs)–Side note

Writing your from CodeMash 2.0.1.2. First time here as an attendee and a presenter. Having a great time meeting many great people. Yesterday I presented on Windows Azure with PHP (see deck and demo script in the resource area) and it went smoothly except for a minor demo hiccup.

I tell folks that even after 3 years of working with the platform, I still learn things. Well once I dug into the root cause of my demo hiccup, it turns out I learned something about the Azure SDK tools (something that my use of Visual Studio always hid from me).

During the demo, I built and deployed a project locally. Doing a “package create” with “-dev=true”. It turns out the PHP SDK is using the Azure SDK tools cspack and csrun to get my project packed and deployed to the Azure Development Emulator. And when its doing this for the local emulator, the package isn’t encrypted as it would be for deployment to a hosted Windows Azure service. Combine that with my system settings to automatically open zip files as if they were folders, and the upload dialog at windows.azure.com simply wouldn’t let me select the file for deployment.

opps!

But by re-running the”package create” again with “-dev=false”, it then uses cspack to create the encrypted file for deployment to the cloud.

I’ve already updated the presentation materials listed above. So the demo script now makes it really clear that this is the case. As I prepare to pack up and leave CodeMash, its knowing that I learned not just from other awesome presenters, but I also managed to teach myself something in the process.

PS – if you have the opportunity to attend CodeMash next year, I couldn’t recommend it enough!

Windows Azure & PHP (for nubs)–Part 2 of 4

Ok, took a bit longer to get back to this than planned, but we’re here. So lets get stated.

In part one of this series, I talked about the tooling and getting a simple test app running in Windows. In this edition, we’ll take that to the next step and make it a Windows Azure web role and get that deployed to the local development emulator.

Solution Scaffolds

Unlike the previous Eclipse toolkit I had used, the new Windows Azure SDK for PHP is entirely command line based. This will be a bit of a shock of those of you used to the integrated experience offered by Visual Studio. But for those familiar with powershell and heaven forbid, older devs like me that used to get by on PERL or KornShell, it’s a pretty comfortable place to be.

This command line centers around the use of scaffolds, or templates. You can use the bare bones, default template, or if you’re more ambitious, create your own custom scaffolds. I’ll leave the topic of creating custom templates to those more experienced (and interested). So lets just focus on the default template in the SDK and what is gives us.

Ben Lobaugh of Microsoft has a great set of tutorials over at the interoperability bridges site, including one on setting up your first Windows Azure and PHP project. As this points out, to create our first, bare bones app we just open up the command prompt and type this:

scaffolder run -out="C:\temp\WindowsAzurePHPApp"

 

This creates an initial project at the location provided and by default it’s a Windows Azure web role. Ben’s tutorial also breaks down the contents of that location (the solution files, subdirectories, etc…). But lets look at that command a bit closer.

 

The “scaffolder” utility has two commands, ‘run’ which executes a scaffold (or template) and ‘build’ which is used to create a new scaffold (again, we’re going skip creating a custom scaffold). The ‘run’ command supports two parameters, –OutputPath (aka –Out) and –Scaffolder (aka –s). We specified the out location and since we didn’t specify a scaffolder, it used the default “Scaffolders/DefaultScaffolder.phar” which sits just under where the scaffolder program sits (which on my machine is C:\Program Files\Windows Azure SDK for PHP). At least that’s what the’scaffolder’ command’s help tells us. But that’s not the entire story. The DefaultScaffolder.phar file is really just a php script, and it defines additional parameters (you can open it with the text editor of your choice).

 

For example, take this comment block from that phar file:

 

/**
 * Runs a scaffolder and creates a Windows Azure project structure which can be customized before packaging.
 *
 * @command-name Run
 * @command-description Runs the scaffolder.
 *
 * @command-parameter-for $scaffolderFile Argv –Phar Required. The scaffolder Phar file path. This is injected automatically.
 * @command-parameter-for $rootPath Argv|ConfigFile –OutputPath|-out Required. The path to create the Windows Azure project structure. This is injected automatically.
 * @command-parameter-for $diagnosticsConnectionString Argv|ConfigFile|Env –DiagnosticsConnectionString|-d Optional. The diagnostics connection string. This defaults to development storage.
 * @command-parameter-for $webRoleNames Argv|ConfigFile|Env –WebRoles|-web Optional. A comma-separated list of names for web roles to create when scaffolding. Set this value to an empty parameter to skip creating a web role.
 * @command-parameter-for $workerRoleNames Argv|ConfigFile|Env –WorkerRoles|-workers Optional. A comma-separated list of names for worker roles to create when scaffolding.
 */

 

We see in this parameters such as “-WebRoles”, “-WorkerRoles”, and even “-DiagnosticsConnectionString” (along with short names for each). And if we go look at this tutorial by Brian Swan (another great Windows Azure and PHP resource), you can see him using some of these parameters.

 

So what we see here is that the scaffolder can be used to quickly create various project templates for us. And if we customize the scaffolds (the php archive or .phar files), we can tailor them for our own unique needs. Powerful, if you don’t mind doing some scripting of your own.

 

Putting the “Azure” in

Alright, back to the bare bones sample project we created. We can easily put the same “phpinfo();” command into it, but that does us nothing. At least in a “Windows Azure” sense. What we really want to do is start leveraging the “Windows Azure” part of the SDK to put some cloud flavor into it.

 

First off, we need to locate the SDK location. In my case, its “C:\Program Files\Windows Azure SDK for PHP”. From there we go down a level into the “library” folder and copy the “Microsoft” folder from that location.  We don’t HAVE to copy everything in the Microsoft folder, but I’d rather not get drug down into mapping dependencies just yet. Suffice to say that that this location contains all the php include files and binary proxies we need to interact with most areas of the Windows Azure platform.

 

The root of these files AutoLoader.php, which helps resolve class names at runtime. This also helps remove the need for you as a developer to juggle lots of includes. If we drill down into the WindowsAzure folder, we’ll also find items like a SessionHandler, a proxy class for RoleEnvironment, and sub-libraries for things like Azure Storage.

 

But I digress, we’re going to copy the Microsoft folder into our sample web role. I put this file right into the .Web folder, same place my default index.php and Web.config files are located. I then update my index.php file to look like this:

<?php    require_once('Microsoft/AutoLoader.php');

    $val = Microsoft_WindowsAzure_RoleEnvironment::isAvailable();

    if ($val)        echo "in environment";    else        echo "not in environment";?>

What this is doing is referencing the AutoLoader, using it to pull in a reference to the SDK’s RoleEnvironment class, and calling a static method of that class, isAvailable, to determine if we are running in the Windows Azure environment or not. Now at this point we could load the site using the virtual directory we created last night, or we could deploy the project to local Windows Azure development emulator with a command like this:

 

package create -in="C:\temp\WindowsAzurePHPApp" -out="C:\temp\WindowsAzurePHPApp\build" -dev=true

This command uses the “package” program of the SDK to create a cloud service deployment package for Windows Azure using the project files at the location specified by the ‘-in’ parameter. Then, by specifying ‘-dev=true’, we tell the utility to deploy that package to the Windows Azure  development fabric. If we load out index.php file from the resulting URI, we should (hopefully, see note below) see that we’re now running in the Windows Azure environment.

 

Note: isAvailable does not always work properly in the local development emulator. It appears to be a bug that creeps up occasionally but is difficult to reproduce. It can also change from fixed to broken depending on the version of the emulator you’re using.

Some debugging tips

Now if you’re like me and bit of a php nub (and you likely are if you’re reading this), you’re going to have some php parser errors when working up this sample. PHP syntax is so close to C# .NET that it is easy to mis-type things when moving back and forth between the languages. When this happens, you’ll get a blank page. The reason for this is that by default PHP doesn’t display detailed errors to help protect your sites. Detailed error messages can provide information that could make it easier for people to hack into a web site.

 

So the first thing we need to do is locate the php.ini file that controls this setting. The easiest way to do this (thanks for this tip Maarten), is to create a page that has “phpinfo();” in it and then load it. Check the “Loaded Configuration File” location and that will show you which file you need to update. Open it with your favorite text editor and track down the “display_errors” setting and set it from “off” to “on”.

 

Now this change only impacts your local environment. If you look at the template project you created using the scaffold, it includes a php folder with a php.ini file already in it. Its fairly bare bones, but this file will be used when you deploy your project to Windows Azure. So any tweak you make to the way PHP is configured for your solution , need to be added to this file as well if you want then to be applied when its deployed out to the Microsoft datacenters.

 

And next time on Script Adventures

So that’s it for part 2. Next time, we’ll explore the deployment process and what happens inside Windows Azure. If I’m lucky, I’ll even be able to show you how to setup remote desktop so we can peak into the virtual machine hosting our application and look at how the deployment process was executed.

 

 

 

Windows Azure & PHP (for nubs)–Part 1 of 4

PHP was my first web development language. I got into web apps just prior to the “Dot Com” boom more than 10 years ago when I was doing “social networking” (we didn’t call it that back then) sites for online games. nfortunately, as a consultant, the skills I get to exercise are often “subject to the demands of the service”. And we don’t really get much call for PHP work these days. I did break those skills back out about a year ago for a project involving a more recent love, Windows Azure for a short yet very sweet reunion. But since then I’ve only gone back to it once or twice for a quick visit.

So when the call for speakers for CodeMash came up, I pitched a session on PHP and WIndows Azure. The topic is a good fit for the conference and I’m really looking forward to it. Fortunately, I have some time between engagements right now so I’m using it to brush up on my PHP+Azure skills (last used on a project almost a year ago). To help ensure that the session is in alignment with the latest developments.

My how things have changed in the last year.

Change in tooling

So when I worked with PHP last year, I relied on the Windows Azure  Tools for Eclipse. It’s still a great toolset that allows for the simple creation and deployment of Windows Azure apps. I loved the level of IDE integration they provided and “built in” support for deployments to the development emulator.

Part of the problem though is that in the last year, it appears that the PHP for Eclipse toolset has lost a bit of steam. Communication isn’t as good as it once was and updates aren’t as frequent. Still a good tool, but it really didn’t seem to be keeping pace with the changes in Azure.

So I ping’d an expert to find out what the latest and greatest was. Turns out things are going command line in a big way with the Windows Azure SDK for PHP. While we do lose the pretty GUI, I can’t say I’m really heart-broken. So lets start by walking through what you need.

Needed Tools

First up, we need to make sure we have the Web Platform Installer because we’re going to use it to snag some of the components we need. The platform installer is nice because it will make sure we have necessary pre-requisites installed and even download them for us if it can.

If you aren’t already a .NET developer, you want to look at start with getting SQL Server Express. Just launch the platform installer and type “SQL server express” into the search box in the top right. Look for “SQL Server Express 2008 R2” and select “install” if its not already.

Do this same thing except search for “Azure” and get the “Windows Azure SDK” and “Windows Azure Libraries”. Lastly, search for PHP and get the latest version of PHP for web matrix.

Lastly, we’ll need to download the PHP SDK for Azure and install it manually by unzipping the file to “C:\Program Files\Windows Azure SDK for PHP”.

Now there’s a lot more to this then what I’ve covered here. For additional, more detailed information I would direct to this this link on setting up PHP on Windows and this link on setting up the PHP SDK.

Our first PHP app

With all the bit installed, we want to do a quick test locally to make sure we have PHP installed and running properly. So fire up the Internet Information Services (IIS) manager (just type “IIS” into the Windows 7 search box) and in there, we’re going to drill down to the default web site and add some stuff in. Open up the branches like you see in the picture below and right click on “Default Web Site” and select “Add Virtual Directory…” from the pop-up menu.

I entered in “phpsample” as the Alias of my test web site and set the physical path to a location just below “C:\inetpub\wwwroot” (the default root location for IIS web sites. I then created a new file named “index.php” and placed it into that location. This file had only a single line of code…

<?php  phpinfo(); ?>

Now if you’re not familiar with PHP, this code will give us a dump of all the PHP settings in use by our system. And if we browse to the new web application (you can click on the browse link on the right in IIS Manager, we hopefully get output like this:

Next time on our show…

So that’s it for part 1 of this series. Next time (and hopefully later this week). We’ll create a Windows Azure web application and show how to deploy and test it locally. We’ve only scratched the surface here. So stay tuned! But if you can’t wait, check out Brian Swan’s PHP on Windows Azure Learning path post.

Until next time.

Long Running Queue Processing Part 2 (Year of Azure–Week 20)

So back in July I published a post on doing long running queue processing. In that post we put together a nice sample app that inserted some messages into a queue, read them one at a time and would take 30 seconds to process each message. It did processing in a background thread so that we could monitor it.

This approach was all good and well but hinged on us knowing the maximum amount of time it would take us to process a message. Well fortunately for us in the latest 1.6 version of the Azure Tools (aka SDK), the storage client was updated to take advantage of the new “update message” functionality introduced to queues by an earlier service update. So I figured it was time to update my sample.

UpdateMessage

Fortunately for me given the upcoming holiday (which doesn’t leave my time for blogging given that my family lives in “the boonies” and haven’t yet opted for an internet connection much less broadband, updating a message is SUPER simple.

myQueue.UpdateMessage(aMsg, new TimeSpan(0, 0, 30), MessageUpdateFields.Visibility);

All we need is the message we read (which contains the pop-receipt the underlying API use to update the invisible mssage), the new timespan, and finally a flag to tell the API if we’re updating the message content/payload or its visibility. In the sample above we of course are setting its visibility.

Ok, time for turkey and dressing! Oh wait, you want the updated project?

QueueBackgroundProcess w/ UpdateMessage

Alright, so I took exactly the same code we used before. It inserts 5 messages into a queue, then reads and processes each individually. The outer processing loop looks like this:

while (true)
{
    // read messages from queue and process one at a time…
    CloudQueueMessage aMsg = myQueue.GetMessage(new TimeSpan(0,0,30)); // 30 second timeout
    // trap no mesage.
    if (aMsg != null)
    {
        Trace.WriteLine("got a message, '"+aMsg.AsString+"'", "Information");
        
        // start processing of message
        Work workerObject = new Work();
        workerObject.Msg = aMsg;
        Thread workerThread = new Thread(workerObject.DoWork);
        workerThread.Start();

        while (workerThread.IsAlive)
        {
            myQueue.UpdateMessage(aMsg, new TimeSpan(0, 0, 30), MessageUpdateFields.Visibility);
            Trace.WriteLine("Updating message expiry");
            Thread.Sleep(15000); // sleep for 15 seconds
        }

        if (workerObject.isFinished)
            myQueue.DeleteMessage(aMsg.Id, aMsg.PopReceipt); // I could just use the message, illustraing a point
        else
        {
            // here, we should check the queue count
            // and move the msg to poison message queue
        }
    }
    else
        Trace.WriteLine("no message found", "Information");

    Thread.Sleep(1000);
    Trace.WriteLine("Working", "Information");
}

The while loop is the processor of the worker role that this all runs in. I decreased the initial visibility timeout from 2 minutes to 30 seconds, increased our monitoring of the background processing thread from every 1/10th of a second to 15 seconds, and added the updating of the message visibility timeout.

The inner process was also upped from 30 seconds to 1 minute. Now here’s where the example kicks in! Since the original read only listed a 30 second visibility timeout, and my background process will take one minute, its important that I update the visibility time or the message would fall back into view. So I’m updating it with another 30 seconds every 15 seconds, thus keeping it invisible.

Ta-da! Here’s the project if you want it.

So unfortunately that’s all I have time for this week. I hope all of you in the US enjoy your Thanksgiving holiday weekend (I’ll be spending it with family and not working thankfully). And we’ll see you next week!

Cloud Computing as a Financial Institution

Ok, I’m sure the title of this post is throwing you a bit but please bear with.

I’ve been travelling the last few weeks. I’m driving as its only about 250 miles away. And driving unlike flying leaves you with a lot of time to think. And this train of thought dawned on me yesterday as I was cruising down I35 south somewhere between Clear Lake and Ames in northern Iowa.

The conventional thinking

So the common theme you’ll often hear when doing “intro to cloud” presentations is comparing it to a utility. I’ve done this myself countless times. This story goes that as a business owner, what you need is say light. Not electricity, no a generator, not a power grid.

Just like the you utility company manages the infrastructure to delivery power to your door so when you turn on the switch, you get the light you wanted. You don’t have to worry about how it gets there. Best yet, you only pay for what you use. No needs to spend hundreds of millions on building a power plant and the infrastructure to deliver that power to your office.

I really don’t have an issue with this comparison. Its easy to relate to and does a good job of illustrating the model. However, what I realized as I was driving is that this example is a one way example. I’m paying a fee and getting something in return. But there’s no real trust issue except in the ability for my provider to give me the service.

Why a financial institution?

Ok, push aside the “occupy” movement and the recent distrust of bankers. A bank is where you put assets for safe keeping. You have various services you get from the provider (atm, checking account) that allow you to leverage those assets. You also have various charges that you will pay for using some of those services while others are free. You have a some insurance in place (FDIC) to help protect your assets.

Lastly, and perhaps most importantly, you need to have a level of trust in the institution. You’re putting your valuables in their care. You either need to trust that they are doing what you have asked them to do, or that they have enough transparency that you can know exactly what’s being done.

What really strikes me about this example is you having some skin in the game and needing to have a certain level of trust in your provider. Just like you trust the airline to get you to your destination on time, you expect your financial provider to protect your assets and deliver the services they have promised.

It’s the same for a cloud provider. You put your data and intellectual property in their hands, or you keep in under your mattress. Their vault is likely more secure then your box spring, but its what you are familiar with and trust. Its up to you to find cloud provider you can trust. You need to ask the proper questions to get to that point. Ask for a tour of the vault, audit their books so to speak. Do your homework.

Do you trust me?

So the point of all this isn’t to get a group of hippies camped out on the doorstep of the nearest datacenter. Instead,the idea here is to make you think about what you’re afraid of, especially when you’re considering a public cloud provider. Cloud Computing is about trusting your provider but also having responsibility for make sure you did your homework. If you’re going to trust someone with your most precious possessions, be sure you know exactly how far you can trust them.

Enhanced Visual Studio Publishing (Year of Azure–Week 19)

With the latest 1.6 SDK (ok, now its actually called Azure Authoring Tools), Scott Guthrie’s promise of a better developer publishing experience has landed. Building upon the multiple cloud configuration options that were delivered back in September with the Visual Studio tools update, we have an even richer experience.

Now the first thing you’ll notice is that the publish dialog has changed. The first time you run it, you’ll need to sign in and get things set up.

Clicking the “Sign in to download credentials” link will send you to the windows.azure.com website where a publish-settings file will be generated for you to download. Following the instructions, you’ll download the file, then import it into the publishing window shown above. Then you can chose a subscription from the populated drop down and proceed.

A wee bit of warning on this though. If you have access to multiple subscriptions (own are or a co-admin), the creation of a publish-settings file will install the new certificate in each subscription. Additionally, if you click the the “Sign in to download”, you will end up with multiple certs. These aren’t things to be horrified about it, just wanted to make sure I gave a heads up.

Publish Settings

Next up is the publication settings. Here we can select a service to deploy too or create a new one (YEAH!). You can also easily set the environment (production or staging), the build configuration, and the service configuration file to be used. Setting up remote desktop is also as easy as a checkbox.

In the end, these settings to get captured into a ‘profile’ that is saved and can then be reused. Upon completion, the cloud service will get a new folder, “Profiles”. In this folder you will find an xml file with the extension azurePubxml that contains the publication settings.

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <AzureCredentials>my subscription</AzureCredentials>
    <AzureHostedServiceName>service name</AzureHostedServiceName>
    <AzureHostedServiceLabel>service label</AzureHostedServiceLabel>
    <AzureSlot>Production</AzureSlot>
    <AzureEnableIntelliTrace>False</AzureEnableIntelliTrace>
    <AzureEnableProfiling>False</AzureEnableProfiling>
    <AzureEnableWebDeploy>False</AzureEnableWebDeploy>
    <AzureStorageAccountName>bmspublic</AzureStorageAccountName>
    <AzureStorageAccountLabel>bmspublic</AzureStorageAccountLabel>
    <AzureDeploymentLabel>newsdk</AzureDeploymentLabel>
    <AzureSolutionConfiguration>Release</AzureSolutionConfiguration>
    <AzureServiceConfiguration>Cloud</AzureServiceConfiguration>
    <AzureAppendTimestampToDeploymentLabel>True</AzureAppendTimestampToDeploymentLabel>
    <AzureAllowUpgrade>True</AzureAllowUpgrade>
    <AzureEnableRemoteDesktop>False</AzureEnableRemoteDesktop>
  </PropertyGroup>
</Project>

This file contains a reference to a storage account and when I looked at the account I noticed that there was a new container in there called “vsdeploy”. Now the folder was empty but I’m betting this is where the cspkg was sent to before being deployed and subsequently deleted. I only wish there was an option to leave the package there after deployment. I love having old packages in the cloud to easily reference.

If we go back into the publish settings again (you may have to click “previous” a few times to get back to the “settings” section_ and can select “advanced” you can set some of the other options in this file. Here we can set the storage account to be used as well as enable IntelliTrace and profiling.

The new experience does this using a management certificate that was created for us at the beginning of this process. If you open up the publish settings file we downloaded at the beginning, you’ll find its an XML document with an encoded string representing the management certificate to be used. Hopefully in a future edition, I’ll be able to poke around at these new features a bit more. It appears we may have one of more new API’s at work as well as some new options to help with service management and build automation.

What next?

There’s additional poking around I need to do with these new features. But there’s some great promise here. Out of the box, developers managing one or two accounts are going to see HUGE benefits. For devs in large, highly structured and security restricted shops, they’re more likely to keep to the existing mechanisms or looking at leveraging this to enhance their existing automated processes.

Meanwhile, I’ll keep poking at this a little bit as well as the other new features of this SDK and report back when I have more.

But that will have to wait until next time.

SQL Azure Throttling & Error Codes (Year of Azure–Week 18)

Ok, last week was weak. Hopefully my grammar is correct. So I want to make it up too you by giving you some meat this week. I’ve been involved lately in several discussions regarding SQL Azure capacity limits (performance, not storage) and throttling. Admittedly, this little topic, often glanced over, has bitten the butts of many an Windows Azure project.

So lets look at it a little more closely. Shall we?

Types of Throttling

Now if you go read that link I posted last week on SQL Azure Error Messages, you’ll see that there are two types of throttling that can occur:

Soft Throttling – “kick in when machine resources such as, CPU, IO, storage, and worker threads exceed predefined thresholds”

Hard Throttling – “happens when the machine is out of a given resource”

Now SQL Azure is a multi-tenant system. This means that we have multiple tenants (databases) that will sit on the same physical server. There cold be several hundred databases that share the same hardware. Now SQL Azure will use soft throttling to try and make sure that all these tenants get a minimum number of resources. When a tenant starts pushing those limits, soft throttling will kick in and the SQL Azure fabric will try to move tenants around to rebalance the load.

Hard throttling means no new connections. You’ve maxed things out (storage space, worker process, cpu) and drastic steps should be taken to free those resources up.

SQL Azure Resources

Also in that article, we find the various resource types that we could get throttled on:

• Physical Database Space
• Physical Log Space
• LogWriteIODelay
• DataReadIODelay
• CPU
• Database Size
• Internal
• SQL Worker Threads
• Internal

Now when an error message is returned, you’ll have throttling types for one or more of these resources. The type could be “no throttling”, could be “soft”, or it could be “hard”.

Throttling Modes

Now if all this wasn’t confusing enough, we have three types of throttling. Well technically four if you count “no throttling”.

Update/Insert –  can’t insert/update, or create anything. But can still drop tables, delete rows, truncate tables, read rows.

All Writes – all you can do is read. You can’t even drop/truncate tables.

Reject All – you can’t do anything except contact Windows Azure Support for help.

Now unlike the throttling type, there is only one mode returned by the reason.

 

But what can we do to stop the errors?

Now honestly, I’d love to explain Andrew’s code to you for deciphering the codes. But I was never good with bitwise operations. So instead, I’ll just share the code and along with a sample usage. I’ll leave it to folks that are better equipped to explain exactly how it works.

The real question is can we help control throttling? Well if you spend enough time iterating through differing loads on your SQL Azure database, you’ll be able to really understand your limits and gain a certain degree of predictability. But the sad thing is that as long as SQL Azure remains a shared multi-tenant environment, there will always be situations where you get throttled. However, those instances should be a wee bit isolated and controllable via some re-try logic.

SQL Azure is a great solution, but you can’t assume the dedicated resources you have with on-premises SQL Server solution. You need to account for variations and make sure your application is robust enough to handle intermittent failures. But now we’re starting down a path of trying to design to exceed SLA’s. And that’s a topic for another day.

SQL Azure Error Codes (Year of Azure–Week 17)

Ok, I’m no counting last week as a ‘Year of Azure’ post. I could,but I feel it was even too much of a softball for me to bare. Unfortunately, I was even less productive this week in getting a new post out. I started a new client and the first weeks, especially when travelling are horrible for me doing anything except going back to the hotel and sleeping.

However, I have spent time the last few week working over a most difficult question. The challenges of SQL Azure throttling behaviors and error reporting.

Reference Materials

Now, on the surface SQL Azure is a perfect wonderful relational database solution. However, when you begin subjecting it to a significant load, its limitations start becoming apparent. And when this happens, you’ll find you get back various error codes that you have to decode.

Now, I could dive into an hours long discussion regarding architectural approaches for creating scalable SQL Azure data stores. A discussion mind you which would be completely enjoyable, very thought provoking, and for which I’m less well equipped then many folks (databases just aren’t my key focus, I leave those to better…. er…. more interested people *grin*). For a nice video on this, be sure to check out the TechEd 2011 video on the subject

Anyways…

Deciphering the code

So if you read the link on error codes, you’ll find that there’s several steps that need to be decoded. Fortunately for me. While I have been fairly busy, I have access to a resource that wasn’t. One fairly brilliant Andrew Espenes. Now Andrew was kind enough to take on a task for me and look at deciphering the code. And in a show of skill that demonstrates to me I’m becoming far older then I would like to believe,

Anyways, pulled together some code that I wanted to share. Some code that leverages a technique I haven’t used since my college days of developing basic assembly (BAL) code. Yes, I am that old.

So lets fast forward down the extensive link I gave you earlier to the “Decoding Reason Codes” section. And our first stop will actually be adjust the reason code into something usable.  The MSDN article says to apply modulo 4 to the reason code:

ThrottlingMode = (AzureReasonDecoder.ThrottlingMode)(codeValue % 4);

Next determine the resource type (data space, CPU, Worker Threads, etc…):

int resourceCode = (codeValue / 256);

And finally, we’ll want to know the throttling type (hard vs. soft):

int adjustedResourceCode = resourceCode & 0×30000;
adjustedResourceCode = adjustedResourceCode >> 4;
adjustedResourceCode = adjustedResourceCode | resourceCode;
resourceCode = adjustedResourceCode & 0xFFFF;
ResourcesThrottled = (ResourceThrottled)resourceCode;

Next Time

Now I warned you that I was short on time, and while I have some items I’m working on for future updates I do want to spend some time this weekend with family. So I need to hold some of this until next week when I’ll post a class Andrew created for using these values and some samples for leveraging them.

Until next time!