Scott Hanselman

A/B Testing and Testing In Production with Azure Web Apps

July 17, 2015 Comment on this post [6] Posted in Azure
Sponsored By

I've got a lot of production web sites running in Azure right now. Some are for small side projects and some are larger like the sites for the Hanselminutes Podcast and This Developer's Life. I like Web Apps/Sites (which is Platform as a Service) rather than Virtual Machines (Infrastructure as a Service) because I don't like thinking about the underlying operating system if I can avoid it. I like to be able to scale the site up (faster, bigger) or out (more machines in the farm) with a slider bar.

In fact, there's some other more advanced and useful features that Azure Web Apps have that keep me using Web Apps almost exclusively.

I'll use a little site I made called that tells you how many keystrokes are left in your hands before you die. Think of it as a productivity awareness tool.

First, I'll add a Deployment Slot to my existing Git-deployed Web App. The source for KeysLeft lives in GitHub here. When I check-in a change it's automatically deployed. But what if I wanted to have a staging branch and automatically deploy to a first? If it works out, then move it to production by swapping sites. That'd be sweet.

Staging Slots for Azure Web Apps

You can see here my main KeysLeft web app has a Staging "side car" app that is totally separate but logically related/adjacent to production. Notice the "swap" button in the toolbar. Love it.

Adding Deployment Slots to an Azure Web App

This Web App has its configuration copied from the main one, and I can setup Continuous Deployment to pull from a different branch, like "staging" for example. The name of the deployment slot becomes a suffix, so unless you set up a custom CNAME like You can have up to 4 deployment slots in addition to production (so dev, test, staging, whatever, production) on Standard Web Apps.

A/B Testing for Azure Web Apps

Once I've got a slot or two set up and running a version of my app, I can do A/B testing if I'd like. I can set up a feature that was called "Testing in Production" and is now "Traffic Routing" and tell Azure what percentage of traffic goes to prod and what goes to staging. Of course, you have to be sure to write your application so such that authentication and session is managed however is appropriate, especially if you'd like the user to have a seamless experience.

Here I've got 10% of the traffic going to staging, seamlessly, and the other 90% is going to production. I can make a small change (background color for example) and then hit the main site over and over and see the occasional (10% of course) request being routed to the staging slot. You can configure this static routing however you'd like.

10% Traffic to Staging

Then I could hook up Application Insights or New Relic or some other event/diagnostics system and measure the difference in user reaction between features that changed.

Advanced Testing in Production

Made it this far? Then you're in for a treat. Static routing is cool, to be clear, but scripting a more dynamic experience is even more interesting. Galin Iliev, one of the developers of this feature, gave me this Powershell script to show off more powerful stuff.

First, you can use PowerShell to manage this stuff. You can change routing values and ramp up or ramp down. For example, here we start at 10% and change it by 5 every 10 minutes.

# Select-AzureSubscription YOURSGOESHERE

$siteName = "keysleft"
$rule1 = New-Object Microsoft.WindowsAzure.Commands.Utilities.Websites.Services.WebEntities.RampUpRule
$rule1.ActionHostName = ""
$rule1.ReroutePercentage = 10;
$rule1.Name = "staging"

$rule1.ChangeIntervalInMinutes = 10;
$rule1.ChangeStep = 5;
$rule1.MinReroutePercentage = 1;
$rule1.MaxReroutePercentage = 80;

Set-AzureWebsite $siteName -Slot Production -RoutingRules $rule1

But! What if you could write code to actually make the decision to continue or fall back dynamically? You can add a callback URL and a Site Extension called the "TiP Callback Extension."

$rule1.ChangeDecisionCallbackUrl =

The Site Extension (and all Site Extensions for that matter) is just a little sidecar Web API. This callback gets a small POST when it's time to make a decision, and you decide what to do based on HTTP-related context that was passed in and then return a ChangeDirectionResult object as JSON. You can adjust traffic dynamically, you can adjust traffic when doing a deployment, do a slow, measured roll out, or back off if you detect issues.

NOTE: The ChangeDescisionCallbackUrl and this code below is totally optional (so don't stress) but it's super powerful. You can just do static routing, you can do basic scripted dynamic traffic routing, or you can have make a decision callback URL. So the choice is yours.

You can check out the code by visiting after installing the TiP callback site extension and look at the Site Extensions folder. That said, here is the general idea.

using System.Web.Http;
using TipCallback.Models;

namespace TipCallback.Controllers
public class RoutingController : ApiController
public ChangeDirectionResult GetRoutingDirection([FromBody] RerouteChangeRequest metrics)
// Use either Step or RoutingPercentage. If both returned RoutingPercentage takes precedence
return new ChangeDirectionResult
Step = (int)metrics.Metrics["self"].Requests,
RoutingPercentage = 10

Here's the object you return. It's just a class with two ints, but this is super-annotated.

/// <summary>
/// Return information how to change TiP ramp up percentage.
/// Use either Step or RoutingPercentage. If both returned RoutingPercentage takes precedence
/// Either way MinRoutingPercentage and MaxRoutingPercentage set in API rule are in force
/// </summary>
public class ChangeDirectionResult
/// <summary>
/// Step to change the Routing percentage. Positive number will increase it routing.
/// Negative will decrease it.
/// </summary>
[DataMember(Name = "step")]
public int? Step { get; set; }

/// <summary>
/// Hard routing percentage to set regardless of step.
/// </summary>
[DataMember(Name = "routingPercentage")]
public int? RoutingPercentage { get; set; }

All this stuff is included in Standard Azure Web Apps so if you're using Standard apps (I have 19 websites running in my one Standard plan) then you already have this feature and it's included in the price. Pretty cool.

Related Links

Sponsor: Big thanks to Infragistics for sponsoring the feed this week. Responsive web design on any browser, any platform and any device with Infragistics jQuery/HTML5 Controls.  Get super-charged performance with the world’s fastest HTML5 Grid - Download for free now!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Hosting By
Hosted in an Azure App Service
July 17, 2015 12:36
Thanks for the article Scott.
For more references, there's an article about TiP in my blog and a video on ch9
July 17, 2015 18:03
Hey Scott, just a quick thought about the ad at the bottom - if it says Download for free now I would expect a bit more than a free trial. But if it keeps this blog going... ;)
July 17, 2015 18:10
Scott I don't understand how do you have 19 sites,

All this stuff is included in Standard Azure Web Apps so if you're using Standard apps (I have 19 websites running in my one Standard plan) then you already have this feature and it's included in the price. Pretty cool.

I have 1 site in shared mode in Azure and I pay around 13-15 euro/monthly, can you please share how can you afford 19 sites? Or is it because of your MSDN subscription?
July 17, 2015 22:46
@Timur you can have I think 100 websites under 1 standard subscription.
July 19, 2015 15:38
@timur, see

@al, it's 500 websites.
July 21, 2015 12:20
Anyone know if it is possible to have people (either on a per browser or ideally per user account basis) opt into the testing instance?

I understand that a cookie is set that determines the routing. Has anyone ever integrated that with an opt-in beta process?

Would be a great way to get feedback from our users that want to be on the bleeding edge

Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.