Scott Hanselman

How to setup a Load Balanced Web Farm of Virtual Machines (Linux or otherwise) on Windows Azure (command line)

May 22, 2013 Comment on this post [19] Posted in Azure
Sponsored By

A buddy of mine was thinking to move some of his Linux-based website to Azure. If you're running a Web Site that this node.js, ASP, ASP.NET, or PHP, it's easiest to use Azure Web Sites. I showed how to do setup Azure Web Sites in minutes with Git in this post. They hide the underlying OS from you, are automatically updated, scale easily, and share disks.

However, he likes VMs and the control they give him, plus he can run whatever he wants, move things around and generally control his world.

I'll be using the open source (GitHub) cross platform CLI tools (command line interface) for Azure. If you have node package manager you can "npm install azure-cli --global" then import your subscription. You can also get the command line tools by downloading and installing from www.windowsazure.com.

Setting up a Linux VM

We'll create a the initial VM using the Portal (I'll show you how to do it from the command line in a minute). This virtual machine will be for setting up a template VM image. I'm going to create an Ubuntu 13.04 server, then add Apache and PHP. Then I'll capture a generic image of my now-set-up machine and use it to create copies that I'll add to a farm. This image will show up later in "My Images" in the Azure Portal.

NOTE: There's an amazing community-driven category of prepped and ready Virtual Machine images at the Open Tech VM Depot. It's like the Azure's best kept secret. I'll do a post on it later but it's really deep and broad and worth checking out.

From within the Azure Portal I'll go New | Virtual Machine | and select Ubuntu Server 13.04.

Selecting Ubuntu from the Azure Gallery

The name doesn't matter, but I'll setup a user name and password (or use a SSH key):

Creating a VM

Note I'll create a stand-alone Virtual Machine:

Making a Stand-Alone VM

TIP: If I didn't want to use the Portal at all to make this VM, I could even find a VM image programmatically, then create a VM instance, all from the command line. Using "azure vm list" would get me the list of VMs to choose from. The Ubuntu one I wanted is "Ubuntu-13_04-amd64-server-20130501-en-us-30GB" (with a guid in the name as well) so I'd just "azure vm create MyDNSName ImageName [options]" and then proceed from there.

Once this Linux VM has started up, I'll SSH into it. You can see that Azure has mapped a random high number public port to the VM's internal SSH port 22.

My VM's IP Address

I SSH in. I'm gonna add Apache, PHP, restart apache, then add a test.php that will show PHP is working as well as output the current IP address so I can tell which machine served the request.

sudo apt-get install apache2
sudo apt-get install libapache2-mod-php5 php5

I'll add a test.php

sudo nano /var/www/test.php

and put in

<?php echo gethostbyname(trim(`hostname`)); ?>
<?php phpinfo(); ?>

Then I'll exit SSH. VMs are locked down by default, so to test this I need to add an endpoint. I can do that via the Portal but I'd like to see what I can do from the Azure command line.

Run vm endpoint create to map external 80 to internal 80.

azure vm endpoint create mylinuxtemplate 80 80

At this point I should be able to hit mylinuxtemplate.cloudapp.net:80/test.php and see it work.

My VM's test.php page

Cool. So I've got my Linux VM template the way I want it. Now I want to "capture it" as an image so I can stamp out more of them. This "waagent" on Linux is like "sysprep" on Windows.

From within a SSH session, run waagent -deprovision.

~$ sudo waagent -deprovision
scott@mylinuxtemplate:~$ sudo waagent -deprovision
WARNING! The waagent service will be stopped.
WARNING! All SSH host key pairs will be deleted.
WARNING! Nameserver configuration in /etc/resolvconf/resolv.conf.d/{tail,originial} will be deleted.
WARNING! Cached DHCP leases will be deleted.
WARNING! root password will be disabled. You will not be able to login as root.
Do you want to proceed (y/n)? y

WINDOWS PEOPLE: If you're a Windows person, you can setup your Windows VM just as you like it, then run %windir%\system32\sysprep.exe on it, capture an image of the VM and do everything described in this post as well!

I could shut it down and capture an image from the Portal, but again, it's command line fun today. Note that shutting down can take a little while.

azure vm shutdown mylinuxtemplate
azure vm capture mylinuxtemplate hanselmanlinuxwebfarmimage --delete

The capture command will DELETE the Virtual Machine. Remember that it was just a template. However, how I have a reusable image! I can see the images available to me, both user images and gallery images with "azure vm list."

NOTE: When you delete Virtual Machines, you're just deleting the "configuration" of the VM. You're not deleting the disk that was associated with it, as it's possible you might want to start that VM up again. If you're really trying to remove things, make sure you delete the instance of the VM and then delete the disk, too.

Creating a Linux VM Farm from the command line

Now I have an image sitting in my storage account that I can use to make "n" VMs from. I'll make one VM to start. I can watch it startup with "azure site list" after making it. When it's ready, I can make more! Make sure you use the --ssh switch or you will NOT be able to SSH into the machine!

C:\> azure vm create hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott MyPassword123 --location "West US" --ssh
info: Executing command vm create
+ Looking up image
+ Looking up cloud service
+ Creating cloud service
+ Creating VM
info: vm create command OK
C:\> azure vm list
info: Executing command vm list
+ Fetching VMs
data: DNS Name VM Name Status
data: ------------------------------- ------------------ ----------
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm CreatingVM
info: vm list command OK
C:\> azure vm list
info: Executing command vm list
+ Fetching VMs
data: DNS Name VM Name Status
data: ------------------------------- ------------------ ---------
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm ReadyRole
info: vm list command OK

WEIRD: Azure has a concept called a "Cloud Service" which is a lousy name, IMHO. For us, let's consider it a "container" for our VMs. It's a logical container that will hold and associate all our VMs (and other cloudy stuff) together. When you have one VM you have one cloud service associated with it, but you can't see it in the Portal but because it doesn't really provide value...yet. When you have TWO VMs in the same container, then you'll see that cloud service "container" appear in the Portal.

I've made a hanselmanlinuxfarm VM now so there's a hanselmanlinuxfarm cloud service 'container.' Now, I'm going to make a few more VMs but I'll connect them to the first VM. There's two ways to do this. First, the --connect option from the command line. Note that you don't have control over the name of your VM this way, if you care. If you have hundreds, you likely don't.

The command will find the existing cloud service (again, 'container') then make a new VM. I'm going to run this command twice so I'll have three VMs total.

SO IMPORTANT: It's the --connect used on this second call that is the key. It makes the second (and then n+1) VM and adds it to the same cloud service "container." It seems the VMs associated with each other. The name of the next VM will be whatever-2, then -3, etc but they will also use the same external name, like hanselmanlinuxfarm.cloudapp.net.

I'll do this twice, each time using a different high SSH port number that will map to 22 internally. If I don't want SSH expose externally, I can delete the public endpoint later.

C:\> azure vm create --connect hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott MyPassword --ssh 12345
info: Executing command vm create
+ Looking up image
+ Looking up cloud service
+ Getting cloud service properties
+ Looking up deployment
+ Creating VM
info: vm create command OK

When creating a Linux VM you MUST add a --ssh flag to the command line or you'll not be able to SSH into the box. Make sure to add a high port number so you'll get a mapping from that port to 22, so maybe 12346 -> 22, etc. If you make these VMs from the Portal, it will pick a random port for you. When you do it from the command line, you need to choose.

At this point, azure vm list says I have three. Two are ready and the last is being created now. You can tell these VMs are running in the same Cloud Service "container" because the DNS name is the same. These VMs are officially a "farm."

C:\> azure vm list
info: Executing command vm list
+ Fetching VMs
data: DNS Name VM Name Status
data: ------------------------------- -------------------- ----------
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm ReadyRole
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm-2 ReadyRole
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm-3 CreatingVM
info: vm list command OK

OK, now here's making a fourth VM from the Portal, just as an FYI.

Creating a Linux VM and adding to the Farm from the Portal

Just so you know, you can add VMs to your farm from the Portal also.

Now I can reuse this VM image

Give your new VM a name, then "connect it to an existing virtual machine." I don't like this phrasing, and I'm curious what you think. Basically it's "add this VM to this collection of VMs." It doesn't matter which one you select from this dropdown, as long as you pick one that's in the Cloud Service "container".

Connecting to an existing VM within a Farm

I won't click OK, but if I did, at this point I've would've a fourth VM, this one via the Portal.

Load Balancing my Linux VM Farm

I have three identical VMs running Apache and PHP and my test.php page.

C:\> azure vm list
info: Executing command vm list
+ Fetching VMs
data: DNS Name VM Name Status
data: ------------------------------- -------------------- ---------
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm ReadyRole
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm-2 ReadyRole
data: hanselmanlinuxfarm.cloudapp.net hanselmanlinuxfarm-3 ReadyRole

info: vm list command OK

Let me open up port 80 on all three.  Since I want them load balanced I can't do this, as this is how you map single non-load-balanced ports.

azure vm endpoint create hanselmanlinuxfarm   80 80
azure vm endpoint create hanselmanlinuxfarm-2 80 80
azure vm endpoint create hanselmanlinuxfarm-3 80 80

For load balanced ports I need to use "create-multiple." Not only may I want to open multiple ports all at once, but also since I want load-balancing, I also may want a probe set up. For example, since I'm using HTTP, if there's a result other than 200 returned from test.php then I'll want that VM pulled out of the farm. It will also pull the VM out of rotation if it doesn't hear back in 30 seconds.

Here I'm creating those 80 to 80 maps, but also watching test.php for anything other than an HTTP 200.

azure vm endpoint create-multiple hanselmanlinuxfarm   80:80:HttpTrafficIn:http:80:/test.php
azure vm endpoint create-multiple hanselmanlinuxfarm-2 80:80:HttpTrafficIn:http:80:/test.php
azure vm endpoint create-multiple hanselmanlinuxfarm-3 80:80:HttpTrafficIn:http:80:/test.php

In this case "HttpTrafficIn" is what I am calling the Load Balancing Set Name.

NOTE: I'm doing a pull request now to add the ability to see the ProbePath from the endpoint command but for now you can see it with "azure vm show" like this:

C:\>azure vm show hanselmanlinuxfarm
info: Executing command vm show
+ Fetching VM
data: DNSName "hanselmanlinuxfarm.cloudapp.net"
data: VMName "hanselmanlinuxfarm"
data: IPAddress "100.68.xx.xx"
data: InstanceStatus "RoleStateUnknown"
data: InstanceSize "Small"
data: InstanceStateDetails ""
data: OSVersion ""
data: Image "hanselmanlinuxwebfarmimage"
data: DataDisks ""
data: Network Endpoints 0 LoadBalancedEndpointSetName "HttpTrafficIn"
data: Network Endpoints 0 LocalPort "80"
data: Network Endpoints 0 Name "endpname-80-80"
data: Network Endpoints 0 Port "80"
data: Network Endpoints 0 LoadBalancerProbe Path "/test.php"
data: Network Endpoints 0 LoadBalancerProbe Port "80"
data: Network Endpoints 0 LoadBalancerProbe Protocol "http"
data: Network Endpoints 0 Protocol "tcp"
data: Network Endpoints 0 Vip "137.135.xx.xx"
info: vm show command OK

Cool, so now let's see if I have a load-balanced farm.

PERF NOTE: In order to get the best performance from your Azure VMs (or any cloud VM) considering putting things like MySQL/PostgreS databases on a separate disk with different caching semantics. You want the OS disk and the Data Disks to be separate. For example. I have a Windows VM running MySQL. The OS is on a standard 30 gig disk, but the MySQL Database is alone on a 5 gig disk that's attached. It keeps things separate and tidy, plus it performs better

Checking on my new Farm

If I log into the Portal, I can look at each individual VM or I can look at the farm as if it's one 'cloud service.' Get it?

My Linux Farm working as a team

All three VMs are "running"

Making my Farm more reliable

I want to make sure my new VMs are all on different racks in the Azure Data Center. I know they are in "West US" because I put them there, but I'm not sure if they are together on the same rack or near each other or what.  Since a rack is within a "fault domain" meaning that a Rack could, I dunno, spontaneously explode, then I want to tell Azure that all these VMs are part of an "availability set." This is a name I apply to the VMs that says "make these more available by keeping them apart."

From the Portal I'm going to pick the first VM and select Configure, then Create an Availability Set. I'll name it "hanselmanlinux" but it can be anything.

I'm making an availability set

Adding an Availability Set can involve Azure moving my VM somewhere else within the Data Center and it may need to restart it if it does. Sometimes this is fast, other times it takes a minute or 10 as it's a big deal initially, so be aware. Once everyone's in the set, it's less of a big deal.

When it's done, head over to the other VMs and add them one at a time to the same availability set. The result looks like this in the Portal, and now I'm assured that my three VMs are all in different Fault Domains (and racks).

All my VMs are in one Availability Set now

Hitting my Web Farm

Now I can hit hanselmanlinuxfarm.cloudapp.net/test.php and see the IP changing (as well as the CPU% changing in my portal!) or even watch HTOP over SSH and get a live view. Hey, I've got a little Linux farm!

My tiny farm has three machines

Here's my SSH'ed into one of them, looking at htop (it's better than top!)

SSH'ed into a Linux machine on Azure looking at CPU time with HTOP

My Complete Script, Summarized

Here's my complete script. I used azure vm image list | find /I "13_04" to find an Ubuntu image. I could have done this with bash as well.

C:\>azure vm image list | find /i "13_04"
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-13_04-amd64-server-20130423-en-us-30GB
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-13_04-amd64-server-20130501-en-us-30GB
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-raring-13_04-amd64-server-20130511-en-us-30GB
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-raring-13_04-amd64-server-20130515-en-us-30GB
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-raring-13_04-amd64-server-20130517-en-us-30GB
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-raring-13_04-amd64-server-20130518-en-us-30GB
data: b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu_DAILY_BUILD-raring-13_04-amd64-server-20130521-en-us-30GB

Once I've found an image, I create my first VM from the command line in a location of my choice. Again, it's Linux, don't forget the -ssh.

azure vm create mylinuxtemplate b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-13_04-amd64-server-20130501-en-us-30GB scott MyPassword --location "West US" --ssh

Get it how you like it. SSH in, set it up, run the waagent to prep it. Shut it down and capture it.

azure vm shutdown mylinuxtemplate
azure vm capture mylinuxtemplate mylinuxfarmimage --delete

Finally, here's a basic batch file to make 5 VMs. Note the first command is different from the Nth. Of course, with bash you could make a script like "spinup 5" and automate to your heart's content. The HTTP probe is optional on the endpoint creation.

azure vm create hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott password --location "West US" --ssh
azure vm create --connect hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott password --ssh 12345
azure vm create --connect hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott password --ssh 12346
azure vm create --connect hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott password --ssh 12347
azure vm create --connect hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott password --ssh 12348
azure vm endpoint create-multiple hanselmanlinuxfarm 80:80:HttpTrafficInLhttp:80:/test.php
azure vm endpoint create-multiple hanselmanlinuxfarm-2 80:80:HttpTrafficInLhttp:80:/test.php
azure vm endpoint create-multiple hanselmanlinuxfarm-3 80:80:HttpTrafficInLhttp:80:/test.php
azure vm endpoint create-multiple hanselmanlinuxfarm-4 80:80:HttpTrafficInLhttp:80:/test.php
azure vm endpoint create-multiple hanselmanlinuxfarm-5 80:80:HttpTrafficInLhttp:80:/test.php

After it ran, I went in to the Portal and set up Availability Sets manually. That's only available in PowerShell today, but setting availability sets is coming soon to the cross-platform tools!

Next time, maybe I'll try"azure vm scale" to move these tiny VMs into 8 processor 56 gig monsters.


Sponsor: Big thanks to SoftFluent for sponsoring the feed this week! Less Plumbing, More Productivity! Generate rock-solid foundations for your .NET applications from Visual Studio and focus on what matters!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Hosting By
Hosted in an Azure App Service
May 22, 2013 4:00
That's pretty cool. An example script to do the same thing with Windows servers hosting a simple ASP.Net MVC site would be pretty cool. Adding boxes would increase the number of simultaneous users a web site could handler as well as making it one reliable. Which would be cheaper, using Azure's scaling feature to bump a single VM to the next size or adding box(es) to the farm.
May 22, 2013 5:03
Bryan - wouldn't you use websites for that scenario?
May 22, 2013 13:20
Thanks for the post Scott, really well structured and there is a lot of info in here. I will definitely come back to this post for looking things up.
I am about to set up 2 linux VM load balanced to host Neo4j in a cluster. Thanks!
May 22, 2013 14:54
You create a new image named "mylinuxfarmimage" and don't use it? Or is it just a typo?
May 22, 2013 15:55
He probably just renamed it. The discrepancy is where the image was created:

azure vm shutdown mylinuxtemplate
azure vm capture mylinuxtemplate mylinuxfarmimage --delete


and then a virtual machine was created but with a different image name:

C:\> azure vm create hanselmanlinuxfarm hanselmanlinuxwebfarmimage scott MyPassword123 --location "West US" --ssh
May 22, 2013 17:17
Typo, as I did this 4 times while writing the post. I'll update!
May 22, 2013 17:39
Scott, next step is to host your site on Linux using Mono. Somehow I have a feeling that is where you are going with this :)

Alex
May 22, 2013 22:10
Hi Scott

Do you know if there will be client affinity in Azure? or do you need to set this up with Application Request Routing (ARR) etc. in IIS in a separate VM?

Thanks for another great post :)
May 22, 2013 23:42
I can think of about 10 different uses for this. Thanks!
May 23, 2013 2:47
Or you can just publish a set of VMs on Dynamic Deploy http://www.dynamicdeploy.com and point-click your way deployment :)
May 23, 2013 3:24
Will they never learn???

"setup" can be an adjective (a setup file).

"setup" can be a verb (the setup I have at home)

"setup" IS NEVER A VERB!!! (How to setup a Load Balanced Web Farm)

"set" along with "up" is a verb (How to set up a Load Balanced Web Farm)
May 23, 2013 3:30
@Scott I probably would use websites. Ok, let's take ASP.Net out of the picture. Suppose I had some existing server app that had the ability to scale by spinning up another server and setting some values in a config file. Maybe a custom database server. Which would have the most bang for the buck, a single monster VM or spinning up enough small VMs to be able to handle the same amount of traffic?
May 23, 2013 3:58
Thanks for the walk-through. This ability looks great, but I believe the ASP.NET environment is still missing a good scale-out/web farm solution for synchronizing site data and IIS configurations. We have been using the Web Farm Framework with great results on IIS 7, but it seems the tool has been abandoned and isn't compatible with IIS 8.

Dynamically adding a server in the cloud is wonderful, but the actual deployment across servers is still challenging and should be addressed.
May 23, 2013 22:16
Brian - Well, with Azure Websites, you just add instances and there is no deployment setup for n instances. For internal IIS instances of your own, you can use tools like Octopus Deploy, MSDeploy, PowerShell, or even run the Azure Portal and a "private cloud" with System center: http://technet.microsoft.com/en-us/evalcenter/hh505660.aspx?wt.mc_id=TEC_103_1_5

As for WFF and IIS8, yes. I will get someone to update this Forum Thread today. http://forums.iis.net/t/1193106.aspx/1?WFF+New+update+and+possible+IIS8+support

Larry - I will try to do better.
May 25, 2013 23:35
Thanks Scott. I know about Azure Websites and we use them for a few projects, but sometimes we need the whole VM in order to configure the environment. MSDeploy and Powershell are our friends, certainly, but I feel like everyone is reinventing the wheel when it comes to web farms on IIS. WFF worked very well and was much easier to configure and maintain than using something like DFS-backed sites that require shared IIS configuration (which is a pain to change) and CAS tweaks. I'll look into System Center.
May 26, 2013 5:00
Brian - I'm talking to the WFF team to get you an update.
May 31, 2013 1:52
Hi Scott,

Quick question - What do you use for your SSH client? I'm currently trying out kITTY, but wondered if you knew of any others worth using?

Thanks,
Phil
June 01, 2013 5:20
Wow! I really enjoyed this post Scott. Really a good one. I appreciate :-)
June 03, 2013 9:44
Phil - I use the SSH.exe client that comes with GitHub for Windows, or I use Putty.

Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.