Scott Hanselman

Distributed Automated Browser Testing with Selenium and BrowserStack

February 20, '14 Comments [26] Posted in Musings
Sponsored By

imageI'm a huge fan of BrowserStack. They are a cloud-based browser testing service that lets you remote into effectively any browser version on any version of any operating system. They've even got Visual Studio integration so you can just hit F5, choose the browser you want, and start debugging.

COUPON: BrowserStack is a service (a cloud of machines do the work) but you can get 3 months free from the IE folks at Modern.ie. To be clear: I'm not affiliated with either of these folks, just letting you know that it's cool. I do personally have a BrowserStack subscription that I pay for.

I've long been a proponent of integration testing websites by putting the browser on a string, like a puppet. I used Watir (Web Application Testing in Ruby) almost 10 (!) years ago, and write WatirMaker (and later WatirRecorder) to make that easier.

I also spent time with Selenium as early as 2007, although it was a very different Selenium than it is today. I also interviewed Jim Evans from the Selenium Team on my podcast.

BrowserStack as a Selenium Cloud with RemoteDriver

Selenium today uses a "Selenium Server" and a "WebDriver." WebDrivers are language-specific bindings to drive a browser - to put strings on your browser and control it.

Now, there's dozens of choices to make and dozens of ways you can get Selenium working. With much respect due to the Selenium team, the docs on their main page spend a LOT of time talking about different versions, older versions, history of the project, and there's no real "Getting Started FAQ" with stuff like "I'm a Windows person, get this and this and that." Or "I'm on a Mac and like Ruby, just get this and this." There is a fairly extensive Wiki, though, but it's still a lot of terms to understand.

Do expect to spend a few hours exploring and messing up, but do know that there's a few dozen ways to be successful with Selenium, which is part of its charm.

First, you can write tests in whatever language you like. So, C#/NUnit, or Ruby, or Python, for example.

You can download "drivers" for a browser and run them directly, having them listen a port, and make them available to yourself or others in your company.

When writing tests, you can ask for browsers ("drivers") directly by asking for them from your test language of choice, like instantiating a "ChromeDriver" or an "IEDriver."

But, you can also launch a more general-purpose server that will present itself as a "WebDriver," then you give it a list of the capabilities you want and it will then find and drive a browser for you. This is what BrowserStack's cloud does. You can also set these up inside your own company, although it's a bit of a hassle.

There's 8 different OSes. 20 mobile devices, 300 browser/version combos. For the free automated trial you get 100 minutes of "drive time" free. Also, it's half the price if you only want desktop browsers.

The Web Interface for BrowserStack showing the tests I've run

I like to use Python for Selenium Tests, for some odd reason. Again, you can use whatever you like. Doesn't matter.

If you don't have Python...

  • Get Python 2.7 - I'm on Windows and I got the x86 one.
  • Get setuptools for 2.7 - note the py2.7 in the file name.
  • Get the latest Pip for 2.7 - Note the py2.7 in the file name.
  • Then run "pip install -U selenium"

Your Python code might look like this. Here I'm using BrowserStack's cloud and asking for IE7 on Windows XP.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

desired_cap = {'os': 'Windows',
'os_version': 'xp',
'browser': 'IE',
'browser_version': '7.0',
'browserstack.debug': 'true' }

driver = webdriver.Remote(
command_executor='http://hanselman:mysecretkey@hub.browserstack.com:80/wd/hub',
desired_capabilities=desired_cap)

driver.get("http://www.google.com")
if not "Google" in driver.title:
raise Exception("Unable to load google page!")
elem = driver.find_element_by_name("q")
elem.send_keys("Hanselman")
elem.submit()
print driver.title
driver.quit()

Note I have "browserstack.debug" on, so I can actually go to the BrowserStack site and see screenshots of each step!

Screenshot of BrowserStack automatically typing Hanselman into Google on IE7 on XP

Here's the next step...

The results of the Google Search for Hanselman

Details on how this automation works are all up at https://www.browserstack.com/automate. Again you can use any language you want. Here's the same thing in C#:

using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;

namespace SeleniumTest {
class Program {
static void Main(string[] args) {
IWebDriver driver;
DesiredCapabilities capability = DesiredCapabilities.Firefox();
capability.SetCapability("browserstack.user", "hanselman");
capability.SetCapability("browserstack.key", "superfancypantssecretapikey");

driver = new RemoteWebDriver(
new Uri("http://hub.browserstack.com/wd/hub/"), capability
);
driver.Navigate().GoToUrl("http://www.google.com/ncr");
Console.WriteLine(driver.Title);

IWebElement query = driver.FindElement(By.Name("q"));
query.SendKeys("hanselman");
query.Submit();
Console.WriteLine(driver.Title);

driver.Quit();
}
}
}

If your site is only available at localhost, you can make a temporary tunnel and make is accessible to BrowserStack, as well. If you've got Chrome or Firefox, there are extensions to make this even easier.

I hope you check out both BrowserStack and Selenium, and perhaps consider how you're testing your site today (humans who click?) and if you could be more effective with different tools?


Sponsor: Big thanks to Red Gate for sponsoring the blog feed this week! Easy release management: Deploy your SQL Server databases in a single, repeatable process with Red Gate’s Deployment Manager. There’s a free Starter edition, so get started now!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Thursday, February 20, 2014 1:00:13 AM UTC
My team has been using SauceLabs which is a similar service. Also of note their CTO is Jason Huggins, creator of Selenium
Thursday, February 20, 2014 1:04:48 AM UTC
Ken - Great point! SauceLabs has a VERY similar offering that also does screenshots. I would encourage readers to try both!
Thursday, February 20, 2014 3:15:10 AM UTC
Curious about the odd reasons to use Python over Ruby, if not C#. Since you'd been using waitr for years ;)
Thursday, February 20, 2014 3:20:25 AM UTC
I dunno why. It just feels better. I'll try the Ruby driver again.
Thursday, February 20, 2014 5:10:55 AM UTC
Another wonderful post! Selenium and BrowserStack are truly great tools. Together they are a fantastic solution to automatically test web sites and applications. I've been a long time "lurker" and wanted to comment on this post just to say thanks.
Mike A
Thursday, February 20, 2014 7:38:21 AM UTC
If you are serious about cross-browser web application testing, you should take a closer look at Silk Test from Borland (happens to be the company, I am working for).
But seriously, if you are into testing advanced workflows (samples like Google search are always easy) and want easy-maintenance, stables scripts you will love Silk Test.

Also, you can have your test scripts in C# (or VB.NET ... or Java).

Just approach me if you need more info on Silk Test.
Thursday, February 20, 2014 3:14:15 PM UTC
I just started using Watir a few weeks ago. Work was rather slow, so I decided to make myself useful by exploring some automated testing. The platform I work with, Demandware, isn't exactly conductive to unit/automated tests.

We just had an issue where order totals were not being calculated correctly. I've been able to start testing a few things automatically, and Watir's/Selenium's ability to read any element on the page is really great! I just wish it had the ability to read HTTP status codes.
Thursday, February 20, 2014 4:55:43 PM UTC
I'm a big fan of BrowserStack too but you might as well forget it for testing internal Windows Authenticated sites. I was recently told that his was addressed but when testing on all of the browser types (Chrome, IE, Safari) they all appear to have trouble with the NTLM handshake. Sometimes it appears to work but then I'm prompted for every single request after the initial success. Figured I'd post this in case anyone were to purchase or get the trial.
Thursday, February 20, 2014 5:30:00 PM UTC
Be careful when signing up/in to the browserstack website. Their main page containing the sign-up form is NOT SSL encrypted and something could potentially grab your typed-in sign-up username & password, or change the sign-up form posting URL to some malicious site.
(and yes, while the form submits to a secure url, the unencrypted page is subject to attack and potential change while being sent from browserstack server to the user's web browser.)

Instead of the main page, click on the Sign In link or the Sign Up link at the top of the main browserstack site to be taken to SSL encrypted pages.
Brian M
Thursday, February 20, 2014 7:13:13 PM UTC
If you want to write your browser tests in C#, I highly recommend FluentAutomation. IMO it has a great fluent, easy to understand API and includes support for Selenium's WebDriver. Check out their Getting Started docs. I've used it in one of my projects and I was able to get it working faster than Selenium alone.

Another cool thing with BrowserStack is that if you have an internal website or programmatically start IIS Express during your tests, you can run their BrowserStackTunnel.jar file to create a tunnel that allows BrowserStack to access your non-public site. It's a great way to run smoke tests on your build server before deploying to a public location.
Thursday, February 20, 2014 8:24:52 PM UTC
You can even record your Selenium tests using the Selenium IDE extension for FireFox. It can export the recorded steps into your favorite language (C#, Python, ect).

http://release.seleniumhq.org/selenium-ide/2.5.0/selenium-ide-2.5.0.xpi
Kurt Wiersma
Thursday, February 20, 2014 9:38:24 PM UTC
Has the BrowserStack 3 month offer ended? It is still displayed proudly on the Modern.Ie site, but says:
"You can redeem your offer anytime before January 10, 2014"
Ray
Friday, February 21, 2014 6:52:20 AM UTC
Great post!! Will consider browser stack for my next project as testing tool.
Friday, February 21, 2014 9:38:23 AM UTC
We were really looking forward to getting our hands dirty with BrowserStack. Unfortunately the price hike from $79 for 5 parallel tests to $199 WITH a FUP in place is insane. This has put the pricing out of what is acceptable for us and also we would break the FUP every single day.

Shame, I loved it when I used the free trial.
Tom
Friday, February 21, 2014 2:06:45 PM UTC
@ Stephen Kuhn

I agree, FluentAutomation is very easy to use. It also gives you the option of using PhantomJS so you can run "headless" tests. I'm not sure how it copes with automatic testing as I have been using Karma (formerly known as Testacular) which is blazing fast but if you are working on a full .NET project FluentAutomation seems to fit nicely.
Matthew Blott
Friday, February 21, 2014 2:19:59 PM UTC
Glad to see some people bring up FluentAutomation - we've got a nice community forming these days.

There is a lot of movement in the hosted cloud testing area right now from Selenium Grid deployments (SauceLabs) to fully custom solutions.

We're building F14N with the goal of tackling the overall issue of automated testing (test creation, maintainability, execution, etc).

Disclaimer: Creator of FluentAutomation, more info at http://fluent.stirno.com
Friday, February 21, 2014 4:33:25 PM UTC
I agree with @Matthew Blott - PhantomJS is nice alternative as it's faster and less prone to accidental errors. I too however used it only for JavaScript unit testing with Karma test runner. This works really well.

We are currently using Jasmine on our own server for acceptance test and it's very unstable. For example Firefox update dialog can "brake" test. For legal reasons we cannot use something like BrowserStack.
Friday, February 21, 2014 4:36:59 PM UTC
Sorry I meant Selenium not Jasmine of course :)
Saturday, February 22, 2014 3:06:14 AM UTC
Looks like I might have to try out browser stack in future projects.
Awesome to see it in Python.
Saturday, February 22, 2014 5:14:58 AM UTC
If test automation is not a priority, then there is a nice www.testize.com service we have been using last few month and love it.

It offers an instant Cross-Mobile & Browser web testing, issue discovery and recommendation how to fix issues. It's free for casual testing and provide quite valuable results in a few seconds such as: spelling errors, broken links, recommendations how to optimize pages, etc.

It could be a good alternative for these who need something cheaper and focus on validation testing and site improvements.

We have been using it to test sites we develop for our clients and share results with them so customers could provide their comments, as service provides the visual annotations on the screenshots as well.
StanBeloff
Saturday, February 22, 2014 6:08:10 AM UTC
Ray - They are extending the free trial.

Brandon - I will check out your stuff!
Saturday, February 22, 2014 11:46:32 AM UTC
Hari here from BrowserStack.

@Justin We're working on the Windows Auth bug(affecting a few customers) with top priority, will notify you once it's fixed.

@Ray The modern.IE offer has been extended and currently available. I've also asked the modern.IE team to fix the message.

@Tom We've responded to your mail with details.
Friday, February 28, 2014 9:06:10 AM UTC
Hi Man i want to know do you need to buy their package before you can get username or keys?
Phira
Saturday, March 01, 2014 9:54:27 AM UTC
@Phira You can sign up for the free trial from here - https://www.browserstack.com/users/sign_up

It'll give you access to 100 minutes of Automate along with a unique username and access key.
Tuesday, April 01, 2014 2:36:55 PM UTC
Hi I'm Jochen from TestingBot

Just wanted to point out that we at TestingBot provide the same automated and manual browser testing. We focus on delivering a pristine virtual machine for every test case and have a TestLab where people can upload tests.

If there are readers of this blog who want to try us out, please let me know and I'll give you a free 1-month small-team plan!
Saturday, April 26, 2014 2:09:21 PM UTC
Scott -any updated on the IE11 + selenium story?
Comments are closed.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.