Scott Hanselman

Distributed Automated Browser Testing with Selenium and BrowserStack

February 20, '14 Comments [56] Posted in Musings
Sponsored By

imageI'm a huge fan of BrowserStack. They are a cloud-based browser testing service that lets you remote into effectively any browser version on any version of any operating system. They've even got Visual Studio integration so you can just hit F5, choose the browser you want, and start debugging.

COUPON: BrowserStack is a service (a cloud of machines do the work) but you can get 3 months free from the IE folks at Modern.ie. To be clear: I'm not affiliated with either of these folks, just letting you know that it's cool. I do personally have a BrowserStack subscription that I pay for.

I've long been a proponent of integration testing websites by putting the browser on a string, like a puppet. I used Watir (Web Application Testing in Ruby) almost 10 (!) years ago, and write WatirMaker (and later WatirRecorder) to make that easier.

I also spent time with Selenium as early as 2007, although it was a very different Selenium than it is today. I also interviewed Jim Evans from the Selenium Team on my podcast.

BrowserStack as a Selenium Cloud with RemoteDriver

Selenium today uses a "Selenium Server" and a "WebDriver." WebDrivers are language-specific bindings to drive a browser - to put strings on your browser and control it.

Now, there's dozens of choices to make and dozens of ways you can get Selenium working. With much respect due to the Selenium team, the docs on their main page spend a LOT of time talking about different versions, older versions, history of the project, and there's no real "Getting Started FAQ" with stuff like "I'm a Windows person, get this and this and that." Or "I'm on a Mac and like Ruby, just get this and this." There is a fairly extensive Wiki, though, but it's still a lot of terms to understand.

Do expect to spend a few hours exploring and messing up, but do know that there's a few dozen ways to be successful with Selenium, which is part of its charm.

First, you can write tests in whatever language you like. So, C#/NUnit, or Ruby, or Python, for example.

You can download "drivers" for a browser and run them directly, having them listen a port, and make them available to yourself or others in your company.

When writing tests, you can ask for browsers ("drivers") directly by asking for them from your test language of choice, like instantiating a "ChromeDriver" or an "IEDriver."

But, you can also launch a more general-purpose server that will present itself as a "WebDriver," then you give it a list of the capabilities you want and it will then find and drive a browser for you. This is what BrowserStack's cloud does. You can also set these up inside your own company, although it's a bit of a hassle.

There's 8 different OSes. 20 mobile devices, 300 browser/version combos. For the free automated trial you get 100 minutes of "drive time" free. Also, it's half the price if you only want desktop browsers.

The Web Interface for BrowserStack showing the tests I've run

I like to use Python for Selenium Tests, for some odd reason. Again, you can use whatever you like. Doesn't matter.

If you don't have Python...

  • Get Python 2.7 - I'm on Windows and I got the x86 one.
  • Get setuptools for 2.7 - note the py2.7 in the file name.
  • Get the latest Pip for 2.7 - Note the py2.7 in the file name.
  • Then run "pip install -U selenium"

Your Python code might look like this. Here I'm using BrowserStack's cloud and asking for IE7 on Windows XP.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

desired_cap = {'os': 'Windows',
'os_version': 'xp',
'browser': 'IE',
'browser_version': '7.0',
'browserstack.debug': 'true' }

driver = webdriver.Remote(
command_executor='http://hanselman:mysecretkey@hub.browserstack.com:80/wd/hub',
desired_capabilities=desired_cap)

driver.get("http://www.google.com")
if not "Google" in driver.title:
raise Exception("Unable to load google page!")
elem = driver.find_element_by_name("q")
elem.send_keys("Hanselman")
elem.submit()
print driver.title
driver.quit()

Note I have "browserstack.debug" on, so I can actually go to the BrowserStack site and see screenshots of each step!

Screenshot of BrowserStack automatically typing Hanselman into Google on IE7 on XP

Here's the next step...

The results of the Google Search for Hanselman

Details on how this automation works are all up at https://www.browserstack.com/automate. Again you can use any language you want. Here's the same thing in C#:

using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;

namespace SeleniumTest {
class Program {
static void Main(string[] args) {
IWebDriver driver;
DesiredCapabilities capability = DesiredCapabilities.Firefox();
capability.SetCapability("browserstack.user", "hanselman");
capability.SetCapability("browserstack.key", "superfancypantssecretapikey");

driver = new RemoteWebDriver(
new Uri("http://hub.browserstack.com/wd/hub/"), capability
);
driver.Navigate().GoToUrl("http://www.google.com/ncr");
Console.WriteLine(driver.Title);

IWebElement query = driver.FindElement(By.Name("q"));
query.SendKeys("hanselman");
query.Submit();
Console.WriteLine(driver.Title);

driver.Quit();
}
}
}

If your site is only available at localhost, you can make a temporary tunnel and make is accessible to BrowserStack, as well. If you've got Chrome or Firefox, there are extensions to make this even easier.

I hope you check out both BrowserStack and Selenium, and perhaps consider how you're testing your site today (humans who click?) and if you could be more effective with different tools?


Sponsor: Big thanks to Red Gate for sponsoring the blog feed this week! Easy release management: Deploy your SQL Server databases in a single, repeatable process with Red Gate’s Deployment Manager. There’s a free Starter edition, so get started now!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Thursday, February 20, 2014 1:00:13 AM UTC
My team has been using SauceLabs which is a similar service. Also of note their CTO is Jason Huggins, creator of Selenium
Thursday, February 20, 2014 1:04:48 AM UTC
Ken - Great point! SauceLabs has a VERY similar offering that also does screenshots. I would encourage readers to try both!
Thursday, February 20, 2014 3:15:10 AM UTC
Curious about the odd reasons to use Python over Ruby, if not C#. Since you'd been using waitr for years ;)
Thursday, February 20, 2014 3:20:25 AM UTC
I dunno why. It just feels better. I'll try the Ruby driver again.
Thursday, February 20, 2014 5:10:55 AM UTC
Another wonderful post! Selenium and BrowserStack are truly great tools. Together they are a fantastic solution to automatically test web sites and applications. I've been a long time "lurker" and wanted to comment on this post just to say thanks.
Mike A
Thursday, February 20, 2014 7:38:21 AM UTC
If you are serious about cross-browser web application testing, you should take a closer look at Silk Test from Borland (happens to be the company, I am working for).
But seriously, if you are into testing advanced workflows (samples like Google search are always easy) and want easy-maintenance, stables scripts you will love Silk Test.

Also, you can have your test scripts in C# (or VB.NET ... or Java).

Just approach me if you need more info on Silk Test.
Thursday, February 20, 2014 3:14:15 PM UTC
I just started using Watir a few weeks ago. Work was rather slow, so I decided to make myself useful by exploring some automated testing. The platform I work with, Demandware, isn't exactly conductive to unit/automated tests.

We just had an issue where order totals were not being calculated correctly. I've been able to start testing a few things automatically, and Watir's/Selenium's ability to read any element on the page is really great! I just wish it had the ability to read HTTP status codes.
Thursday, February 20, 2014 4:55:43 PM UTC
I'm a big fan of BrowserStack too but you might as well forget it for testing internal Windows Authenticated sites. I was recently told that his was addressed but when testing on all of the browser types (Chrome, IE, Safari) they all appear to have trouble with the NTLM handshake. Sometimes it appears to work but then I'm prompted for every single request after the initial success. Figured I'd post this in case anyone were to purchase or get the trial.
Thursday, February 20, 2014 5:30:00 PM UTC
Be careful when signing up/in to the browserstack website. Their main page containing the sign-up form is NOT SSL encrypted and something could potentially grab your typed-in sign-up username & password, or change the sign-up form posting URL to some malicious site.
(and yes, while the form submits to a secure url, the unencrypted page is subject to attack and potential change while being sent from browserstack server to the user's web browser.)

Instead of the main page, click on the Sign In link or the Sign Up link at the top of the main browserstack site to be taken to SSL encrypted pages.
Brian M
Thursday, February 20, 2014 7:13:13 PM UTC
If you want to write your browser tests in C#, I highly recommend FluentAutomation. IMO it has a great fluent, easy to understand API and includes support for Selenium's WebDriver. Check out their Getting Started docs. I've used it in one of my projects and I was able to get it working faster than Selenium alone.

Another cool thing with BrowserStack is that if you have an internal website or programmatically start IIS Express during your tests, you can run their BrowserStackTunnel.jar file to create a tunnel that allows BrowserStack to access your non-public site. It's a great way to run smoke tests on your build server before deploying to a public location.
Thursday, February 20, 2014 8:24:52 PM UTC
You can even record your Selenium tests using the Selenium IDE extension for FireFox. It can export the recorded steps into your favorite language (C#, Python, ect).

http://release.seleniumhq.org/selenium-ide/2.5.0/selenium-ide-2.5.0.xpi
Kurt Wiersma
Thursday, February 20, 2014 9:38:24 PM UTC
Has the BrowserStack 3 month offer ended? It is still displayed proudly on the Modern.Ie site, but says:
"You can redeem your offer anytime before January 10, 2014"
Ray
Friday, February 21, 2014 6:52:20 AM UTC
Great post!! Will consider browser stack for my next project as testing tool.
Friday, February 21, 2014 9:38:23 AM UTC
We were really looking forward to getting our hands dirty with BrowserStack. Unfortunately the price hike from $79 for 5 parallel tests to $199 WITH a FUP in place is insane. This has put the pricing out of what is acceptable for us and also we would break the FUP every single day.

Shame, I loved it when I used the free trial.
Tom
Friday, February 21, 2014 2:06:45 PM UTC
@ Stephen Kuhn

I agree, FluentAutomation is very easy to use. It also gives you the option of using PhantomJS so you can run "headless" tests. I'm not sure how it copes with automatic testing as I have been using Karma (formerly known as Testacular) which is blazing fast but if you are working on a full .NET project FluentAutomation seems to fit nicely.
Matthew Blott
Friday, February 21, 2014 2:19:59 PM UTC
Glad to see some people bring up FluentAutomation - we've got a nice community forming these days.

There is a lot of movement in the hosted cloud testing area right now from Selenium Grid deployments (SauceLabs) to fully custom solutions.

We're building F14N with the goal of tackling the overall issue of automated testing (test creation, maintainability, execution, etc).

Disclaimer: Creator of FluentAutomation, more info at http://fluent.stirno.com
Friday, February 21, 2014 4:33:25 PM UTC
I agree with @Matthew Blott - PhantomJS is nice alternative as it's faster and less prone to accidental errors. I too however used it only for JavaScript unit testing with Karma test runner. This works really well.

We are currently using Jasmine on our own server for acceptance test and it's very unstable. For example Firefox update dialog can "brake" test. For legal reasons we cannot use something like BrowserStack.
Friday, February 21, 2014 4:36:59 PM UTC
Sorry I meant Selenium not Jasmine of course :)
Saturday, February 22, 2014 3:06:14 AM UTC
Looks like I might have to try out browser stack in future projects.
Awesome to see it in Python.
Saturday, February 22, 2014 5:14:58 AM UTC
If test automation is not a priority, then there is a nice www.testize.com service we have been using last few month and love it.

It offers an instant Cross-Mobile & Browser web testing, issue discovery and recommendation how to fix issues. It's free for casual testing and provide quite valuable results in a few seconds such as: spelling errors, broken links, recommendations how to optimize pages, etc.

It could be a good alternative for these who need something cheaper and focus on validation testing and site improvements.

We have been using it to test sites we develop for our clients and share results with them so customers could provide their comments, as service provides the visual annotations on the screenshots as well.
StanBeloff
Saturday, February 22, 2014 6:08:10 AM UTC
Ray - They are extending the free trial.

Brandon - I will check out your stuff!
Saturday, February 22, 2014 11:46:32 AM UTC
Hari here from BrowserStack.

@Justin We're working on the Windows Auth bug(affecting a few customers) with top priority, will notify you once it's fixed.

@Ray The modern.IE offer has been extended and currently available. I've also asked the modern.IE team to fix the message.

@Tom We've responded to your mail with details.
Friday, February 28, 2014 9:06:10 AM UTC
Hi Man i want to know do you need to buy their package before you can get username or keys?
Phira
Saturday, March 01, 2014 9:54:27 AM UTC
@Phira You can sign up for the free trial from here - https://www.browserstack.com/users/sign_up

It'll give you access to 100 minutes of Automate along with a unique username and access key.
Sunday, March 23, 2014 5:10:11 PM UTC
Magnificent beat ! I wish to apprentice even as you amend your web site, how can
i subscribe for a weblog site? The account helped me
a acceptable deal. I were a little bit familiar of this your broadcast provided vivid clear concept
Sunday, March 23, 2014 7:31:01 PM UTC
This design is incredible! You most certainly know how to keep
a reader entertained. Between your wit and your videos, I was almost moved to start my own blog (well,
almost...HaHa!) Wonderful job. I really enjoyed what you
had to say, and more than that, how you presented it. Too cool!
Monday, March 24, 2014 2:58:03 AM UTC
Do you mind if I quote a couple of your posts as long as I
provide credit and sources back to your website?

My website is in the exact same area of interest as yours and my visitors would definitely benefit
from a lot of the information you present here.

Please let me know if this okay with you. Regards!
Tuesday, March 25, 2014 5:17:32 PM UTC
You need to take part in a contest for one of the best
blogs on the web. I'm going to highly recommend this blog!


Feel free to visit my site - Castle Clash Hack (www.facebook.com)
Thursday, March 27, 2014 2:24:23 PM UTC
I am curious to find out what blog platform
you have been utilizing? I'm having some
small security issues with my latest site and I'd like
to find something more safe. Do you have any recommendations?


my web-site ... Eternity Warriors 3 Hack (http://imgur.com)
Friday, March 28, 2014 2:47:04 AM UTC
This is my first time pay a quick visit at here and i am really
impressed to read everthing at single place.
Friday, March 28, 2014 7:58:12 PM UTC
Hi! Do you use Twitter? I'd like to follow you if that would be ok.
I'm undoubtedly enjoying your blog and look forward to new updates.


my page - best laptop 2013
Saturday, March 29, 2014 9:31:06 AM UTC
You shouldn't be gaming for more than a few hours each day.
SEPARATION OF CHURCH AND STATEAnd, of course, there’s that analytically tricky puzzle of the
doctrine of “Separation of Church and State” to very simply demystify here, since it is one of the most misunderstood and abused subjects, on both sides,
which was ever structurally primed for being as inherently, rationally indispensable,
in its true meaning, as it is capable of falsifyingly and
destructively cutting, again, either way. ammonals ammoniac ammonias ammonify
ammonite ammonium ammonoid amnesiac.
Saturday, March 29, 2014 7:54:37 PM UTC
hey there and thank you for your information –
I have certainly picked up something new from right here.
I did however expertise some technical points using this website, as I experienced to reload the web site many times
previous to I could get it to load correctly. I had been wondering if your web hosting is OK?
Not that I am complaining, but slow loading instances times will sometimes affect your placement in google and can damage your quality score if
advertising and marketing with Adwords. Well I'm adding this RSS to my
e-mail and can look out for much more of your respective intriguing content.
Ensure that you update this again soon.
Sunday, March 30, 2014 1:47:45 AM UTC
The Xperia Play employs the previously popular Android operating process,
and it really is satisfying to search out out the telephone boasting the latest two.

Altair is an Arabic word, which means "the flying eagle. At the beginning of Spike Video Game Awards 2009 announced a continuation of this game.
Sunday, March 30, 2014 11:17:38 AM UTC
Hi! Do you know if they make any plugins to help with SEO?

I'm trying to get my blog to rank for some targeted keywords but I'm not seeing very good
success. If you know of any please share. Kudos!
Sunday, March 30, 2014 7:19:42 PM UTC
I'm extremely inspired along with your writing talents
and also with the format in your weblog. Is this
a paid topic or did you modify it yourself? Anyway keep up the nice quality writing, it is rare to see a great blog
like this one these days..
Sunday, March 30, 2014 8:24:51 PM UTC
What's up everyone, it's my first go to see at this
web page, and post is truly fruitful designed for me, keep up posting such articles or reviews.
Monday, March 31, 2014 12:31:36 PM UTC
It's very simple to find out any matter on net as compared to books, as I
found this paragraph at this website.
Monday, March 31, 2014 7:00:36 PM UTC
Helpo everyone, it's my first pay a visit at this website, and article is actually
fruitful inn support of me, keep up posting such articles or reviews.
Monday, March 31, 2014 8:32:14 PM UTC
Hello there, I found your site by way of Google even as searching for a related matter,
your web site came up, it appears great. I have bookmarked it in my
google bookmarks.
Hi there, just was alert to your blog through Google, and located
that it is really informative. I'm going to be careful for brussels.
I will appreciate if you happen to continue this in future.
A lot of people might be benefited from your writing.
Cheers!
Tuesday, April 01, 2014 2:36:55 PM UTC
Hi I'm Jochen from TestingBot

Just wanted to point out that we at TestingBot provide the same automated and manual browser testing. We focus on delivering a pristine virtual machine for every test case and have a TestLab where people can upload tests.

If there are readers of this blog who want to try us out, please let me know and I'll give you a free 1-month small-team plan!
Wednesday, April 02, 2014 12:13:08 AM UTC
I love to share information that I have built up
with the 12 months to assist improve team efficiency.


my web-site SEO - madonnaschirmeist.pen.io -
Wednesday, April 02, 2014 8:05:55 AM UTC
cumquats cumshaws cumulate cumulous cuneated cuneatic cuniform cunnings.
To the chance to make changes, make resolutions, make lists.
sorb     sord     sore     sori     sorn
    sort     soth     sots.
Wednesday, April 02, 2014 4:51:46 PM UTC
Hearing someone inform you: “You’re beautiful,
inside and out. Wouldn't simply like to have your entire family's favorite cookies anytime they really want one.
The basic point of holding a shower is always to indulge the bride.
Wednesday, April 02, 2014 7:52:48 PM UTC
Thank you for another informative web site. The place else may I get that kind of
info written in such an ideal way? I have a undertaking that I
am just now operating on, and I have been at the glance out
for such information.
Thursday, April 03, 2014 9:55:33 AM UTC
I am really enjoying the theme/design of your weblog. Do you ever run into any browser compatibility problems?
A small number of my blog audience have complained about my blog not operating correctly in
Explorer but looks great in Chrome. Do you have any advice to
help fix this issue?

Also visit my web-site Asphalt 8 Airborne Hack Android (Astucespourlesjeuxmobiles.Blogspot.Com)
Thursday, April 03, 2014 12:38:52 PM UTC
What's up everyone, it's my first visit at this web page,
and post is truly fruitful for me, keep up
posting these articles.
Thursday, April 03, 2014 9:02:36 PM UTC
I do agree with all of the concepts you've offered on your post.

They're very convincing and will definitely work.
Still, the posts are very short for beginners.

May just you please lengthen them a bit from next time?
Thanks for the post.
Friday, April 04, 2014 9:46:10 AM UTC
Good day! I just would like to give an enormous thumbs up for the great data you��ve here on this post.
I will probably be coming again to your blog for more soon.


My webpage: 即時新聞 (Porter)
Sunday, April 06, 2014 9:04:44 AM UTC
s home screen of the original style of the mark, or any of the applications.
Orange San Francisco provides with an attractive collection of essential functionalities covered in a compressed device, which uses technology of smart phone.
You can transfer files or download from one the other.


Have a look at my page; Smash Hit Hack Iphone
Tuesday, April 08, 2014 10:51:09 PM UTC
I love to disseminate information that will I have accumulated with the season to help enhance
group performance.

Here is my web page ... 情趣用品 (Diego)
Wednesday, April 09, 2014 1:41:54 AM UTC
It's actually a nice and helpful piece of info. I'm glad that you just shared this helpful information with us.
Please stay us up to date like this. Thanks for sharing.


Look into my webpage: sbobet
Wednesday, April 09, 2014 9:02:29 PM UTC
Nice post. I was checking constantly this blog and I am impressed!
Extremely helpful info particularly the last part :) I
care for such information a lot. I was looking for this certain
information for a very long time. Thank you and best of luck.
Thursday, April 10, 2014 11:53:13 AM UTC
I simply could not go away your site before suggesting that I extremely loved the standard info an individual provide for your guests?
Is gonna be again steadily in order to investigate cross-check new posts
Thursday, April 10, 2014 11:58:18 AM UTC
Currently it sounds like Movable Type is the preferred blogging platform out there right now.
(from what I've read) Is that what you are using on your blog?


Also visit my website - Gamdroids CSR Racing Hack (Karol)
Thursday, April 24, 2014 9:49:59 AM UTC
It's genuinely very complex in this active life to listen news on TV,
thus I simply use world wide web for that reason, and take the most recent information.
Name
E-mail (will show your gravatar icon)
Home page
 
Comment (Some html is allowed: a@href@title, b, blockquote@cite, em, i, li, ol, pre, strike, strong, sub, super, u, ul) where the @ means "attribute." For example, you can use <a href="" title=""> or <blockquote cite="Scott">.
Live Comment Preview

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.