Distributed Automated Browser Testing with Selenium and BrowserStack
I'm a huge fan of BrowserStack. They are a cloud-based browser testing service that lets you remote into effectively any browser version on any version of any operating system. They've even got Visual Studio integration so you can just hit F5, choose the browser you want, and start debugging.
COUPON: BrowserStack is a service (a cloud of machines do the work) but you can get 3 months free from the IE folks at Modern.ie. To be clear: I'm not affiliated with either of these folks, just letting you know that it's cool. I do personally have a BrowserStack subscription that I pay for.
I've long been a proponent of integration testing websites by putting the browser on a string, like a puppet. I used Watir (Web Application Testing in Ruby) almost 10 (!) years ago, and write WatirMaker (and later WatirRecorder) to make that easier.
I also spent time with Selenium as early as 2007, although it was a very different Selenium than it is today. I also interviewed Jim Evans from the Selenium Team on my podcast.
BrowserStack as a Selenium Cloud with RemoteDriver
Selenium today uses a "Selenium Server" and a "WebDriver." WebDrivers are language-specific bindings to drive a browser - to put strings on your browser and control it.
Now, there's dozens of choices to make and dozens of ways you can get Selenium working. With much respect due to the Selenium team, the docs on their main page spend a LOT of time talking about different versions, older versions, history of the project, and there's no real "Getting Started FAQ" with stuff like "I'm a Windows person, get this and this and that." Or "I'm on a Mac and like Ruby, just get this and this." There is a fairly extensive Wiki, though, but it's still a lot of terms to understand.
Do expect to spend a few hours exploring and messing up, but do know that there's a few dozen ways to be successful with Selenium, which is part of its charm.
First, you can write tests in whatever language you like. So, C#/NUnit, or Ruby, or Python, for example.
You can download "drivers" for a browser and run them directly, having them listen a port, and make them available to yourself or others in your company.
When writing tests, you can ask for browsers ("drivers") directly by asking for them from your test language of choice, like instantiating a "ChromeDriver" or an "IEDriver."
But, you can also launch a more general-purpose server that will present itself as a "WebDriver," then you give it a list of the capabilities you want and it will then find and drive a browser for you. This is what BrowserStack's cloud does. You can also set these up inside your own company, although it's a bit of a hassle.
There's 8 different OSes. 20 mobile devices, 300 browser/version combos. For the free automated trial you get 100 minutes of "drive time" free. Also, it's half the price if you only want desktop browsers.
I like to use Python for Selenium Tests, for some odd reason. Again, you can use whatever you like. Doesn't matter.
If you don't have Python...
- Get Python 2.7 - I'm on Windows and I got the x86 one.
- Get setuptools for 2.7 - note the py2.7 in the file name.
- Get the latest Pip for 2.7 - Note the py2.7 in the file name.
- Then run "pip install -U selenium"
Your Python code might look like this. Here I'm using BrowserStack's cloud and asking for IE7 on Windows XP.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
desired_cap = {'os': 'Windows',
'os_version': 'xp',
'browser': 'IE',
'browser_version': '7.0',
'browserstack.debug': 'true' }
driver = webdriver.Remote(
command_executor='http://hanselman:mysecretkey@hub.browserstack.com:80/wd/hub',
desired_capabilities=desired_cap)
driver.get("http://www.google.com")
if not "Google" in driver.title:
raise Exception("Unable to load google page!")
elem = driver.find_element_by_name("q")
elem.send_keys("Hanselman")
elem.submit()
print driver.title
driver.quit()
Note I have "browserstack.debug" on, so I can actually go to the BrowserStack site and see screenshots of each step!
Here's the next step...
Details on how this automation works are all up at https://www.browserstack.com/automate. Again you can use any language you want. Here's the same thing in C#:
using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;
namespace SeleniumTest {
class Program {
static void Main(string[] args) {
IWebDriver driver;
DesiredCapabilities capability = DesiredCapabilities.Firefox();
capability.SetCapability("browserstack.user", "hanselman");
capability.SetCapability("browserstack.key", "superfancypantssecretapikey");
driver = new RemoteWebDriver(
new Uri("http://hub.browserstack.com/wd/hub/"), capability
);
driver.Navigate().GoToUrl("http://www.google.com/ncr");
Console.WriteLine(driver.Title);
IWebElement query = driver.FindElement(By.Name("q"));
query.SendKeys("hanselman");
query.Submit();
Console.WriteLine(driver.Title);
driver.Quit();
}
}
}
If your site is only available at localhost, you can make a temporary tunnel and make is accessible to BrowserStack, as well. If you've got Chrome or Firefox, there are extensions to make this even easier.
I hope you check out both BrowserStack and Selenium, and perhaps consider how you're testing your site today (humans who click?) and if you could be more effective with different tools?
Sponsor: Big thanks to Red Gate for sponsoring the blog feed this week! Easy release management: Deploy your SQL Server databases in a single, repeatable process with Red Gate’s Deployment Manager. There’s a free Starter edition, so get started now!
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
But seriously, if you are into testing advanced workflows (samples like Google search are always easy) and want easy-maintenance, stables scripts you will love Silk Test.
Also, you can have your test scripts in C# (or VB.NET ... or Java).
Just approach me if you need more info on Silk Test.
We just had an issue where order totals were not being calculated correctly. I've been able to start testing a few things automatically, and Watir's/Selenium's ability to read any element on the page is really great! I just wish it had the ability to read HTTP status codes.
(and yes, while the form submits to a secure url, the unencrypted page is subject to attack and potential change while being sent from browserstack server to the user's web browser.)
Instead of the main page, click on the Sign In link or the Sign Up link at the top of the main browserstack site to be taken to SSL encrypted pages.
Another cool thing with BrowserStack is that if you have an internal website or programmatically start IIS Express during your tests, you can run their BrowserStackTunnel.jar file to create a tunnel that allows BrowserStack to access your non-public site. It's a great way to run smoke tests on your build server before deploying to a public location.
http://release.seleniumhq.org/selenium-ide/2.5.0/selenium-ide-2.5.0.xpi
"You can redeem your offer anytime before January 10, 2014"
Shame, I loved it when I used the free trial.
I agree, FluentAutomation is very easy to use. It also gives you the option of using PhantomJS so you can run "headless" tests. I'm not sure how it copes with automatic testing as I have been using Karma (formerly known as Testacular) which is blazing fast but if you are working on a full .NET project FluentAutomation seems to fit nicely.
There is a lot of movement in the hosted cloud testing area right now from Selenium Grid deployments (SauceLabs) to fully custom solutions.
We're building F14N with the goal of tackling the overall issue of automated testing (test creation, maintainability, execution, etc).
Disclaimer: Creator of FluentAutomation, more info at http://fluent.stirno.com
We are currently using Jasmine on our own server for acceptance test and it's very unstable. For example Firefox update dialog can "brake" test. For legal reasons we cannot use something like BrowserStack.
Awesome to see it in Python.
It offers an instant Cross-Mobile & Browser web testing, issue discovery and recommendation how to fix issues. It's free for casual testing and provide quite valuable results in a few seconds such as: spelling errors, broken links, recommendations how to optimize pages, etc.
It could be a good alternative for these who need something cheaper and focus on validation testing and site improvements.
We have been using it to test sites we develop for our clients and share results with them so customers could provide their comments, as service provides the visual annotations on the screenshots as well.
Brandon - I will check out your stuff!
@Justin We're working on the Windows Auth bug(affecting a few customers) with top priority, will notify you once it's fixed.
@Ray The modern.IE offer has been extended and currently available. I've also asked the modern.IE team to fix the message.
@Tom We've responded to your mail with details.
It'll give you access to 100 minutes of Automate along with a unique username and access key.
Just wanted to point out that we at TestingBot provide the same automated and manual browser testing. We focus on delivering a pristine virtual machine for every test case and have a TestLab where people can upload tests.
If there are readers of this blog who want to try us out, please let me know and I'll give you a free 1-month small-team plan!
Comments are closed.