Run Selenium Test in build vNext

Previous Articles:

To run a Selenium test in a build vNext there are some modification to do apply to previous example. Let’s see how simple is running our Selenium Tests in a VSTS Build vNext.

The first modification requires adding a reference to PhantomJS, an Headless browser based on webkit that is capable of browsing a site and run javascript without a UI. Since we are interested in running test in a build server, this is a requirement if the agent runs as a service and has not access to a UI. To use PhantomJS just reference PhantomJS package on nunit and modify your base class to support this browser.

switch (browser.ToLower())
{
    case "chrome":
        DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
        WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
        break;
    case "firefox":
        WebDriver = new FirefoxDriver();
        break;
    case "phantomjs":
        WebDriver = new PhantomJSDriver();
        break;

Using Phantomjs is just a matter of creating a PhantomJSDriver with Selenium WebDriver test and the game is done. Now add “phantomjs” to test config file and you should be able to run the test.

image

Figure 1: Selenium test that uses Phantomjs headless browser to run test.

Now it is time to change the ValueSourceAttribute, to allow overriding the list of browser with an Environment Variable. While the Json Configuration file to configure tests is a really simple and useful solution for developers, running tests on their machines, when I need to run tests on a bulid server I want to be able to specify the list of browser with a build Variable.

In build vNext, each variable you add to the build will be copied in an environment variable with the same name of the variable, converted in uppercase and with dot char substituted with underscore. If I use variable Selenium.BrowsersToTest the environment variable is called: SELENIUM_BROWSERSTOTEST

Here is the new code of the ValueSourceAttribute that use that environment variable to find list of browsers to use.

public class BrowserList : ValueSourceAttribute
{
    private static IEnumerable Browsers;

    public BrowserList()
        : base(typeof(BrowserList), "Browsers")
    {
        Browsers = GetBrowserFromConfig();
    }

    private static IEnumerable GetBrowserFromConfig()
    {
        var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
        if (!String.IsNullOrEmpty(envVar))
        {
            return envVar.Split('|', ',', ';', ':');
        }
        else
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject)JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }
}

The only change is that the attribute searches an envorinment variable called SELENIUM_BROWSERSTOTEST to grab list of browser. If the variable is not present, it use json configuration file as showed in previous article. Now we can choose browser list directly from build definition.

image

Figure 2: Specifying browser list with Build Variables.

Variables can be specified a Queue Time, this allow the user to change browserslist event when queueing a new build. Here is the result of a run

image

Figure 3: Outcome of the test using browsers list from build variable

Previous build outcome of Figure 3 is obtained running the test with an agent that does not run as a service, because it should be able to launch browser and access UI. If I queue the same build with Hosted Agent, or with an Agent that is running as a service, here is the result.

image

Figure 4: Failure running selenium tests

The problem with queued agent is that he has no firefox installed, but even with firefox installed, it would not be able to run the test because hosted agent runs as a service and has no access to UI. To solve this problem we can modify our ValueSourceAttribute

  public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            
        }

        static BrowserList()
        {
            Browsers = GetBrowserFromConfig();
            if (!Environment.UserInteractive)
            {
                Browsers = Browsers
                    .Where(b => b.ToLower() == "phantomjs")
                    .ToArray();
            }
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
            if (!String.IsNullOrEmpty(envVar))
            {
                return envVar.Split('|', ',', ';', ':');
            }
            else
            {
                var settings = File.ReadAllText("testParameters.json");
                var config = (JObject)JsonConvert.DeserializeObject(settings);
                var seleniumSettings = config["Selenium"];
                var browsers = (JArray)seleniumSettings["BrowsersToTest"];

                return browsers
                    .Select(b => b.ToString())
                    .ToArray();
            }
        }
    }

 

A couple of modification are worth of notice, the first one is that initialization of browser list is done in static constructor, and it will be executed once for each test run.

Then, if the test is not executing in a UserInteractive session, the attribute remove all browsers except phantomjs, the only ones that is guaranteed to run without having access to a UI. With this simple trick we avoid to run tests that will fail because they could not run. To understand which browser can run with agent running as a service you can simply try to run all of them and verify which ones returns error. Actually some browser can run in headless mode (without UI) so they can be used even if the agent has no access to the UI, so use this technique to remove all browsers that does not supports this mode.

To verify that everything works, I changed configuration of my Visual Studio Agent to run as a service instead that running interactively and queued a new build. Here is the result of the Tests.

image

Figure 5: Now only phantomjs test is run because agent is running as a service

Gian Maria.

Parametrize NUnit Selenium Test to run with different browsers

Parametrizing NUnit Tests is a new feature introduced with version 2.5 and this feature can be really useful in a variety of scenarios, but when it is time to use Selenium this is a killer feature.

I’m not going to cover Selenium WebDriver component, but basically it allows to write tests that can drive a Browser to execute test against your web application. In this scenario a killer feature is being able to specify the list of the browsers to use in a way that is completely indipendent from your test.

Thanks to the ValueSourceAttribute obtaining this result is really simple. First of all I create a base class for the test that creates different Selenium Web Drivers based on a string specified as argument.

 public class SeleniumTestFixtureBase
    {
        protected IWebDriver WebDriver;

        public void Initialize(String browser)
        {
            switch (browser.ToLower())
            {
                case "chrome":
                    DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
                    WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
                    break;
                case "firefox":
                    WebDriver = new FirefoxDriver();
                    break;
                default:
                    throw new NotSupportedException("Browser " + browser + " not supported");
            }
        }
    }

This is really trivial, and for this example I’m going to support only chrome and firefox. Then I create a json configuration file where I specify every parameter needed by Unit Tests.

Using an external json file to specify parameters for your Unit Test makes trivial passing those parameters for every runner (VS, TeamCity, GUI, command line runner).

This is a simple configuration file that contains only the list of the browsers I want to use. Remember to ask Visual Studio to copy this file in output folder when it changes.

{
  "Selenium": {
    "BrowsersToTest": [ "Firefox", "Chrome", "IE"]
  }
}

I’ve choosen Json because it is easy to write and easy to parse. Now it is time to create my attribute based on ValueSourceAttribute; it will read the above configuration file and provide the list of the browsers I want to use.

public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            Browsers = GetBrowserFromConfig();
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject) JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }

Again, this is a simple and trivial class that simply looks for a testParameters.json in the test run directory and search for the Selenium.BrowserToTest array, that containing the list of the browser to use as test.

Thanks to ValueSourceAttribute we can drive and parametrize the test with a simple external file.

Thanks to those two classes, I can simply specify for each test the Browsers List I want to use.

    [TestFixture]
    public class ParametrizedSeleniumTest : SeleniumTestFixtureBase
    {
        [Test]
        public void Test_Browsers([BrowserList] String browser)
        {
            base.Initialize(browser);
            WebDriver.Navigate().GoToUrl("http://www.microsoft.com");

            WebDriver.Close();
        }
    }

Using BrowserList attribute I’m asking Nunit to create an instance of that attribute to get the list value to be bind to that specific parameter. For each value a different test is created. Now if I build my project I can verify from VS Test Runner that indeed I have three different test to run.

image

 Figure 1: My test running with different versions of browsers

If I run the tests, as I’m expecting, the IE based test fails because I’ve not configured my test to use IE Driver.

image

Figure 2: Test output with Visual Studio Test Runner.

Now remove the IE from the list of Browsers to use in the testParameters.json and rebuild the solution. The IE version of the test now is disappeared from Test Runner.

image

Figure 3: Changing test configuration parameter and rebuilding will update test list

This technique does not depend on anyhthing, and can be used to run test with different configuration if you need (dev machine, different build machines, etc).

Thanks to this technique you can specify whitch browser to use with a simple configuration file and this is a killer feature if you are planning to run test during the build. You can simply modify your testParameters.json file before running the test in your Build to choose which browser to use, or disable completely selenium testing specifying an empty array.

Another option is keeping the list of browsers to use as a comma separated string stored in an Environment Variable, something like

SELENIUM_BROWSERS=Ie,chrome,firefox

With such a technique you can run the test and let the environment specify whitch browser are available to use for testing in that environment.

Gian Maria.

Quickly run Unit Test With VS 2012/2013/..

With VS 2012 and newer versions we can run Unit Tests from various frameworks directly from Visual Studio IDE, thanks to the concept of Test Adapters. When you are doing Test Driven Development you usually go with Red/Green/Refactor workflow; what you need is a way to quickly run all or part of your unit tests after you modified the code. The quickest solution is using the option to Run Tests After Build but it is available only for Premium and Ultimate edition, but you can also run test with little manual intervention resorting to Keyboard Shortcut.

Simply go to TOOLS->Customize menu, then choose to customize keyboard.

image

Figure 1: Keyboard customization in Visual Studio

Once configuration window is opened, search for TestExplorer.RunAllTests command, place cursor in the “Press shortcut keys:” textbox and press a shortcut, then press “assign” button to assign to this command. In my standard configuration I like to have CTRL+SHIFT+ALT+A shortcut, because it is not assigned to any other command and it is easy to press with left hand.

image

Figure 2: Assign CTRL+SHIFT+ALT+A to TestExplorer.RunAllTests command

Now I can write code, use shortcut and VS will build solution and run test for me automatically, without leaving my hands from the keyboard. Thanks to the various test adapter and the various grouping and filtering possibility offered by Test Explorer, you can do TDD in Visual Studio without the need of third party tools.

Gian Maria.

Nunit test not found for some assemblies with Visual Studio Test Runner

I’ve a project in Visual Studio 2013 where one of the assembly containing Tests refuses to show tests in Test Explorer window. The solution has tests written both in Nunit and in MSpec, and everything is good except for that specific assembly. If you notice that Test Explorer window misses some tests, the first thing you need to check is the output windows, where you can find some output for Test Adapters.

image

Figure 1: Output for test adapters can be found in standard Output Window

In the above situation the assembly that is ignored is Intranet.Tests.dll and the reason is that it is built for Framework45 and x64 platform, while the test runner is executing against x86 platform. Everything seems ok, every project is compiled in .ANY CPU, but looking at the raw project file I can confirm that PlatformTarget is set in x64. Changing to x86 (or removing it completely) solves the problem.

image

Figure 2: Platform target changed from x64 to x86

After I changed the PlatformTarget attribute, all tests belonging to that assembly are now available Test Explorer window.

Gian Maria.

Programmatically use of Coded UI in Visual Studio

Coded UI Tests are a specific type of UI testing introduced with Visual Studio 2010. You can create your first Coded UI test following simple instruction from MSDN documentation. Most of the introductory examples shows you how you can use the Recorder tools to record interaction with a software (Web, WinForm, Wpf. etc) to generate what is called a UiMap. An UiMap is nothing more than a big Xml files where the recorder records the interaction with the UI and a bunch of automatic generated classes to interact with the UI.

Using a UiMap is probably not the best option for large projects, because the cost of maintaining it could become really high. This is usually not a big problem, because UiMap is used to generate code based on a set of classes belonging to Visual Studio Testing Framework that makes possible to interact with a UI from code. If maintaining a UiMap is difficult for you you can directly use these classes in your test. To show you the “hello world” equivalent of CUIT, here is the code needed to open a page in a browser and click an hyperlink.

using ( BrowserWindow browserWindow =
            BrowserWindow.Launch
            (
                new System.Uri("http://tailspintoys.azurewebsites.net/")
            ))
{
               
    HtmlHyperlink link = new HtmlHyperlink(browserWindow);
    link.SearchProperties.Add
        (
            HtmlHyperlink.PropertyNames.InnerText,
            "Model Airplanes"
        );
    Mouse.Click(link); 
}

The code is really simply, you must use the BrowserWindow.Launch static method to create an instance of BrowserWindow class pointing to a given Url. The BrowserWindow class is a wrapper defined Visual Studio Coded Ui assemby used to abstract the interaction with a web browser. The next step is locating the hyperlink you want to click, operation that can be accomplished with the HtmlHyperlink object. This object derives from the UiTestControl base class, and abstracts the concept of a control in the User Interface. The constructor of HtmlHyperlink object needs an instance of a containing control, in this example the whole browserWindows object. The need for the Container is having a root control that will be searched for the control.

To specify the exact Hyperlink control you want to interact with, you should populate SearchProperties collection, specifying the criteria you want to use. In this example I used the InnerText property, but you can use a lot of other criteria. Thanks to PropertyNames static collection of HtmlHyperlink object you can enumerate all the properties that can be used to locate the control. Inner Text is not usually the best option, using unique Id is usually a better approach, but the key concept is: You should use the criteria that is most stable in your scenario/environment. If you can ask to development team to assign unique id or unique names to each control, tests will be more robust and quicker.

Once SearchProperties collection is filled with critera, you can interact with the control, accessing properties or passing it to Mouse.Click method to simulate a click. CodedUI engine will locate the control on the page only when You will access properties or pass the control to some method that interact with it. This is really important, until you do not access properties the engine will not try to locate the control on the UI.

Remember to enclose the BrowserWindow object in a using block, this will ensure that the instance of the browser opened during the test will be always closed. This prevents multiple browser windows to remain opened after the test if some exception occurred.

Gian Maria.