Create Parametrized test to allow for simpler Builds

When it is time of running unit test in a TFS or TeamCity Build, often you face the problem to run tests with options different from those one used in Developer Machine. As an example we have tons of tests that requires a MongoDb and and ElasticSearch or Solr integration.

While it is quite normal for developers to have everything installed in local dev box, it could be annoying to provide MongoDb and ElasticSearch installed on all agent machines. This approach complicates the setup of a build servers and create a situation that is less manageable.

While there is the ability to create a dedicated pool composed only by agent that have MongoDb and ElasticSearch installed, I prefer being able to run my test in all test agents, without any restriction.

The best solution is having parametrized tests, so you can execute tests with differnet parameters during the build, as an example you should parametrize connection strings.

Having parametrized tests greatly improve the ability to run test during build without creating complex requirement for agents.

In .NET environment it is quite common to use app.settings file to contain all connection strings, here it is an example taken from one of our project.

  
		
		
		
		
		
    

  

All Tests use ConfigurationManager object to access connectionstring from configuration file, and there is a single point where we specified all connection strings used by tests.

The obvious problem of storing setting in app.config is that this file is source controlled, and it is not possible to have different settings for different machine / developers.

A possible solution to this approach is using a powershell script that modifies all the configuration files in bin directories before running the test. Here is a naive approach with PowerShell.

param(
    [string] $baseMongoConnection = "mongodb://admin:xxxxxx##localhost/{0}",
    [string] $connectionQueryString = "?authSource=admin",
    [string] $configuration = "debug"
)

##Logging tests
$configFileName = "..\Logging\Jarvis.Framework.LoggingTests\bin\$configuration\Jarvis.Framework.LoggingTests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='testDb']/@connectionString" -value "$baseMongoConnection$connectionQueryString"

$xml.save($configFileName)

##main tests
$configFileName = "..\Jarvis.Framework.Tests\bin\$configuration\Jarvis.Framework.Tests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='eventstore']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-es-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='saga']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-saga-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='readmodel']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-readmodel-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='system']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-system-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='engine']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-engine-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='rebus']/@connectionString" -value ($baseMongoConnection -f "jarvis-rebus-test" + $connectionQueryString)

$xml.save($configFileName)

function Edit-XmlNodes {
param (
     $doc = $(throw "doc is a required parameter"),
    [string] $xpath = $(throw "xpath is a required parameter"),
    [string] $value = $(throw "value is a required parameter"),
    [bool] $condition = $true
)    
    if ($condition -eq $true) {
        $nodes = $doc.SelectNodes($xpath)
         
        foreach ($node in $nodes) {
            if ($node -ne $null) {
                if ($node.NodeType -eq "Element") {
                    $node.InnerXml = $value
                }
                else {
                    $node.Value = $value
                }
            }
        }
    }
}

Do not shoot the pianist :), this is a quick and dirty script that can edit configuration files, but this is not a good approach.

1) If a developer want to change this setting in its machine, it is really complex to instruct VS to run this script with parameter before running the test
2) It lead to unnecessary complicated builds, because you need to run this script before running the test and it introduces another point of failure.
3) It is not possibile to have agent dependant settings, I cannot specify that Agent X should run the test against mongo instance Y.

A better solution is to use Environment Variables to override app.config connection string, and create a Nunit SetupFixture that is executed before the first test.

NUnit has the ability to run a global setup that is run before the very first is run, and it is the perfect place where you can put logic to change configuration of the tests. In the following example the init script check some environment variables, then changes the connectionstring.


[SetUpFixture]
public class GlobalSetup
{
    [SetUp]
    public void ShowSomeTrace()
    {
        var overrideTestDb = Environment.GetEnvironmentVariable("TEST_MONGODB");
        if (String.IsNullOrEmpty(overrideTestDb)) return;

        var overrideTestDbQueryString = Environment.GetEnvironmentVariable("TEST_MONGODB_QUERYSTRING");
        var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        var connectionStringsSection = (ConnectionStringsSection)config.GetSection("connectionStrings");
        connectionStringsSection.ConnectionStrings["eventstore"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-es-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["saga"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-saga-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["readmodel"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-readmodel-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["system"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-system-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["engine"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-engine-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["rebus"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-rebus-test" + overrideTestDbQueryString;
        config.Save();
        ConfigurationManager.RefreshSection("connectionStrings");
    }
}

This approach has numerous advantages:

1) It works even for developers workstations, if you want to use a different connection string just populate corresponding Environment Variable and you are ready to go
2) You can simply define variables that are valid for all agents on the build definition, or setup environment variables different for each build agent, so each build agent will run tests against different Mongo instances/database.

This means that one of the best approach is parametrizing the tests with defaults that are good for developer machines, then allow override of configuration with Environment Variables to allow easy configuration for Build Agents.

Gian Maria

Running Unit Tests on different machine during TFS 2015 build

First of all I need to thanks my friend Jackob Ehn that pointed me to the right direction to create a particular build.  In this post I’ll share with you my journey to run tests on a different machine than the one that is running the build.

For some build it is interesting to have the ability to run some Unit Test (nunit in my scenario) on a machine different from that one that is running the build. There are a lot of legitimate reasons for doing this, for a project I’m working with, to run a set of test I need to have a huge amount of pre-requisites installed (LibreOffice, ghostscript, etc). Instead of installing those prerequisite on all agent machines, or install those one on a single build agent and using capabilities, I’d like to being able to run the build on any build agent, but run the test in a specific machine that had all the prerequisite installed.

Sometimes it is necessary to run tests during build on machine different from that one where the build agent is running.

The solution is quite simple, because VSTS / TFS already had all build tasks needed to solve my need and to execute tests on different machine.

The very first steps is copying all the dlls that contains tests on the target machine, this is accomplished by the Windows Machine File Copy task.

image 

Figure 1: File copy task in action

This is a really simple task, the only suggestion is to never specify the password in clear format, because everyone that can edit the build can read the password. In this situation the password is stored in the RmTestAdminPassword variable, and that variable is setup as secret.

image

Figure 2: Store sensitive information as secret variables of the build

Then we need to add a Visual Studio Test Agent Deployment task, to deploy Visual Studio Test Runner on target machine.

image

Figure 3: Visual Studio Test Agent Deployment

Configuration is straightforward, you need to specify a machine group or a list of target machines (point 1), then you should specify the user that will be used to run test agent (point 2), finally I’ve specified a custom location in my network for the Test Agent Installer. If you do not specify anything, the agent is downloaded from http://go.microsoft.com/fwlink/?LinkId=536423 but this will download approximately 130MB of data. For faster build it would be preferrable to download the agent and move the installer in a shared network folder to instruct the Task to grab the agent from that location.

Finally you use the Run Functional Tests task to actually execute tests in the target machine.

image

Figure 4: Running functional test from target machine configuration

You specify the machine(s) you want to use (point 1), then all the dll that contains tests (point 2) and you can also specify code coverage (point 3). Even if the task is called Run Functional Tests, it actually use Visual Studio Test runner to run tests, so you can run whatever test you like.

Thanks to TFS 2015 / VSTS build, we already have all tasks needed to run Unit Tests on target machines.

If you are running Nunit test or whatever test framework different from MsTest, this task will fail, because the target machine has no test adapter to run the test. The failure output tells you that the agent was not capable of finding any test to run in specified location. This happens even if you added Nuget Nunit adapter to the project. The solution is simple, first of all locates all needed dll in package location of your project.

image

Figure 5: Installing Nunit TestAdapter Nuget package, downloads all required dll to your machine

Once you located those four dll in your HD, you should copy them to the target machine in folder: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions, this folder was created by Visual Studio Test Agent deployment task and will contain all extension that will be automatically loaded by Visual Studio Test Agent. Once you copied the dll, those machine will be able to run nunit test without problem.

Once you copied all required dll in target folder, re-run the build, and verify that tests are indeed executed on the target machine.

SNAGHTML3c4ad0

Figure 6: Output of running tests on remote machine

Test output is transferred to the build machine, and attached to the build result as usual, so you do not need anything else to visualize test result in the same way as if the test were executed by agent machine.

image

Figure 7: Output of test is included in the build output like standard Unit Tests ran by the build agent

Once tasks are in place, everything is carried over by the test agent, test results are downloaded and attached to the build results, as for standard unit tests executed on Build Agent machine.

The only drawback of this approach is that it needs some times (in my system about 30 seconds) before the test started execution in target machine, but apart this problem, you can execute tests on remote machine with little effort.

Gian Maria.

Run Selenium Test in build vNext

Previous Articles:

To run a Selenium test in a build vNext there are some modification to do apply to previous example. Let’s see how simple is running our Selenium Tests in a VSTS Build vNext.

The first modification requires adding a reference to PhantomJS, an Headless browser based on webkit that is capable of browsing a site and run javascript without a UI. Since we are interested in running test in a build server, this is a requirement if the agent runs as a service and has not access to a UI. To use PhantomJS just reference PhantomJS package on nunit and modify your base class to support this browser.

switch (browser.ToLower())
{
    case "chrome":
        DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
        WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
        break;
    case "firefox":
        WebDriver = new FirefoxDriver();
        break;
    case "phantomjs":
        WebDriver = new PhantomJSDriver();
        break;

Using Phantomjs is just a matter of creating a PhantomJSDriver with Selenium WebDriver test and the game is done. Now add “phantomjs” to test config file and you should be able to run the test.

image

Figure 1: Selenium test that uses Phantomjs headless browser to run test.

Now it is time to change the ValueSourceAttribute, to allow overriding the list of browser with an Environment Variable. While the Json Configuration file to configure tests is a really simple and useful solution for developers, running tests on their machines, when I need to run tests on a bulid server I want to be able to specify the list of browser with a build Variable.

In build vNext, each variable you add to the build will be copied in an environment variable with the same name of the variable, converted in uppercase and with dot char substituted with underscore. If I use variable Selenium.BrowsersToTest the environment variable is called: SELENIUM_BROWSERSTOTEST

Here is the new code of the ValueSourceAttribute that use that environment variable to find list of browsers to use.

public class BrowserList : ValueSourceAttribute
{
    private static IEnumerable Browsers;

    public BrowserList()
        : base(typeof(BrowserList), "Browsers")
    {
        Browsers = GetBrowserFromConfig();
    }

    private static IEnumerable GetBrowserFromConfig()
    {
        var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
        if (!String.IsNullOrEmpty(envVar))
        {
            return envVar.Split('|', ',', ';', ':');
        }
        else
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject)JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }
}

The only change is that the attribute searches an envorinment variable called SELENIUM_BROWSERSTOTEST to grab list of browser. If the variable is not present, it use json configuration file as showed in previous article. Now we can choose browser list directly from build definition.

image

Figure 2: Specifying browser list with Build Variables.

Variables can be specified a Queue Time, this allow the user to change browserslist event when queueing a new build. Here is the result of a run

image

Figure 3: Outcome of the test using browsers list from build variable

Previous build outcome of Figure 3 is obtained running the test with an agent that does not run as a service, because it should be able to launch browser and access UI. If I queue the same build with Hosted Agent, or with an Agent that is running as a service, here is the result.

image

Figure 4: Failure running selenium tests

The problem with queued agent is that he has no firefox installed, but even with firefox installed, it would not be able to run the test because hosted agent runs as a service and has no access to UI. To solve this problem we can modify our ValueSourceAttribute

  public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            
        }

        static BrowserList()
        {
            Browsers = GetBrowserFromConfig();
            if (!Environment.UserInteractive)
            {
                Browsers = Browsers
                    .Where(b => b.ToLower() == "phantomjs")
                    .ToArray();
            }
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
            if (!String.IsNullOrEmpty(envVar))
            {
                return envVar.Split('|', ',', ';', ':');
            }
            else
            {
                var settings = File.ReadAllText("testParameters.json");
                var config = (JObject)JsonConvert.DeserializeObject(settings);
                var seleniumSettings = config["Selenium"];
                var browsers = (JArray)seleniumSettings["BrowsersToTest"];

                return browsers
                    .Select(b => b.ToString())
                    .ToArray();
            }
        }
    }

 

A couple of modification are worth of notice, the first one is that initialization of browser list is done in static constructor, and it will be executed once for each test run.

Then, if the test is not executing in a UserInteractive session, the attribute remove all browsers except phantomjs, the only ones that is guaranteed to run without having access to a UI. With this simple trick we avoid to run tests that will fail because they could not run. To understand which browser can run with agent running as a service you can simply try to run all of them and verify which ones returns error. Actually some browser can run in headless mode (without UI) so they can be used even if the agent has no access to the UI, so use this technique to remove all browsers that does not supports this mode.

To verify that everything works, I changed configuration of my Visual Studio Agent to run as a service instead that running interactively and queued a new build. Here is the result of the Tests.

image

Figure 5: Now only phantomjs test is run because agent is running as a service

Gian Maria.

Parametrize NUnit Selenium Test to run with different browsers

Parametrizing NUnit Tests is a new feature introduced with version 2.5 and this feature can be really useful in a variety of scenarios, but when it is time to use Selenium this is a killer feature.

I’m not going to cover Selenium WebDriver component, but basically it allows to write tests that can drive a Browser to execute test against your web application. In this scenario a killer feature is being able to specify the list of the browsers to use in a way that is completely indipendent from your test.

Thanks to the ValueSourceAttribute obtaining this result is really simple. First of all I create a base class for the test that creates different Selenium Web Drivers based on a string specified as argument.

 public class SeleniumTestFixtureBase
    {
        protected IWebDriver WebDriver;

        public void Initialize(String browser)
        {
            switch (browser.ToLower())
            {
                case "chrome":
                    DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
                    WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
                    break;
                case "firefox":
                    WebDriver = new FirefoxDriver();
                    break;
                default:
                    throw new NotSupportedException("Browser " + browser + " not supported");
            }
        }
    }

This is really trivial, and for this example I’m going to support only chrome and firefox. Then I create a json configuration file where I specify every parameter needed by Unit Tests.

Using an external json file to specify parameters for your Unit Test makes trivial passing those parameters for every runner (VS, TeamCity, GUI, command line runner).

This is a simple configuration file that contains only the list of the browsers I want to use. Remember to ask Visual Studio to copy this file in output folder when it changes.

{
  "Selenium": {
    "BrowsersToTest": [ "Firefox", "Chrome", "IE"]
  }
}

I’ve choosen Json because it is easy to write and easy to parse. Now it is time to create my attribute based on ValueSourceAttribute; it will read the above configuration file and provide the list of the browsers I want to use.

public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            Browsers = GetBrowserFromConfig();
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject) JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }

Again, this is a simple and trivial class that simply looks for a testParameters.json in the test run directory and search for the Selenium.BrowserToTest array, that containing the list of the browser to use as test.

Thanks to ValueSourceAttribute we can drive and parametrize the test with a simple external file.

Thanks to those two classes, I can simply specify for each test the Browsers List I want to use.

    [TestFixture]
    public class ParametrizedSeleniumTest : SeleniumTestFixtureBase
    {
        [Test]
        public void Test_Browsers([BrowserList] String browser)
        {
            base.Initialize(browser);
            WebDriver.Navigate().GoToUrl("http://www.microsoft.com");

            WebDriver.Close();
        }
    }

Using BrowserList attribute I’m asking Nunit to create an instance of that attribute to get the list value to be bind to that specific parameter. For each value a different test is created. Now if I build my project I can verify from VS Test Runner that indeed I have three different test to run.

image

 Figure 1: My test running with different versions of browsers

If I run the tests, as I’m expecting, the IE based test fails because I’ve not configured my test to use IE Driver.

image

Figure 2: Test output with Visual Studio Test Runner.

Now remove the IE from the list of Browsers to use in the testParameters.json and rebuild the solution. The IE version of the test now is disappeared from Test Runner.

image

Figure 3: Changing test configuration parameter and rebuilding will update test list

This technique does not depend on anyhthing, and can be used to run test with different configuration if you need (dev machine, different build machines, etc).

Thanks to this technique you can specify whitch browser to use with a simple configuration file and this is a killer feature if you are planning to run test during the build. You can simply modify your testParameters.json file before running the test in your Build to choose which browser to use, or disable completely selenium testing specifying an empty array.

Another option is keeping the list of browsers to use as a comma separated string stored in an Environment Variable, something like

SELENIUM_BROWSERS=Ie,chrome,firefox

With such a technique you can run the test and let the environment specify whitch browser are available to use for testing in that environment.

Gian Maria.

Build vNext, support for deploying bits to Windows machines

One of the most interesting trend of DevOps movement is continuous deployment using build machines. Once you get your continuous build up and running, the next step is customizing the build to deploy on one or more test environments. If you do not need to deploy in production, there is no need of a controlled release pipeline (Ex: Release Management) and using a simple build is the most productive choiche. In this scenario one of the biggest pain is moving bits from the build machine to target machines. Once build output is moved to a machine, installing bits is usually only a matter of using some PowerShell script.

In Build, Deploy, Test scenario, quite often copying build output to target machine is the most difficult part

Thanks to Build vNext solving this problem is super easy. If you go to your visualstudio.com account, and choose the TEST hub, you can notice a submenu called machines.

image

Figure 1: Machines functionality in Visualstudio.com

This new menu is related to a new feature, still in preview, used to define groups of machine that can be use for deploy and testing workflows. In Figure 1 you notice a group called Cyberpunk1. Creating a group is super-easy, you just need to give it a name, specify administrative credentials and the list of the machines that will compose the group. You can also use different credentials for each machine, but using machine in Active Directory domain is usually the simplest scenario.

image

Figure 1: Editing of machine groups

Actually this feature still does not support Azure Virtual Machines, but you can easily target machine in your on-premise infrastructure. You just need to be sure that

  • All machines are reachable from the machine where the build agent is running
  • Check your DNS to verify that the names resolution is ok
  • All target machines should have Powershell remoting enabled
  • All target machines should have sharing of file system enabled
  • Firewall port are opened.

I’ve tested with an environment where both machines are running Windows Server 2012 R2, with latest update and file sharing enabled. Once you defined a machine group, you can use it to automatically copy files from build agent to all machines with a simple task of build vNext.

SNAGHTML3b78ea

Figure 3: Windows machine File Copy task

Thanks to this simple task you can simply copy files from build machine to destination machine, without the need to install any agent or other components. All you need to do is choose the machine group, target folder and source folder.

If you get error running the build, a nice new feature of build vNext is the ability to download full log as zip, where all the logs are separated by tasks

image

Figure 4: All build logs are separated for each step, to simplify troubleshooting

Opening the file 4_Copy … you can read logs related to the copy build step, to understand why the step is failing. Here is what I find in one of my build that failed.

Failed to connect to the path \\vsotest.cyberpunk.local with the user administrator@cyberpunk.local for copying.System error 53 has occurred. 
 2015-06-10T08:35:02.5591803Z The network path was not found.

In this specific situation the RoboCopy tool is complaining that the network path was not found, because I forgot to enable file sharing on the target machine. Once I enabled file sharing an running again the build everything was green, and I can verify that all files were correctly copied on target machines.

As a general rule, whenever a build fails, download all log and inspect the specific log for the task that failed.

image

Figure 5: Sample application was correctly copied to target machines.

In my first sample, I’ve used TailspinToys sample application, I’ve configured MsBuild to use StagingDirectory as output folder with parameter OutDir: /p:OutDir=$(build.stagingDirectory) and thanks to Windows Machine File Copy task all build output is automatically copied on target machines.

Once you got your build output copied on target machine, you need only to create a script to install the new bits, and maybe some integration test to verify that the application is in healthy state.

Gian Maria