Running UAT and integration tests during a VSTS Build

There are a lots of small suggestions I’ve learned from experience when it is time to create a suite of integration / UAT test for your project. A UAT or integration test is a test that exercise the entire application, sometimes composed by several services that are collaborating to create the final result. The difference from UAT tests and Integration test, in my personal terminology, is that the UAT uses direct automation of User Interface, while an integration tests can skip the UI and exercise the system directly from public API (REST, MSMQ Commands, etc).

The typical problem

When it is time to create a solid set of such kind of tests, having them to run in in an automated build is a must, because it is really difficult for a developer to run them constantly as you do for standard Unit Tests.

Such tests are usually slow, developers cannot waste time waiting for them to run before each commit, we need to have an automated server to execute them constantly while the developer continue to work.

Those tests are UI invasive, while the UAT test runs for a web project, browser windows continues to open and close, making virtually impossible for a developer to run an entire UAT suite while continue working.

Integration tests are resource intensive, when you have 5 services, MongoDB, ElasticSearch and a test engine that fully exercise the system, there is little more resource available for doing other work, even if you have a real powerful machine.

Large sets of Integration / UAT tests are usually really difficult to run for a developer, we need to find a way to automate everything.

Creating a build to run everything in a remote machine can be done with VSTS / TFS, and here is an example.

image

Figure 1: A complete build to run integration and UAT tests.

The build is simple, the purpose is having a dedicated Virtual Machine with everything needed to run the software already in place. Then the build will copy the new version of the software in that machine, with all the integration test assemblies and finally run the tests on the remote machine.

Running Integration and UAT test is a task that is usually done in a different machine from that one running the build. This happens because that machine should be carefully prepared to run test and simplify deployment of the new version.

Phase 1 – Building everything

First of all I have the phase 1, where I compile everything. In this example we have a solution that contains all .NET code, then a project with Angular 2, so we first build the solution, then npm restore all the packages and compile the application with NG Cli, finally I publish a couple of Web Site. Those two web sites are part of the original solution, but I publish them separately with MSBuild command to have a full control on publish parameters.

In this phase I need to build every component, every service, every piece of the UI needed to run the tests. Also I need to build all the test assemblies.

Phase 2 – Pack release

In the second phase I need to pack the release, a task usually accomplished by a dedicated PowerShell script included in source code. That script knows where to look for dll, configuration files, modify configuration files etc, copying everything in a couple of folders: masters and configuration. In the masters directory we have everything is needed to run everything.

To simplify everything, the remote machine that will run the tests, is prepared to accept an XCopy deployment. This means that the remote machine is already configured to run the software in a specific folder. Every prerequisite, everything is needed by every service is prepared to run everything from a specific folder.

This phase finish with a couple of Windows File Machine copy to copy this version on the target computer.

Phase 3 – Pack UAT testing

This is really similar to Phase 2, but in this phase the pack PowerShell scripts creates a folder with all the dll of UAT and integration tests, then copy all test adapters (we use NUnit for UnitTesting). Once pack script finished, another Windows File Machine Copy task will copy all integration tests on the remote machine used for testing.

Phase 4 – Running the tests

This is a simple phase, where you use Deploy Test Agent on test machine followed by a Run Functional Test tasks. Please be sure always place a Deploy Test Agent task before EACH Run Functional Test task as described in this post that explain how to Deploy Test Agent and run Functional Tests

Conclusions

For a complex software, creating an automated build that runs UAT and integration test is not always an easy task and in my experience the most annoying problem is setting up WinRm to allow remote communication between agent machine and Test Machine. If you are in a Domain everything is usually simpler, but if for some reason the Test Machine is not in the domain, prepare yourself to have some problem before you can make the two machine talk togheter.

In a next post I’ll show you how to automate the run of UAT and Integration test in a more robust and more productive way.

Gian Maria.

Manage Environment Variables during a TFS / VSTS Build

To avoid creating unnecessary build definition, it is a best practice to allow for parameter overriding in every task that can be executed from a build. I’ve dealt on how to parametrize tests to use a different connection string when tests are executed during the build and I’ve used Environment variables for a lot of reasons.

Environment variables are not source controlled, this allows every developer to override settings in own machine without disturbing other developers. If I do not have a MongoDb in my machine I can simply choose to use some other instance in my network.

image

Figure 1: Overriding settings with environment variables.

Noone in the team is affected by this settings, and everyone has the freedom of changing this value to whatever he/she like. This is important because you can have different version of MongoDb installed in your network, with various different configuration (MMapV1 or Wired Tiger) and you want the freedom to choose the instance you want to use.

Another interesting aspect of Environment variables, is that they can be set during a VSTS/TFS build directly from build definition. This is possible because Variables defined for a build were set as environment variables when the build runs.

image

Figure 2: Specifing environment variables directly from Variables tab of a bulid

If you allow this value to be allowed at Queue Time, you can set the value when you manually queue a build.

image

Figure 3: Specifying variables value at Queue Time

If you look at Figure 3, you can verify that I’m able to change the value at queue time, but also, I can simply press “Add variable” to add any variable, even if it is not included in build definition. In this specific situation I can trigger a build and have my tests run against a specific MongoDb instance.

Remember that the value specified in the build definition overrides any value that is set on environment variables on build machine. This imply that, once you set a value in the Build definition, you are not able to change the value for a specific build agent.

It you want to be able to choose a different value for each build agent machine you can simply avoid setting the value on the Variables tab and instead define the variable on each build machine to have a different value for each agent. Another alternate approach is using two Environment variables, Es: TEST_MONGODB and TEST_MONGODB_OVERRIDE, and configure your tests to use TEST_MONGODB_OVERRIDE if present, if not present use TEST_MONGODB. This allows you to use TEST_MONGODB on build definition, but if you set TEST_MONGODB_OVERRIDE for a specific test agent, that agent will use that value.

Another interesting aspect of Environment Variable is that they are included in agent capabilities, as you can see from Figure 4.

SNAGHTMLd39383

Figure 4: All environment variables are part of agent Capabilities

This is an important aspect because if you want that variable to be set in the agent, you can avoid to include in Variables tab, and you can require this build to be run on an agent that has TEST_MONGODB environment variable specified.

image

Figure 5: Add a demand for a specific Environment Variable to be defined in Agent Machine

Setting the demands is not always necessary, in my example if the TEST_MONGODB variable is not definied, tests are executed against local MongDb instance. It is always a good strategy to use a reasonable default if some setting is not present in Environment Variables.

Gian Maria.

Create Parametrized test to allow for simpler Builds

When it is time of running unit test in a TFS or TeamCity Build, often you face the problem to run tests with options different from those one used in Developer Machine. As an example we have tons of tests that requires a MongoDb and and ElasticSearch or Solr integration.

While it is quite normal for developers to have everything installed in local dev box, it could be annoying to provide MongoDb and ElasticSearch installed on all agent machines. This approach complicates the setup of a build servers and create a situation that is less manageable.

While there is the ability to create a dedicated pool composed only by agent that have MongoDb and ElasticSearch installed, I prefer being able to run my test in all test agents, without any restriction.

The best solution is having parametrized tests, so you can execute tests with differnet parameters during the build, as an example you should parametrize connection strings.

Having parametrized tests greatly improve the ability to run test during build without creating complex requirement for agents.

In .NET environment it is quite common to use app.settings file to contain all connection strings, here it is an example taken from one of our project.

  
		
		
		
		
		
    

  

All Tests use ConfigurationManager object to access connectionstring from configuration file, and there is a single point where we specified all connection strings used by tests.

The obvious problem of storing setting in app.config is that this file is source controlled, and it is not possible to have different settings for different machine / developers.

A possible solution to this approach is using a powershell script that modifies all the configuration files in bin directories before running the test. Here is a naive approach with PowerShell.

param(
    [string] $baseMongoConnection = "mongodb://admin:xxxxxx##localhost/{0}",
    [string] $connectionQueryString = "?authSource=admin",
    [string] $configuration = "debug"
)

##Logging tests
$configFileName = "..\Logging\Jarvis.Framework.LoggingTests\bin\$configuration\Jarvis.Framework.LoggingTests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='testDb']/@connectionString" -value "$baseMongoConnection$connectionQueryString"

$xml.save($configFileName)

##main tests
$configFileName = "..\Jarvis.Framework.Tests\bin\$configuration\Jarvis.Framework.Tests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='eventstore']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-es-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='saga']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-saga-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='readmodel']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-readmodel-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='system']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-system-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='engine']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-engine-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='rebus']/@connectionString" -value ($baseMongoConnection -f "jarvis-rebus-test" + $connectionQueryString)

$xml.save($configFileName)

function Edit-XmlNodes {
param (
     $doc = $(throw "doc is a required parameter"),
    [string] $xpath = $(throw "xpath is a required parameter"),
    [string] $value = $(throw "value is a required parameter"),
    [bool] $condition = $true
)    
    if ($condition -eq $true) {
        $nodes = $doc.SelectNodes($xpath)
         
        foreach ($node in $nodes) {
            if ($node -ne $null) {
                if ($node.NodeType -eq "Element") {
                    $node.InnerXml = $value
                }
                else {
                    $node.Value = $value
                }
            }
        }
    }
}

Do not shoot the pianist :), this is a quick and dirty script that can edit configuration files, but this is not a good approach.

1) If a developer want to change this setting in its machine, it is really complex to instruct VS to run this script with parameter before running the test
2) It lead to unnecessary complicated builds, because you need to run this script before running the test and it introduces another point of failure.
3) It is not possibile to have agent dependant settings, I cannot specify that Agent X should run the test against mongo instance Y.

A better solution is to use Environment Variables to override app.config connection string, and create a Nunit SetupFixture that is executed before the first test.

NUnit has the ability to run a global setup that is run before the very first is run, and it is the perfect place where you can put logic to change configuration of the tests. In the following example the init script check some environment variables, then changes the connectionstring.


[SetUpFixture]
public class GlobalSetup
{
    [SetUp]
    public void ShowSomeTrace()
    {
        var overrideTestDb = Environment.GetEnvironmentVariable("TEST_MONGODB");
        if (String.IsNullOrEmpty(overrideTestDb)) return;

        var overrideTestDbQueryString = Environment.GetEnvironmentVariable("TEST_MONGODB_QUERYSTRING");
        var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        var connectionStringsSection = (ConnectionStringsSection)config.GetSection("connectionStrings");
        connectionStringsSection.ConnectionStrings["eventstore"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-es-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["saga"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-saga-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["readmodel"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-readmodel-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["system"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-system-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["engine"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-engine-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["rebus"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-rebus-test" + overrideTestDbQueryString;
        config.Save();
        ConfigurationManager.RefreshSection("connectionStrings");
    }
}

This approach has numerous advantages:

1) It works even for developers workstations, if you want to use a different connection string just populate corresponding Environment Variable and you are ready to go
2) You can simply define variables that are valid for all agents on the build definition, or setup environment variables different for each build agent, so each build agent will run tests against different Mongo instances/database.

This means that one of the best approach is parametrizing the tests with defaults that are good for developer machines, then allow override of configuration with Environment Variables to allow easy configuration for Build Agents.

Gian Maria

Run Selenium Test in build vNext

Previous Articles:

To run a Selenium test in a build vNext there are some modification to do apply to previous example. Let’s see how simple is running our Selenium Tests in a VSTS Build vNext.

The first modification requires adding a reference to PhantomJS, an Headless browser based on webkit that is capable of browsing a site and run javascript without a UI. Since we are interested in running test in a build server, this is a requirement if the agent runs as a service and has not access to a UI. To use PhantomJS just reference PhantomJS package on nunit and modify your base class to support this browser.

switch (browser.ToLower())
{
    case "chrome":
        DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
        WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
        break;
    case "firefox":
        WebDriver = new FirefoxDriver();
        break;
    case "phantomjs":
        WebDriver = new PhantomJSDriver();
        break;

Using Phantomjs is just a matter of creating a PhantomJSDriver with Selenium WebDriver test and the game is done. Now add “phantomjs” to test config file and you should be able to run the test.

image

Figure 1: Selenium test that uses Phantomjs headless browser to run test.

Now it is time to change the ValueSourceAttribute, to allow overriding the list of browser with an Environment Variable. While the Json Configuration file to configure tests is a really simple and useful solution for developers, running tests on their machines, when I need to run tests on a bulid server I want to be able to specify the list of browser with a build Variable.

In build vNext, each variable you add to the build will be copied in an environment variable with the same name of the variable, converted in uppercase and with dot char substituted with underscore. If I use variable Selenium.BrowsersToTest the environment variable is called: SELENIUM_BROWSERSTOTEST

Here is the new code of the ValueSourceAttribute that use that environment variable to find list of browsers to use.

public class BrowserList : ValueSourceAttribute
{
    private static IEnumerable Browsers;

    public BrowserList()
        : base(typeof(BrowserList), "Browsers")
    {
        Browsers = GetBrowserFromConfig();
    }

    private static IEnumerable GetBrowserFromConfig()
    {
        var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
        if (!String.IsNullOrEmpty(envVar))
        {
            return envVar.Split('|', ',', ';', ':');
        }
        else
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject)JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }
}

The only change is that the attribute searches an envorinment variable called SELENIUM_BROWSERSTOTEST to grab list of browser. If the variable is not present, it use json configuration file as showed in previous article. Now we can choose browser list directly from build definition.

image

Figure 2: Specifying browser list with Build Variables.

Variables can be specified a Queue Time, this allow the user to change browserslist event when queueing a new build. Here is the result of a run

image

Figure 3: Outcome of the test using browsers list from build variable

Previous build outcome of Figure 3 is obtained running the test with an agent that does not run as a service, because it should be able to launch browser and access UI. If I queue the same build with Hosted Agent, or with an Agent that is running as a service, here is the result.

image

Figure 4: Failure running selenium tests

The problem with queued agent is that he has no firefox installed, but even with firefox installed, it would not be able to run the test because hosted agent runs as a service and has no access to UI. To solve this problem we can modify our ValueSourceAttribute

  public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            
        }

        static BrowserList()
        {
            Browsers = GetBrowserFromConfig();
            if (!Environment.UserInteractive)
            {
                Browsers = Browsers
                    .Where(b => b.ToLower() == "phantomjs")
                    .ToArray();
            }
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
            if (!String.IsNullOrEmpty(envVar))
            {
                return envVar.Split('|', ',', ';', ':');
            }
            else
            {
                var settings = File.ReadAllText("testParameters.json");
                var config = (JObject)JsonConvert.DeserializeObject(settings);
                var seleniumSettings = config["Selenium"];
                var browsers = (JArray)seleniumSettings["BrowsersToTest"];

                return browsers
                    .Select(b => b.ToString())
                    .ToArray();
            }
        }
    }

 

A couple of modification are worth of notice, the first one is that initialization of browser list is done in static constructor, and it will be executed once for each test run.

Then, if the test is not executing in a UserInteractive session, the attribute remove all browsers except phantomjs, the only ones that is guaranteed to run without having access to a UI. With this simple trick we avoid to run tests that will fail because they could not run. To understand which browser can run with agent running as a service you can simply try to run all of them and verify which ones returns error. Actually some browser can run in headless mode (without UI) so they can be used even if the agent has no access to the UI, so use this technique to remove all browsers that does not supports this mode.

To verify that everything works, I changed configuration of my Visual Studio Agent to run as a service instead that running interactively and queued a new build. Here is the result of the Tests.

image

Figure 5: Now only phantomjs test is run because agent is running as a service

Gian Maria.

Parametrize NUnit Selenium Test to run with different browsers

Parametrizing NUnit Tests is a new feature introduced with version 2.5 and this feature can be really useful in a variety of scenarios, but when it is time to use Selenium this is a killer feature.

I’m not going to cover Selenium WebDriver component, but basically it allows to write tests that can drive a Browser to execute test against your web application. In this scenario a killer feature is being able to specify the list of the browsers to use in a way that is completely indipendent from your test.

Thanks to the ValueSourceAttribute obtaining this result is really simple. First of all I create a base class for the test that creates different Selenium Web Drivers based on a string specified as argument.

 public class SeleniumTestFixtureBase
    {
        protected IWebDriver WebDriver;

        public void Initialize(String browser)
        {
            switch (browser.ToLower())
            {
                case "chrome":
                    DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
                    WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
                    break;
                case "firefox":
                    WebDriver = new FirefoxDriver();
                    break;
                default:
                    throw new NotSupportedException("Browser " + browser + " not supported");
            }
        }
    }

This is really trivial, and for this example I’m going to support only chrome and firefox. Then I create a json configuration file where I specify every parameter needed by Unit Tests.

Using an external json file to specify parameters for your Unit Test makes trivial passing those parameters for every runner (VS, TeamCity, GUI, command line runner).

This is a simple configuration file that contains only the list of the browsers I want to use. Remember to ask Visual Studio to copy this file in output folder when it changes.

{
  "Selenium": {
    "BrowsersToTest": [ "Firefox", "Chrome", "IE"]
  }
}

I’ve choosen Json because it is easy to write and easy to parse. Now it is time to create my attribute based on ValueSourceAttribute; it will read the above configuration file and provide the list of the browsers I want to use.

public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            Browsers = GetBrowserFromConfig();
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject) JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }

Again, this is a simple and trivial class that simply looks for a testParameters.json in the test run directory and search for the Selenium.BrowserToTest array, that containing the list of the browser to use as test.

Thanks to ValueSourceAttribute we can drive and parametrize the test with a simple external file.

Thanks to those two classes, I can simply specify for each test the Browsers List I want to use.

    [TestFixture]
    public class ParametrizedSeleniumTest : SeleniumTestFixtureBase
    {
        [Test]
        public void Test_Browsers([BrowserList] String browser)
        {
            base.Initialize(browser);
            WebDriver.Navigate().GoToUrl("http://www.microsoft.com");

            WebDriver.Close();
        }
    }

Using BrowserList attribute I’m asking Nunit to create an instance of that attribute to get the list value to be bind to that specific parameter. For each value a different test is created. Now if I build my project I can verify from VS Test Runner that indeed I have three different test to run.

image

 Figure 1: My test running with different versions of browsers

If I run the tests, as I’m expecting, the IE based test fails because I’ve not configured my test to use IE Driver.

image

Figure 2: Test output with Visual Studio Test Runner.

Now remove the IE from the list of Browsers to use in the testParameters.json and rebuild the solution. The IE version of the test now is disappeared from Test Runner.

image

Figure 3: Changing test configuration parameter and rebuilding will update test list

This technique does not depend on anyhthing, and can be used to run test with different configuration if you need (dev machine, different build machines, etc).

Thanks to this technique you can specify whitch browser to use with a simple configuration file and this is a killer feature if you are planning to run test during the build. You can simply modify your testParameters.json file before running the test in your Build to choose which browser to use, or disable completely selenium testing specifying an empty array.

Another option is keeping the list of browsers to use as a comma separated string stored in an Environment Variable, something like

SELENIUM_BROWSERS=Ie,chrome,firefox

With such a technique you can run the test and let the environment specify whitch browser are available to use for testing in that environment.

Gian Maria.