Manage Environment Variables during a TFS / VSTS Build

To avoid creating unnecessary build definition, it is a best practice to allow for parameter overriding in every task that can be executed from a build. I’ve dealt on how to parametrize tests to use a different connection string when tests are executed during the build and I’ve used Environment variables for a lot of reasons.

Environment variables are not source controlled, this allows every developer to override settings in own machine without disturbing other developers. If I do not have a MongoDb in my machine I can simply choose to use some other instance in my network.

image

Figure 1: Overriding settings with environment variables.

Noone in the team is affected by this settings, and everyone has the freedom of changing this value to whatever he/she like. This is important because you can have different version of MongoDb installed in your network, with various different configuration (MMapV1 or Wired Tiger) and you want the freedom to choose the instance you want to use.

Another interesting aspect of Environment variables, is that they can be set during a VSTS/TFS build directly from build definition. This is possible because Variables defined for a build were set as environment variables when the build runs.

image

Figure 2: Specifing environment variables directly from Variables tab of a bulid

If you allow this value to be allowed at Queue Time, you can set the value when you manually queue a build.

image

Figure 3: Specifying variables value at Queue Time

If you look at Figure 3, you can verify that I’m able to change the value at queue time, but also, I can simply press “Add variable” to add any variable, even if it is not included in build definition. In this specific situation I can trigger a build and have my tests run against a specific MongoDb instance.

Remember that the value specified in the build definition overrides any value that is set on environment variables on build machine. This imply that, once you set a value in the Build definition, you are not able to change the value for a specific build agent.

It you want to be able to choose a different value for each build agent machine you can simply avoid setting the value on the Variables tab and instead define the variable on each build machine to have a different value for each agent. Another alternate approach is using two Environment variables, Es: TEST_MONGODB and TEST_MONGODB_OVERRIDE, and configure your tests to use TEST_MONGODB_OVERRIDE if present, if not present use TEST_MONGODB. This allows you to use TEST_MONGODB on build definition, but if you set TEST_MONGODB_OVERRIDE for a specific test agent, that agent will use that value.

Another interesting aspect of Environment Variable is that they are included in agent capabilities, as you can see from Figure 4.

SNAGHTMLd39383

Figure 4: All environment variables are part of agent Capabilities

This is an important aspect because if you want that variable to be set in the agent, you can avoid to include in Variables tab, and you can require this build to be run on an agent that has TEST_MONGODB environment variable specified.

image

Figure 5: Add a demand for a specific Environment Variable to be defined in Agent Machine

Setting the demands is not always necessary, in my example if the TEST_MONGODB variable is not definied, tests are executed against local MongDb instance. It is always a good strategy to use a reasonable default if some setting is not present in Environment Variables.

Gian Maria.

Create Parametrized test to allow for simpler Builds

When it is time of running unit test in a TFS or TeamCity Build, often you face the problem to run tests with options different from those one used in Developer Machine. As an example we have tons of tests that requires a MongoDb and and ElasticSearch or Solr integration.

While it is quite normal for developers to have everything installed in local dev box, it could be annoying to provide MongoDb and ElasticSearch installed on all agent machines. This approach complicates the setup of a build servers and create a situation that is less manageable.

While there is the ability to create a dedicated pool composed only by agent that have MongoDb and ElasticSearch installed, I prefer being able to run my test in all test agents, without any restriction.

The best solution is having parametrized tests, so you can execute tests with differnet parameters during the build, as an example you should parametrize connection strings.

Having parametrized tests greatly improve the ability to run test during build without creating complex requirement for agents.

In .NET environment it is quite common to use app.settings file to contain all connection strings, here it is an example taken from one of our project.

  
		
		
		
		
		
    

  

All Tests use ConfigurationManager object to access connectionstring from configuration file, and there is a single point where we specified all connection strings used by tests.

The obvious problem of storing setting in app.config is that this file is source controlled, and it is not possible to have different settings for different machine / developers.

A possible solution to this approach is using a powershell script that modifies all the configuration files in bin directories before running the test. Here is a naive approach with PowerShell.

param(
    [string] $baseMongoConnection = "mongodb://admin:xxxxxx##localhost/{0}",
    [string] $connectionQueryString = "?authSource=admin",
    [string] $configuration = "debug"
)

##Logging tests
$configFileName = "..\Logging\Jarvis.Framework.LoggingTests\bin\$configuration\Jarvis.Framework.LoggingTests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='testDb']/@connectionString" -value "$baseMongoConnection$connectionQueryString"

$xml.save($configFileName)

##main tests
$configFileName = "..\Jarvis.Framework.Tests\bin\$configuration\Jarvis.Framework.Tests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='eventstore']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-es-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='saga']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-saga-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='readmodel']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-readmodel-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='system']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-system-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='engine']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-engine-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='rebus']/@connectionString" -value ($baseMongoConnection -f "jarvis-rebus-test" + $connectionQueryString)

$xml.save($configFileName)

function Edit-XmlNodes {
param (
     $doc = $(throw "doc is a required parameter"),
    [string] $xpath = $(throw "xpath is a required parameter"),
    [string] $value = $(throw "value is a required parameter"),
    [bool] $condition = $true
)    
    if ($condition -eq $true) {
        $nodes = $doc.SelectNodes($xpath)
         
        foreach ($node in $nodes) {
            if ($node -ne $null) {
                if ($node.NodeType -eq "Element") {
                    $node.InnerXml = $value
                }
                else {
                    $node.Value = $value
                }
            }
        }
    }
}

Do not shoot the pianist :), this is a quick and dirty script that can edit configuration files, but this is not a good approach.

1) If a developer want to change this setting in its machine, it is really complex to instruct VS to run this script with parameter before running the test
2) It lead to unnecessary complicated builds, because you need to run this script before running the test and it introduces another point of failure.
3) It is not possibile to have agent dependant settings, I cannot specify that Agent X should run the test against mongo instance Y.

A better solution is to use Environment Variables to override app.config connection string, and create a Nunit SetupFixture that is executed before the first test.

NUnit has the ability to run a global setup that is run before the very first is run, and it is the perfect place where you can put logic to change configuration of the tests. In the following example the init script check some environment variables, then changes the connectionstring.


[SetUpFixture]
public class GlobalSetup
{
    [SetUp]
    public void ShowSomeTrace()
    {
        var overrideTestDb = Environment.GetEnvironmentVariable("TEST_MONGODB");
        if (String.IsNullOrEmpty(overrideTestDb)) return;

        var overrideTestDbQueryString = Environment.GetEnvironmentVariable("TEST_MONGODB_QUERYSTRING");
        var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        var connectionStringsSection = (ConnectionStringsSection)config.GetSection("connectionStrings");
        connectionStringsSection.ConnectionStrings["eventstore"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-es-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["saga"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-saga-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["readmodel"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-readmodel-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["system"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-system-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["engine"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-engine-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["rebus"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-rebus-test" + overrideTestDbQueryString;
        config.Save();
        ConfigurationManager.RefreshSection("connectionStrings");
    }
}

This approach has numerous advantages:

1) It works even for developers workstations, if you want to use a different connection string just populate corresponding Environment Variable and you are ready to go
2) You can simply define variables that are valid for all agents on the build definition, or setup environment variables different for each build agent, so each build agent will run tests against different Mongo instances/database.

This means that one of the best approach is parametrizing the tests with defaults that are good for developer machines, then allow override of configuration with Environment Variables to allow easy configuration for Build Agents.

Gian Maria

Run Selenium Test in build vNext

Previous Articles:

To run a Selenium test in a build vNext there are some modification to do apply to previous example. Let’s see how simple is running our Selenium Tests in a VSTS Build vNext.

The first modification requires adding a reference to PhantomJS, an Headless browser based on webkit that is capable of browsing a site and run javascript without a UI. Since we are interested in running test in a build server, this is a requirement if the agent runs as a service and has not access to a UI. To use PhantomJS just reference PhantomJS package on nunit and modify your base class to support this browser.

switch (browser.ToLower())
{
    case "chrome":
        DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
        WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
        break;
    case "firefox":
        WebDriver = new FirefoxDriver();
        break;
    case "phantomjs":
        WebDriver = new PhantomJSDriver();
        break;

Using Phantomjs is just a matter of creating a PhantomJSDriver with Selenium WebDriver test and the game is done. Now add “phantomjs” to test config file and you should be able to run the test.

image

Figure 1: Selenium test that uses Phantomjs headless browser to run test.

Now it is time to change the ValueSourceAttribute, to allow overriding the list of browser with an Environment Variable. While the Json Configuration file to configure tests is a really simple and useful solution for developers, running tests on their machines, when I need to run tests on a bulid server I want to be able to specify the list of browser with a build Variable.

In build vNext, each variable you add to the build will be copied in an environment variable with the same name of the variable, converted in uppercase and with dot char substituted with underscore. If I use variable Selenium.BrowsersToTest the environment variable is called: SELENIUM_BROWSERSTOTEST

Here is the new code of the ValueSourceAttribute that use that environment variable to find list of browsers to use.

public class BrowserList : ValueSourceAttribute
{
    private static IEnumerable Browsers;

    public BrowserList()
        : base(typeof(BrowserList), "Browsers")
    {
        Browsers = GetBrowserFromConfig();
    }

    private static IEnumerable GetBrowserFromConfig()
    {
        var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
        if (!String.IsNullOrEmpty(envVar))
        {
            return envVar.Split('|', ',', ';', ':');
        }
        else
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject)JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }
}

The only change is that the attribute searches an envorinment variable called SELENIUM_BROWSERSTOTEST to grab list of browser. If the variable is not present, it use json configuration file as showed in previous article. Now we can choose browser list directly from build definition.

image

Figure 2: Specifying browser list with Build Variables.

Variables can be specified a Queue Time, this allow the user to change browserslist event when queueing a new build. Here is the result of a run

image

Figure 3: Outcome of the test using browsers list from build variable

Previous build outcome of Figure 3 is obtained running the test with an agent that does not run as a service, because it should be able to launch browser and access UI. If I queue the same build with Hosted Agent, or with an Agent that is running as a service, here is the result.

image

Figure 4: Failure running selenium tests

The problem with queued agent is that he has no firefox installed, but even with firefox installed, it would not be able to run the test because hosted agent runs as a service and has no access to UI. To solve this problem we can modify our ValueSourceAttribute

  public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            
        }

        static BrowserList()
        {
            Browsers = GetBrowserFromConfig();
            if (!Environment.UserInteractive)
            {
                Browsers = Browsers
                    .Where(b => b.ToLower() == "phantomjs")
                    .ToArray();
            }
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
            if (!String.IsNullOrEmpty(envVar))
            {
                return envVar.Split('|', ',', ';', ':');
            }
            else
            {
                var settings = File.ReadAllText("testParameters.json");
                var config = (JObject)JsonConvert.DeserializeObject(settings);
                var seleniumSettings = config["Selenium"];
                var browsers = (JArray)seleniumSettings["BrowsersToTest"];

                return browsers
                    .Select(b => b.ToString())
                    .ToArray();
            }
        }
    }

 

A couple of modification are worth of notice, the first one is that initialization of browser list is done in static constructor, and it will be executed once for each test run.

Then, if the test is not executing in a UserInteractive session, the attribute remove all browsers except phantomjs, the only ones that is guaranteed to run without having access to a UI. With this simple trick we avoid to run tests that will fail because they could not run. To understand which browser can run with agent running as a service you can simply try to run all of them and verify which ones returns error. Actually some browser can run in headless mode (without UI) so they can be used even if the agent has no access to the UI, so use this technique to remove all browsers that does not supports this mode.

To verify that everything works, I changed configuration of my Visual Studio Agent to run as a service instead that running interactively and queued a new build. Here is the result of the Tests.

image

Figure 5: Now only phantomjs test is run because agent is running as a service

Gian Maria.

Parametrize NUnit Selenium Test to run with different browsers

Parametrizing NUnit Tests is a new feature introduced with version 2.5 and this feature can be really useful in a variety of scenarios, but when it is time to use Selenium this is a killer feature.

I’m not going to cover Selenium WebDriver component, but basically it allows to write tests that can drive a Browser to execute test against your web application. In this scenario a killer feature is being able to specify the list of the browsers to use in a way that is completely indipendent from your test.

Thanks to the ValueSourceAttribute obtaining this result is really simple. First of all I create a base class for the test that creates different Selenium Web Drivers based on a string specified as argument.

 public class SeleniumTestFixtureBase
    {
        protected IWebDriver WebDriver;

        public void Initialize(String browser)
        {
            switch (browser.ToLower())
            {
                case "chrome":
                    DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
                    WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
                    break;
                case "firefox":
                    WebDriver = new FirefoxDriver();
                    break;
                default:
                    throw new NotSupportedException("Browser " + browser + " not supported");
            }
        }
    }

This is really trivial, and for this example I’m going to support only chrome and firefox. Then I create a json configuration file where I specify every parameter needed by Unit Tests.

Using an external json file to specify parameters for your Unit Test makes trivial passing those parameters for every runner (VS, TeamCity, GUI, command line runner).

This is a simple configuration file that contains only the list of the browsers I want to use. Remember to ask Visual Studio to copy this file in output folder when it changes.

{
  "Selenium": {
    "BrowsersToTest": [ "Firefox", "Chrome", "IE"]
  }
}

I’ve choosen Json because it is easy to write and easy to parse. Now it is time to create my attribute based on ValueSourceAttribute; it will read the above configuration file and provide the list of the browsers I want to use.

public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            Browsers = GetBrowserFromConfig();
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject) JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }

Again, this is a simple and trivial class that simply looks for a testParameters.json in the test run directory and search for the Selenium.BrowserToTest array, that containing the list of the browser to use as test.

Thanks to ValueSourceAttribute we can drive and parametrize the test with a simple external file.

Thanks to those two classes, I can simply specify for each test the Browsers List I want to use.

    [TestFixture]
    public class ParametrizedSeleniumTest : SeleniumTestFixtureBase
    {
        [Test]
        public void Test_Browsers([BrowserList] String browser)
        {
            base.Initialize(browser);
            WebDriver.Navigate().GoToUrl("http://www.microsoft.com");

            WebDriver.Close();
        }
    }

Using BrowserList attribute I’m asking Nunit to create an instance of that attribute to get the list value to be bind to that specific parameter. For each value a different test is created. Now if I build my project I can verify from VS Test Runner that indeed I have three different test to run.

image

 Figure 1: My test running with different versions of browsers

If I run the tests, as I’m expecting, the IE based test fails because I’ve not configured my test to use IE Driver.

image

Figure 2: Test output with Visual Studio Test Runner.

Now remove the IE from the list of Browsers to use in the testParameters.json and rebuild the solution. The IE version of the test now is disappeared from Test Runner.

image

Figure 3: Changing test configuration parameter and rebuilding will update test list

This technique does not depend on anyhthing, and can be used to run test with different configuration if you need (dev machine, different build machines, etc).

Thanks to this technique you can specify whitch browser to use with a simple configuration file and this is a killer feature if you are planning to run test during the build. You can simply modify your testParameters.json file before running the test in your Build to choose which browser to use, or disable completely selenium testing specifying an empty array.

Another option is keeping the list of browsers to use as a comma separated string stored in an Environment Variable, something like

SELENIUM_BROWSERS=Ie,chrome,firefox

With such a technique you can run the test and let the environment specify whitch browser are available to use for testing in that environment.

Gian Maria.

Quickly run Unit Test With VS 2012/2013/..

With VS 2012 and newer versions we can run Unit Tests from various frameworks directly from Visual Studio IDE, thanks to the concept of Test Adapters. When you are doing Test Driven Development you usually go with Red/Green/Refactor workflow; what you need is a way to quickly run all or part of your unit tests after you modified the code. The quickest solution is using the option to Run Tests After Build but it is available only for Premium and Ultimate edition, but you can also run test with little manual intervention resorting to Keyboard Shortcut.

Simply go to TOOLS->Customize menu, then choose to customize keyboard.

image

Figure 1: Keyboard customization in Visual Studio

Once configuration window is opened, search for TestExplorer.RunAllTests command, place cursor in the “Press shortcut keys:” textbox and press a shortcut, then press “assign” button to assign to this command. In my standard configuration I like to have CTRL+SHIFT+ALT+A shortcut, because it is not assigned to any other command and it is easy to press with left hand.

image

Figure 2: Assign CTRL+SHIFT+ALT+A to TestExplorer.RunAllTests command

Now I can write code, use shortcut and VS will build solution and run test for me automatically, without leaving my hands from the keyboard. Thanks to the various test adapter and the various grouping and filtering possibility offered by Test Explorer, you can do TDD in Visual Studio without the need of third party tools.

Gian Maria.