Task versioning for TFS / VSTS Build

The build system of TFS / VSTS, introduced with TFS 2015, is evolving quite fast and release after release is always full of interesting new features. One of my favorites is the simple extention point to write a custom task to perform custom operation. The whole process is really simple and is described in this post.

One of the major problem when you manage a task is versioning, because you can have lots of builds using that task and if an update require to break the compatibility with the past, releasing a new version can really be a nightmare of many broken build.

Whenever you update a build task, there are many builds that are using it and if you introduce breaking changes, you risk to break all the builds that are using that task.

This is not a problem in VSTS / TFS, because a nice versionig control is in place; if you open a build you can find a flag near some build tasks.

image

Figure 1: Alerting on task that warn you about a new version of the task.

In Figure 1 you can find that a purple flag is telling you that you are using an old version of the task and that there are newer version available. This happens because a major version is available, but to preserve compatibility, existing build still uses the old version. 

If you look at task detail you can choose the version to use, as shown in Figure 2

SNAGHTMLf0c767

Figure 2: Choose version of task to use in the build

As you can see, you do not need to select an exact version, instead you can choose only the MAJOR version of the task, according to Semver. This logic follows semantic versioning, that states: whenever a breaking changes occurs, you need to change Major number. TFS / VSTS Build system assumes that a task, until a major version is not changes, is fully compatibile so it always uses the most recent version of that major.

When upgrading your custom task, update the major number if you are introducing breaking changes, this will prevent existing build to automatically use the new version.

This works not only with built-in tasks, but the same method works for any task, so, whenever you need to introduce some breaking changes in your custom build task, be sure to increment the major version. This will prevent existing build to use the new version, so you can upload the task in your account without problem. Once the new version is uploaded you can migrate each build without the risk of a big bang upgrade.

Also, if you find bug in an older version, you can still increment minor or patch number to fix task for all build that still uses old version. If you are at version 3.x.x but some builds still uses 2.x.x, if you need to fix a bug in that version you can simply patch version 2.x.x, assign number 2.x.x+1 (increment patch) and then upload the task.

Due to TFS /VSTS task versioning, I strongly suggests you to use GitFlow to develop your task, so it is easy for you to patch older version if needed.

If you include release notes in task.json you will also give the user information on what was changed.

image

Figure 3: Task.json file with release notes

Release notes are shown in the Build ui when you select specific version, and this can be used to give details on what is changed to explain why the major version is changed.

image

Figure 4: Release notes for the task are shown on the UI during build edit.

Version checking will really made management of build task a breeze.

Happy TFS / VSTS.

Gian Maria

Building with agent without Visual Studio installed

I had a build that runs fine on some agents, then I try running the build on a different agent but the build failed with the error.

Error MSB4019: The imported project “C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\WebApplications\Microsoft.WebApplication.targets” was not found

The problem originated by the fact that the build was configured to compile with VS2015 and use VS2015 test runner, but in build machine the only version of Visual Studio installed is VS2013.

For various reason I do not want to install VS 2015 on that build agent, so I tried to manually configure the agent to have my build and my unit tests running without the need of a full VS 2015 installation.

Warning: this technique worked for my build, but I cannot assure that it would work for your build.

Step 1: Install MSbuild 14 and targets

First of all I’ve installed Microsoft Build tools 2015 to have the very same version of MSBuild that VS2015 uses, but this not enough, because the build still complains that it was unable to find Microsoft.WebApplication.targets. The solution was copying the entire directory C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0 from my developer machine (where everything compiles perfectly) to the very same directory of my build server.

This solution usually works, because you are actually do a manual copy of all .targets that are needed by MsBuild to compile the solution (Asp.Net, etc etc). Now the source code compiles, but I’ve an error during test execution.

Step 2: Execute test with Visual Studio Test runner

Now the Test action failed with this error

System.IO.FileNotFoundException: Unable to determine the location of vstest.console.exe

Since I’ve not installed Visual Studio 2015 test runner is missing and tests could not execute.  To solve this problem I’ve installed Visual Studio 2015 agents, but the error is still there, even if I checked that the test runner was correctly installed. After some googling I’ve discovered that I need to modify a Registry Key called HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0 adding a simple string value named ShellFolder that points to the standard Visual Studio directory: C:\Program Files (x86)\Microsoft Visual Studio 14.0\

image

Figure 1: New registry key needed to the build agent to locate test runner.

After this last modification the solution builds and all test runs perfectly without VS 2015 installed on the build machine.

Please remember that this solution could not work for your environment.  The official and suggestged solution is installing to the build agents all versions of Visual Studio you need to bulid your code.

Gian Maria

Manage Environment Variables during a TFS / VSTS Build

To avoid creating unnecessary build definition, it is a best practice to allow for parameter overriding in every task that can be executed from a build. I’ve dealt on how to parametrize tests to use a different connection string when tests are executed during the build and I’ve used Environment variables for a lot of reasons.

Environment variables are not source controlled, this allows every developer to override settings in own machine without disturbing other developers. If I do not have a MongoDb in my machine I can simply choose to use some other instance in my network.

image

Figure 1: Overriding settings with environment variables.

Noone in the team is affected by this settings, and everyone has the freedom of changing this value to whatever he/she like. This is important because you can have different version of MongoDb installed in your network, with various different configuration (MMapV1 or Wired Tiger) and you want the freedom to choose the instance you want to use.

Another interesting aspect of Environment variables, is that they can be set during a VSTS/TFS build directly from build definition. This is possible because Variables defined for a build were set as environment variables when the build runs.

image

Figure 2: Specifing environment variables directly from Variables tab of a bulid

If you allow this value to be allowed at Queue Time, you can set the value when you manually queue a build.

image

Figure 3: Specifying variables value at Queue Time

If you look at Figure 3, you can verify that I’m able to change the value at queue time, but also, I can simply press “Add variable” to add any variable, even if it is not included in build definition. In this specific situation I can trigger a build and have my tests run against a specific MongoDb instance.

Remember that the value specified in the build definition overrides any value that is set on environment variables on build machine. This imply that, once you set a value in the Build definition, you are not able to change the value for a specific build agent.

It you want to be able to choose a different value for each build agent machine you can simply avoid setting the value on the Variables tab and instead define the variable on each build machine to have a different value for each agent. Another alternate approach is using two Environment variables, Es: TEST_MONGODB and TEST_MONGODB_OVERRIDE, and configure your tests to use TEST_MONGODB_OVERRIDE if present, if not present use TEST_MONGODB. This allows you to use TEST_MONGODB on build definition, but if you set TEST_MONGODB_OVERRIDE for a specific test agent, that agent will use that value.

Another interesting aspect of Environment Variable is that they are included in agent capabilities, as you can see from Figure 4.

SNAGHTMLd39383

Figure 4: All environment variables are part of agent Capabilities

This is an important aspect because if you want that variable to be set in the agent, you can avoid to include in Variables tab, and you can require this build to be run on an agent that has TEST_MONGODB environment variable specified.

image

Figure 5: Add a demand for a specific Environment Variable to be defined in Agent Machine

Setting the demands is not always necessary, in my example if the TEST_MONGODB variable is not definied, tests are executed against local MongDb instance. It is always a good strategy to use a reasonable default if some setting is not present in Environment Variables.

Gian Maria.

Create Parametrized test to allow for simpler Builds

When it is time of running unit test in a TFS or TeamCity Build, often you face the problem to run tests with options different from those one used in Developer Machine. As an example we have tons of tests that requires a MongoDb and and ElasticSearch or Solr integration.

While it is quite normal for developers to have everything installed in local dev box, it could be annoying to provide MongoDb and ElasticSearch installed on all agent machines. This approach complicates the setup of a build servers and create a situation that is less manageable.

While there is the ability to create a dedicated pool composed only by agent that have MongoDb and ElasticSearch installed, I prefer being able to run my test in all test agents, without any restriction.

The best solution is having parametrized tests, so you can execute tests with differnet parameters during the build, as an example you should parametrize connection strings.

Having parametrized tests greatly improve the ability to run test during build without creating complex requirement for agents.

In .NET environment it is quite common to use app.settings file to contain all connection strings, here it is an example taken from one of our project.

  
		
		
		
		
		
    

  

All Tests use ConfigurationManager object to access connectionstring from configuration file, and there is a single point where we specified all connection strings used by tests.

The obvious problem of storing setting in app.config is that this file is source controlled, and it is not possible to have different settings for different machine / developers.

A possible solution to this approach is using a powershell script that modifies all the configuration files in bin directories before running the test. Here is a naive approach with PowerShell.

param(
    [string] $baseMongoConnection = "mongodb://admin:xxxxxx##localhost/{0}",
    [string] $connectionQueryString = "?authSource=admin",
    [string] $configuration = "debug"
)

##Logging tests
$configFileName = "..\Logging\Jarvis.Framework.LoggingTests\bin\$configuration\Jarvis.Framework.LoggingTests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='testDb']/@connectionString" -value "$baseMongoConnection$connectionQueryString"

$xml.save($configFileName)

##main tests
$configFileName = "..\Jarvis.Framework.Tests\bin\$configuration\Jarvis.Framework.Tests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='eventstore']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-es-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='saga']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-saga-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='readmodel']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-readmodel-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='system']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-system-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='engine']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-engine-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='rebus']/@connectionString" -value ($baseMongoConnection -f "jarvis-rebus-test" + $connectionQueryString)

$xml.save($configFileName)

function Edit-XmlNodes {
param (
     $doc = $(throw "doc is a required parameter"),
    [string] $xpath = $(throw "xpath is a required parameter"),
    [string] $value = $(throw "value is a required parameter"),
    [bool] $condition = $true
)    
    if ($condition -eq $true) {
        $nodes = $doc.SelectNodes($xpath)
         
        foreach ($node in $nodes) {
            if ($node -ne $null) {
                if ($node.NodeType -eq "Element") {
                    $node.InnerXml = $value
                }
                else {
                    $node.Value = $value
                }
            }
        }
    }
}

Do not shoot the pianist :), this is a quick and dirty script that can edit configuration files, but this is not a good approach.

1) If a developer want to change this setting in its machine, it is really complex to instruct VS to run this script with parameter before running the test
2) It lead to unnecessary complicated builds, because you need to run this script before running the test and it introduces another point of failure.
3) It is not possibile to have agent dependant settings, I cannot specify that Agent X should run the test against mongo instance Y.

A better solution is to use Environment Variables to override app.config connection string, and create a Nunit SetupFixture that is executed before the first test.

NUnit has the ability to run a global setup that is run before the very first is run, and it is the perfect place where you can put logic to change configuration of the tests. In the following example the init script check some environment variables, then changes the connectionstring.


[SetUpFixture]
public class GlobalSetup
{
    [SetUp]
    public void ShowSomeTrace()
    {
        var overrideTestDb = Environment.GetEnvironmentVariable("TEST_MONGODB");
        if (String.IsNullOrEmpty(overrideTestDb)) return;

        var overrideTestDbQueryString = Environment.GetEnvironmentVariable("TEST_MONGODB_QUERYSTRING");
        var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        var connectionStringsSection = (ConnectionStringsSection)config.GetSection("connectionStrings");
        connectionStringsSection.ConnectionStrings["eventstore"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-es-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["saga"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-saga-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["readmodel"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-readmodel-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["system"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-system-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["engine"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-engine-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["rebus"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-rebus-test" + overrideTestDbQueryString;
        config.Save();
        ConfigurationManager.RefreshSection("connectionStrings");
    }
}

This approach has numerous advantages:

1) It works even for developers workstations, if you want to use a different connection string just populate corresponding Environment Variable and you are ready to go
2) You can simply define variables that are valid for all agents on the build definition, or setup environment variables different for each build agent, so each build agent will run tests against different Mongo instances/database.

This means that one of the best approach is parametrizing the tests with defaults that are good for developer machines, then allow override of configuration with Environment Variables to allow easy configuration for Build Agents.

Gian Maria

No agent could be found with the following capabilities

In TFS 2015 / VSTS new build system each task contains a series of requirements that needs to be matched by agents capabilities for the task to run. Usually you install Visual Studio in the machine with the build agent and you can schedule standard .NET builds without problem, but what happens when the build starts to evolve?

When you start creating more complex build, you can find that your agent does not meets requirements because it miss some of the required capabilities. As an example, in TFS 2015 Build I’ve added task to run Sonar Qube Analysis on my code.

Build definition with Sonar Qube analysis enabled

Figure 1: A build with SonarQube analysis enabled

Now if I queue a build manually, TFS warned me that it is not able to find a suitable agent, and if you ignore that warning and queue the build here is the result.

The build failed because no agent with required capabilities was found in the pool.

Figure 2: Build failed because no suitable agent was find

The build failed because there are no agent capable to run it. Now you should go to the Project Collection administration page and verify all the agents.

Thanks to Capabilities tab in the administration page I can see a complete list of all agent capability.

figure 3: View agents capabilities in administration page

From that picture I verified that the only agent I’ve configured missed the Java capability, so I simply remote desktop to the server, installed java on the machine and then restart the agent service (VSO-Agemt-TFS2013Preview). After agent is restarted the build rans just fine.

Thanks to the new build system, TFS Build Agents can automatically determines some known capabilities (such as Java installed) and this is atuomatically matched with all the requirements that are contained in tasks you use in the build, so TFS can choose to run the build only in the agent that satisfies requirements, and if no agent is found it warns you with a clear error.

Gian Maria.