Building with agent without Visual Studio installed

I had a build that runs fine on some agents, then I try running the build on a different agent but the build failed with the error.

Error MSB4019: The imported project “C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\WebApplications\Microsoft.WebApplication.targets” was not found

The problem originated by the fact that the build was configured to compile with VS2015 and use VS2015 test runner, but in build machine the only version of Visual Studio installed is VS2013.

For various reason I do not want to install VS 2015 on that build agent, so I tried to manually configure the agent to have my build and my unit tests running without the need of a full VS 2015 installation.

Warning: this technique worked for my build, but I cannot assure that it would work for your build.

Step 1: Install MSbuild 14 and targets

First of all I’ve installed Microsoft Build tools 2015 to have the very same version of MSBuild that VS2015 uses, but this not enough, because the build still complains that it was unable to find Microsoft.WebApplication.targets. The solution was copying the entire directory C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0 from my developer machine (where everything compiles perfectly) to the very same directory of my build server.

This solution usually works, because you are actually do a manual copy of all .targets that are needed by MsBuild to compile the solution (Asp.Net, etc etc). Now the source code compiles, but I’ve an error during test execution.

Step 2: Execute test with Visual Studio Test runner

Now the Test action failed with this error

System.IO.FileNotFoundException: Unable to determine the location of vstest.console.exe

Since I’ve not installed Visual Studio 2015 test runner is missing and tests could not execute.  To solve this problem I’ve installed Visual Studio 2015 agents, but the error is still there, even if I checked that the test runner was correctly installed. After some googling I’ve discovered that I need to modify a Registry Key called HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\14.0 adding a simple string value named ShellFolder that points to the standard Visual Studio directory: C:\Program Files (x86)\Microsoft Visual Studio 14.0\

image

Figure 1: New registry key needed to the build agent to locate test runner.

After this last modification the solution builds and all test runs perfectly without VS 2015 installed on the build machine.

Please remember that this solution could not work for your environment.  The official and suggestged solution is installing to the build agents all versions of Visual Studio you need to bulid your code.

Gian Maria

Manage Environment Variables during a TFS / VSTS Build

To avoid creating unnecessary build definition, it is a best practice to allow for parameter overriding in every task that can be executed from a build. I’ve dealt on how to parametrize tests to use a different connection string when tests are executed during the build and I’ve used Environment variables for a lot of reasons.

Environment variables are not source controlled, this allows every developer to override settings in own machine without disturbing other developers. If I do not have a MongoDb in my machine I can simply choose to use some other instance in my network.

image

Figure 1: Overriding settings with environment variables.

Noone in the team is affected by this settings, and everyone has the freedom of changing this value to whatever he/she like. This is important because you can have different version of MongoDb installed in your network, with various different configuration (MMapV1 or Wired Tiger) and you want the freedom to choose the instance you want to use.

Another interesting aspect of Environment variables, is that they can be set during a VSTS/TFS build directly from build definition. This is possible because Variables defined for a build were set as environment variables when the build runs.

image

Figure 2: Specifing environment variables directly from Variables tab of a bulid

If you allow this value to be allowed at Queue Time, you can set the value when you manually queue a build.

image

Figure 3: Specifying variables value at Queue Time

If you look at Figure 3, you can verify that I’m able to change the value at queue time, but also, I can simply press “Add variable” to add any variable, even if it is not included in build definition. In this specific situation I can trigger a build and have my tests run against a specific MongoDb instance.

Remember that the value specified in the build definition overrides any value that is set on environment variables on build machine. This imply that, once you set a value in the Build definition, you are not able to change the value for a specific build agent.

It you want to be able to choose a different value for each build agent machine you can simply avoid setting the value on the Variables tab and instead define the variable on each build machine to have a different value for each agent. Another alternate approach is using two Environment variables, Es: TEST_MONGODB and TEST_MONGODB_OVERRIDE, and configure your tests to use TEST_MONGODB_OVERRIDE if present, if not present use TEST_MONGODB. This allows you to use TEST_MONGODB on build definition, but if you set TEST_MONGODB_OVERRIDE for a specific test agent, that agent will use that value.

Another interesting aspect of Environment Variable is that they are included in agent capabilities, as you can see from Figure 4.

SNAGHTMLd39383

Figure 4: All environment variables are part of agent Capabilities

This is an important aspect because if you want that variable to be set in the agent, you can avoid to include in Variables tab, and you can require this build to be run on an agent that has TEST_MONGODB environment variable specified.

image

Figure 5: Add a demand for a specific Environment Variable to be defined in Agent Machine

Setting the demands is not always necessary, in my example if the TEST_MONGODB variable is not definied, tests are executed against local MongDb instance. It is always a good strategy to use a reasonable default if some setting is not present in Environment Variables.

Gian Maria.

Create Parametrized test to allow for simpler Builds

When it is time of running unit test in a TFS or TeamCity Build, often you face the problem to run tests with options different from those one used in Developer Machine. As an example we have tons of tests that requires a MongoDb and and ElasticSearch or Solr integration.

While it is quite normal for developers to have everything installed in local dev box, it could be annoying to provide MongoDb and ElasticSearch installed on all agent machines. This approach complicates the setup of a build servers and create a situation that is less manageable.

While there is the ability to create a dedicated pool composed only by agent that have MongoDb and ElasticSearch installed, I prefer being able to run my test in all test agents, without any restriction.

The best solution is having parametrized tests, so you can execute tests with differnet parameters during the build, as an example you should parametrize connection strings.

Having parametrized tests greatly improve the ability to run test during build without creating complex requirement for agents.

In .NET environment it is quite common to use app.settings file to contain all connection strings, here it is an example taken from one of our project.

  
		
		
		
		
		
    

  

All Tests use ConfigurationManager object to access connectionstring from configuration file, and there is a single point where we specified all connection strings used by tests.

The obvious problem of storing setting in app.config is that this file is source controlled, and it is not possible to have different settings for different machine / developers.

A possible solution to this approach is using a powershell script that modifies all the configuration files in bin directories before running the test. Here is a naive approach with PowerShell.

param(
    [string] $baseMongoConnection = "mongodb://admin:xxxxxx##localhost/{0}",
    [string] $connectionQueryString = "?authSource=admin",
    [string] $configuration = "debug"
)

##Logging tests
$configFileName = "..\Logging\Jarvis.Framework.LoggingTests\bin\$configuration\Jarvis.Framework.LoggingTests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='testDb']/@connectionString" -value "$baseMongoConnection$connectionQueryString"

$xml.save($configFileName)

##main tests
$configFileName = "..\Jarvis.Framework.Tests\bin\$configuration\Jarvis.Framework.Tests.dll.config"
Write-Output "Config File Name Is: $configFileName"

$xml = (Get-Content $configFileName)
 
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='eventstore']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-es-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='saga']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-saga-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='readmodel']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-readmodel-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='system']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-system-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='engine']/@connectionString" -value ($baseMongoConnection -f "jarvis-framework-engine-test" + $connectionQueryString)
Edit-XmlNodes $xml -xpath "/configuration/connectionStrings/add[@name='rebus']/@connectionString" -value ($baseMongoConnection -f "jarvis-rebus-test" + $connectionQueryString)

$xml.save($configFileName)

function Edit-XmlNodes {
param (
     $doc = $(throw "doc is a required parameter"),
    [string] $xpath = $(throw "xpath is a required parameter"),
    [string] $value = $(throw "value is a required parameter"),
    [bool] $condition = $true
)    
    if ($condition -eq $true) {
        $nodes = $doc.SelectNodes($xpath)
         
        foreach ($node in $nodes) {
            if ($node -ne $null) {
                if ($node.NodeType -eq "Element") {
                    $node.InnerXml = $value
                }
                else {
                    $node.Value = $value
                }
            }
        }
    }
}

Do not shoot the pianist :), this is a quick and dirty script that can edit configuration files, but this is not a good approach.

1) If a developer want to change this setting in its machine, it is really complex to instruct VS to run this script with parameter before running the test
2) It lead to unnecessary complicated builds, because you need to run this script before running the test and it introduces another point of failure.
3) It is not possibile to have agent dependant settings, I cannot specify that Agent X should run the test against mongo instance Y.

A better solution is to use Environment Variables to override app.config connection string, and create a Nunit SetupFixture that is executed before the first test.

NUnit has the ability to run a global setup that is run before the very first is run, and it is the perfect place where you can put logic to change configuration of the tests. In the following example the init script check some environment variables, then changes the connectionstring.


[SetUpFixture]
public class GlobalSetup
{
    [SetUp]
    public void ShowSomeTrace()
    {
        var overrideTestDb = Environment.GetEnvironmentVariable("TEST_MONGODB");
        if (String.IsNullOrEmpty(overrideTestDb)) return;

        var overrideTestDbQueryString = Environment.GetEnvironmentVariable("TEST_MONGODB_QUERYSTRING");
        var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        var connectionStringsSection = (ConnectionStringsSection)config.GetSection("connectionStrings");
        connectionStringsSection.ConnectionStrings["eventstore"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-es-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["saga"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-saga-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["readmodel"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-readmodel-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["system"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-system-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["engine"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-framework-engine-test" + overrideTestDbQueryString;
        connectionStringsSection.ConnectionStrings["rebus"].ConnectionString = overrideTestDb.TrimEnd('/') + "/jarvis-rebus-test" + overrideTestDbQueryString;
        config.Save();
        ConfigurationManager.RefreshSection("connectionStrings");
    }
}

This approach has numerous advantages:

1) It works even for developers workstations, if you want to use a different connection string just populate corresponding Environment Variable and you are ready to go
2) You can simply define variables that are valid for all agents on the build definition, or setup environment variables different for each build agent, so each build agent will run tests against different Mongo instances/database.

This means that one of the best approach is parametrizing the tests with defaults that are good for developer machines, then allow override of configuration with Environment Variables to allow easy configuration for Build Agents.

Gian Maria

No agent could be found with the following capabilities

In TFS 2015 / VSTS new build system each task contains a series of requirements that needs to be matched by agents capabilities for the task to run. Usually you install Visual Studio in the machine with the build agent and you can schedule standard .NET builds without problem, but what happens when the build starts to evolve?

When you start creating more complex build, you can find that your agent does not meets requirements because it miss some of the required capabilities. As an example, in TFS 2015 Build I’ve added task to run Sonar Qube Analysis on my code.

Build definition with Sonar Qube analysis enabled

Figure 1: A build with SonarQube analysis enabled

Now if I queue a build manually, TFS warned me that it is not able to find a suitable agent, and if you ignore that warning and queue the build here is the result.

The build failed because no agent with required capabilities was found in the pool.

Figure 2: Build failed because no suitable agent was find

The build failed because there are no agent capable to run it. Now you should go to the Project Collection administration page and verify all the agents.

Thanks to Capabilities tab in the administration page I can see a complete list of all agent capability.

figure 3: View agents capabilities in administration page

From that picture I verified that the only agent I’ve configured missed the Java capability, so I simply remote desktop to the server, installed java on the machine and then restart the agent service (VSO-Agemt-TFS2013Preview). After agent is restarted the build rans just fine.

Thanks to the new build system, TFS Build Agents can automatically determines some known capabilities (such as Java installed) and this is atuomatically matched with all the requirements that are contained in tasks you use in the build, so TFS can choose to run the build only in the agent that satisfies requirements, and if no agent is found it warns you with a clear error.

Gian Maria.

Uploading custom Build Task to TFS 2015

In a previous article I wrote on how to write a custom Task For Visual Studio Team Services, but a usual question is: can I use the same technique to write a task to TFS 2015 on-premise?

The answer is yes, and it is really simple, thanks to this fantastic article by Jesse, that explain how to use Fiddler to being able to authenticate to on-premise TFS without the hassle of enabling basic authentication. Thanks to that article and Fiddler, you can simply login from tfx-cli to your TFS 2015 without any problem.

image

Figure 1: Login against your local TFS Service

Once I’m logged in, I can simply use the very same command used for VSTS to upload the directory where I defined the Build Task on my local TFS on-premise Server.

image

Figure 2: Task was uploaded to the server

As you can see the task was uploaded to the server, exactly in the very same way I uploaded to my VSTS account. The task is now available to be used in my TFS.

image

Figure 3: Custom task in action in a TFS 2015 on-premise build

Thanks to the extensibility with PowerShell I did not need to care about versions of VSTS or TFS, because the script does not have reference on any dll or package and the same task can be used both in VSTS or in TFS 2015 without changing a single line.

Thanks to the new Buidl System, extending a build for both VSTS and TFS is now a simple and easy task.

Gian Maria.