Running Unit Tests on different machine during TFS 2015 build

First of all I need to thanks my friend Jackob Ehn that pointed me to the right direction to create a particular build.  In this post I’ll share with you my journey to run tests on a different machine than the one that is running the build.

For some build it is interesting to have the ability to run some Unit Test (nunit in my scenario) on a machine different from that one that is running the build. There are a lot of legitimate reasons for doing this, for a project I’m working with, to run a set of test I need to have a huge amount of pre-requisites installed (LibreOffice, ghostscript, etc). Instead of installing those prerequisite on all agent machines, or install those one on a single build agent and using capabilities, I’d like to being able to run the build on any build agent, but run the test in a specific machine that had all the prerequisite installed.

Sometimes it is necessary to run tests during build on machine different from that one where the build agent is running.

The solution is quite simple, because VSTS / TFS already had all build tasks needed to solve my need and to execute tests on different machine.

The very first steps is copying all the dlls that contains tests on the target machine, this is accomplished by the Windows Machine File Copy task.

image 

Figure 1: File copy task in action

This is a really simple task, the only suggestion is to never specify the password in clear format, because everyone that can edit the build can read the password. In this situation the password is stored in the RmTestAdminPassword variable, and that variable is setup as secret.

image

Figure 2: Store sensitive information as secret variables of the build

Then we need to add a Visual Studio Test Agent Deployment task, to deploy Visual Studio Test Runner on target machine.

image

Figure 3: Visual Studio Test Agent Deployment

Configuration is straightforward, you need to specify a machine group or a list of target machines (point 1), then you should specify the user that will be used to run test agent (point 2), finally I’ve specified a custom location in my network for the Test Agent Installer. If you do not specify anything, the agent is downloaded from http://go.microsoft.com/fwlink/?LinkId=536423 but this will download approximately 130MB of data. For faster build it would be preferrable to download the agent and move the installer in a shared network folder to instruct the Task to grab the agent from that location.

Finally you use the Run Functional Tests task to actually execute tests in the target machine.

image

Figure 4: Running functional test from target machine configuration

You specify the machine(s) you want to use (point 1), then all the dll that contains tests (point 2) and you can also specify code coverage (point 3). Even if the task is called Run Functional Tests, it actually use Visual Studio Test runner to run tests, so you can run whatever test you like.

Thanks to TFS 2015 / VSTS build, we already have all tasks needed to run Unit Tests on target machines.

If you are running Nunit test or whatever test framework different from MsTest, this task will fail, because the target machine has no test adapter to run the test. The failure output tells you that the agent was not capable of finding any test to run in specified location. This happens even if you added Nuget Nunit adapter to the project. The solution is simple, first of all locates all needed dll in package location of your project.

image

Figure 5: Installing Nunit TestAdapter Nuget package, downloads all required dll to your machine

Once you located those four dll in your HD, you should copy them to the target machine in folder: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions, this folder was created by Visual Studio Test Agent deployment task and will contain all extension that will be automatically loaded by Visual Studio Test Agent. Once you copied the dll, those machine will be able to run nunit test without problem.

Once you copied all required dll in target folder, re-run the build, and verify that tests are indeed executed on the target machine.

SNAGHTML3c4ad0

Figure 6: Output of running tests on remote machine

Test output is transferred to the build machine, and attached to the build result as usual, so you do not need anything else to visualize test result in the same way as if the test were executed by agent machine.

image

Figure 7: Output of test is included in the build output like standard Unit Tests ran by the build agent

Once tasks are in place, everything is carried over by the test agent, test results are downloaded and attached to the build results, as for standard unit tests executed on Build Agent machine.

The only drawback of this approach is that it needs some times (in my system about 30 seconds) before the test started execution in target machine, but apart this problem, you can execute tests on remote machine with little effort.

Gian Maria.

Avoid using Shell command in PowerShell scipts

I have setup scripts that are used to install software, they are simply based on this paradigm

The build produces a zip file that contains everything needed to install the software, then we have a script that accepts the zip file as parameter as well as some other parameters and does install sofwtare on a local machine

This simple paradigm is perfect, because we can manually install a software launching powershell, or we can create a Chocolatey package to automate the installation. Clearly you can use the very same script to release the software inside TFS Release Management.

When PowerShell script runs in the RM Agent (or in a build agent) it has no access to the Shell. This is usually the default, because the agents does not run interactively and usually in Release Management, PowerShell scripts are executed remotely. This fact imply that you should not use anything related to shell in your script. Unfortunately, if you looks in the internet for code to unzip a zip file with PowerShell you can find code that uses shell.application object:

function Expand-WithShell(
    [string] $zipFile,
    [string] $destinationFolder,
    [bool] $deleteOld = $true,
    [bool] $quietMode = $false) 
{
    $shell = new-object -com shell.application

    if ((Test-Path $destinationFolder) -and $deleteOld)
    {
          Remove-Item $destinationFolder -Recurse -Force
    }

    New-Item $destinationFolder -ItemType directory

    $zip = $shell.NameSpace($zipFile)
    foreach($item in $zip.items())
    {
        if (!$quietMode) { Write-Host "unzipping " + $item.Name }
        $shell.Namespace($destinationFolder).copyhere($item)
    }
}

This is absolutely a problem if you run a script that uses this function in a build or RM agent. A real better alternative is a funcion that uses classes from Framework.

function Expand-WithFramework(
    [string] $zipFile,
    [string] $destinationFolder,
    [bool] $deleteOld = $true
)
{
    Add-Type -AssemblyName System.IO.Compression.FileSystem
    if ((Test-Path $destinationFolder) -and $deleteOld)
    {
          Remove-Item $destinationFolder -Recurse -Force
    }
    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $destinationFolder)
}

With this function you are not using anything related to Shell, and it can be run without problem even during a build or a release management. As you can see the function is simple and probably is a better solution even for interactive run of the script.

Another solution is using 7zip from command line, it gives you a better compression, it is free, but you need to install into any server where you want to run the script. This imply that probably zip files are still the simplest way to package your build artifacts for deployment.

Gian Maria.

No agent could be found with the following capabilities

In TFS 2015 / VSTS new build system each task contains a series of requirements that needs to be matched by agents capabilities for the task to run. Usually you install Visual Studio in the machine with the build agent and you can schedule standard .NET builds without problem, but what happens when the build starts to evolve?

When you start creating more complex build, you can find that your agent does not meets requirements because it miss some of the required capabilities. As an example, in TFS 2015 Build I’ve added task to run Sonar Qube Analysis on my code.

Build definition with Sonar Qube analysis enabled

Figure 1: A build with SonarQube analysis enabled

Now if I queue a build manually, TFS warned me that it is not able to find a suitable agent, and if you ignore that warning and queue the build here is the result.

The build failed because no agent with required capabilities was found in the pool.

Figure 2: Build failed because no suitable agent was find

The build failed because there are no agent capable to run it. Now you should go to the Project Collection administration page and verify all the agents.

Thanks to Capabilities tab in the administration page I can see a complete list of all agent capability.

figure 3: View agents capabilities in administration page

From that picture I verified that the only agent I’ve configured missed the Java capability, so I simply remote desktop to the server, installed java on the machine and then restart the agent service (VSO-Agemt-TFS2013Preview). After agent is restarted the build rans just fine.

Thanks to the new build system, TFS Build Agents can automatically determines some known capabilities (such as Java installed) and this is atuomatically matched with all the requirements that are contained in tasks you use in the build, so TFS can choose to run the build only in the agent that satisfies requirements, and if no agent is found it warns you with a clear error.

Gian Maria.

Uploading custom Build Task to TFS 2015

In a previous article I wrote on how to write a custom Task For Visual Studio Team Services, but a usual question is: can I use the same technique to write a task to TFS 2015 on-premise?

The answer is yes, and it is really simple, thanks to this fantastic article by Jesse, that explain how to use Fiddler to being able to authenticate to on-premise TFS without the hassle of enabling basic authentication. Thanks to that article and Fiddler, you can simply login from tfx-cli to your TFS 2015 without any problem.

image

Figure 1: Login against your local TFS Service

Once I’m logged in, I can simply use the very same command used for VSTS to upload the directory where I defined the Build Task on my local TFS on-premise Server.

image

Figure 2: Task was uploaded to the server

As you can see the task was uploaded to the server, exactly in the very same way I uploaded to my VSTS account. The task is now available to be used in my TFS.

image

Figure 3: Custom task in action in a TFS 2015 on-premise build

Thanks to the extensibility with PowerShell I did not need to care about versions of VSTS or TFS, because the script does not have reference on any dll or package and the same task can be used both in VSTS or in TFS 2015 without changing a single line.

Thanks to the new Buidl System, extending a build for both VSTS and TFS is now a simple and easy task.

Gian Maria.

Execute powershell code during a build in TFS 2015 Update 2

In TFS 2015 build system, running a PowerShell script stored in source code is simple because there is a dedicated action to accomplish this task, but if you want to run a PowerShell script that is not in source control, you have no option out of the Box.

Thanks to Update 2, you have now the ability to use the extensions you can find on Visual Studio Marketplace directly in your on-premise TFS installation. In the Upper Right corner of Web Ui you can find an icon that link to

This image shows the link in TFS UI to open the control panel to manage extension.

Figure 1: Link to manage extensions

This opens an administration page that allows you to manage extensions of your TFS installation. The first time you open this page, you will see no extension installed, so it is time to browse Visual Studio MarketPlace to find an extension. Actually I want to run inline PowerShell and I can immediately find that there is an extension that can solve my need.

As you can see from Figure 2, this extension have a couple of button, the first one, Install, is used to install this extension on one of your VSTS account, but if the extension is compatible with on-premise TFS you have also the option to Download the extension.

This image shows that extensions can be downloaded to local pc to be installed to TFS

Figure 2: Once you find an extension compatible with on-premise TFS you can download it.

The extension is a simple .vsix file, and from extension administration page you can upload to your TFS.

This image shows how you can upload downloaded extension to your TFS instance.

Figure 3: Once downloaded you can upload the extension to your TFS.

The process of uploading is really simple:

This image shows how you can choose the file to upload

Figure 4: Browse for downloaded extension

Tfs shows info for file to upload

Figure 5: Upload UI shows all the detail of the file

Now you can press Upload button, and the extension is uploaded to TFS and ready to be activated.

TFS shows the list of uploaded and installed extensions

Figure 6: Uploaded extension is now listed in the list of available extension

For each extension TFS ask you to install in specific project collection

  1. Figure 7: Install the extension on one or more collection

The concept behind “install then activate” is that, once uploaded the extension is available for the TFS Server instance, but then you should install in each Project Collection where you want to use that extension.

TFS asks you to choose the collection where you want to use the extension.

Figure 8: Choose the collection you want to install the extension into

Once you choose the Project Collection you need to confirm the operation

Figure 9: Confirm the selection

Once the Extension is installed in one Project Collection it can be used. In this example the extension contains a new build Task that allows you to execute an arbitrary PowerShell script during the build. To verify that everything is ok you can create a build and verify that the new Task is available and ready to use.

the extension is used in a build, where I was able to add the new task that executes arbitrary powershell code

Figure 10: The extension is ready and can be used

Thanks to TFS 2015 Update 2 you can Download and install extension for VSTS Marketplace to your on-premise TFS installation with few clicks.

Gian Maria.