Running NUnit Tests in a TFS 2015 Build vNext

Scenario

In TFS2015 CTP you can have a preview of the new build system that will be available with the next version of Team Foundation Server. If you start scheduling a new build that has NUnit test you probably will notice that your unit tests are not executed during the build.

To execute NUnit test in a vNext build you should ensure that appropriate tests adapters are available to agents that will run the build

The easiest way to make NUnit adapters available is downloading them from Visual Studio Gallery, unzip the .vsix extension file, then copy all the extension file in the folder.

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions

This operation should be done in every machine where you deployed Test Agents that will be used to run build with nunit tests. This can be annoying if you have lots of agents, if you prefer, Test Runner Task has an options to specify the path where the agent can find needed test runners.

The easiest way to automate all the process is adding Nunit Test Adapter nuget package to your test project, adding a standard Nuget Reference.

image

Figure 1: Add Nuget Package for NUnit Test Adapter

Once you’ve added the package, you should see all needed assemblies under the package directory of your solution. Now you can specify the location that contains the Nunit Test Adapter directly from Build Definition using this string

“$(Build.SourcesDirectory)\src\packages\NUnitTestAdapter.1.2\lib”

Please not the use of quotes (“) and the use of the $(Build.SourceDirectory) macro to specify the location where the build is taking place.

image

Figure 2: Specify path of custom Test Adapter inside build definition.

Pro and con of the two approaches:

Copying adapters inside Visual Studio TestWindows folder

Pro: You do not need to specify path of the adapters inside each build definition

Cons: You should do this for every machine where agent is deployed.

Specify Path to Custom Test Adapter with nunit packages

Pro: You can use the version you need referencing different nuget packages. You should not have access to machines where agent is installed.

Cons: You need to remember to use that settings inside each build where you want to run NUnit tests. All Test Adapters should be placed inside the same directory.

Gian Maria.

Unable to create workspace after TFS upgrade, a workspace already exists

At a customer site we performed an upgrade from TFS 2010 to TFS 2013, moving from a computer in Workspace to a computer in Domain. With this operation we finally defined a DNS name for tfs so all user can access it from tfs.company.local name, instead of using machine name. After we performed the upgrade, all the user were warned of this change and we simply told them to recreate workspaces pointing to the new server.

After the upgrade some of the users were not able to create a workspace in the same location of the old workspace, Visual Studio complains because a workspace already exists in the same location.

Using tf workspaces or tf workspace commands did not solve the problem, even if we delete all workspaces associated to the user, when he try to recreate the workspace he get an error because a workspace already existed at that location.

After a brief investigation we found the reason. TFS instasnce was migrated from a workspace to a domain and some of the user in domain have the same name they have in workspace machine. Upgrading a TFS with backup and restore preserves everything, migrated TFS instance still contains old workspaces. When one of the user try to create a new workspace in the same location on Disk, TFS complains because the old workspace is still there, but if we try to delete the workspace with the tf workspaces command it failed because it uses the new users, that is different, even if the name is identical.

Whenever you have this kind of problem, the easiest solution is to download and install TFS Sidekicks, connect to TFS with an account that has administrative privilege, and verify all the workspaces that are configured in the system.

image

Figure 1: Tfs Sidekicks in action, showing all workspaces defined in a Project Collection

Thanks to this tools, we verified that even if we deleted old workspaces with command line, they are still there because the user, despite having the same name, are different. Deleting old workspaces from Sideckicks resolved the problem and users were able to recreate all workspaces.

If you have problems with workspaces, sidekicks is a really useful tool because it gives you a nice GUI tool to list, check, delete and manage all workspaces defined in your TFS.

Gian Maria.

User added to Team Project have no permission after upgrade from TFS2010 to TFS2013

I’ve performed an upgrade from TFS2010 to TFS2013 at a customer site last week. The upgrade consisted in moving to a different machine and from a Workstation to an Active Directory Domain. The operation was simple, because the customer uses only Source Control and they want to spent minimal time in the operation, so we decided for this strategy

  1. Stop TFS in the old machine
  2. Backup and restore db in the new machine
  3. Upgrade and verify that everything works correctly

They do not care about user remapping, or reporting services or other stuff, they just want to do a quick  migration to new version to use local workspaces new feature (introduced with TFS 2012). The do not care to remap old user to new user, they only care not to spend too much time in the upgrade.

The upgrade went smoothly, but we start facing a couple of problem. The first one is: after the migration, each team project has no user, because the machine is now joined to a domain with different users, but if we add users to a team project, they are not able to connect to team project, and they seems to have no permission. All the users that are Project collection Administrators can use TFS with no problem.

The reason is simple, in TFS2012 the concept of Teams was introduced in the product. Each Team Project can have multiple Teams and when you add users from the home page of the Team, you are actually adding people to a TFS Group that correspond to that Team. For each Team Project a default Team with the same name of the Team Project is automatically created.

image

Figure 1: Users added to Team through home page.

In the above picture, I’ve added two user to the BuildExperiments Team, we can verify this in the Settings page of the Team Project.

image

Figure 2: User added through the home page, are added to the corresponding Tfs Security Group

To understand the permission of that users, you should use the administration section of TFS, as you can see from Figure 3, BuildExperiments team has no permission associated.

image

Figure 3: Permission associated to the Team Group

The reason for this is: the Team is not part of the Contributors TFS Group, it can be verified from the Member Of part of group properties

image

Figure 4: Team group belongs only to the Project Valid User

When you create a new Team Project, the default team (with the same name of the Team Project) is automatically added to the Contributors group, it is that team that gives user the right to access the Team Project. To fix the above problem you can manually add the Team Tfs Group to the Contributors group using the Join Group button. Once the Team group is added to the Contributors group, all the people you add with web interface are now able to access the Team Project.

This behavior is the standard in TFS, if you create a new Team, the Ui suggests you to choose to add the new Team Group to an existing group to inherit permission.

image

Team 5: Default option for a new group is to be part of the Contributors group.

This is an optional choice, you can choose a different security group or you can choose no group, but you should then remember to explicitly add permission to the corresponding Team Group.

When people does not access TFS and you believe that they should, always double check all the groups they belong and the effective permissions associated to them.

Gian Maria.

Error TF53001: The database operation was canceled by an administrator

A customer updated his TFS 2010 to 2013 in a new machine running Windows Server 2012 R2 and Sql Server 2014. Everything went fine, until after few days they started having an error whenever he tried to do a GetLatest or a Check-in or Check-out operation.

Error TF53001: The database operation was canceled by an administrator

Actually this error is not really informative, so I asked them to verify Event Viewer on the server (an operation you should always do whenever you have wrong behavior of your TFS). For each client operation that gave error they have this Event Error logged

Log Name:      Application
Source:        MSSQL$SQL2014TFS
Date:          19/02/2015 17:15:54
Event ID:      17310
Task Category: Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      xxxx.xxx.local
Description:
A user request from the session with SPID 70 generated a fatal exception. SQL Server is terminating this session. Contact Product Support Services with the dump produced in the log directory.
Event Xml:

This is an internal error of Sql Server, and we verified that SQL 2014 was in RTM, with no cumulative update installed. After installing latest Cumulative Update for SQL Server 2014 everything started working again. Since Cumulative Update usually address bugs in Sql Server product, it is always a good practice to keep your Sql Server up to date, and if you are experiencing strange Sql error, it could be the solution to your problems.

Gian Maria.

Monitor if your branch will generate merge conflicts with TFS Build

One of the greatest advantage in using Git over a centralized Version Control System is branching system. It is quite common for developers to start branching whenever they need to add a new feature, work on that branch and finally merge back to mainline when the feature is finished. One of the most famous workflow is called GitFlow, a way to work in Git that is implemented even in some GUI tool like SourceTree.

One of the main problem when you heavily use feature branches is the moment you merge back to mainline. One of the technique to easy the pain is periodically do a forward integration from mainline to feature branch, because it is usually easier doing several little merge than doing a big merge when the feature is complete.

Assuming that the mainline contains only stable code, because all feature branch merge to mainline only when they are stable, it is usually safe to periodically merge from mainline to branch to avoid a big merge at the end. In this scenario the question usually is: how frequently I have to merge? One possible solution is to take advantage of your Continuous Integration system to warn you when a modification you did on a branch introduces problem because:

  • Does not merge automatically with mainline but it generates conflicts
  • Even if the branch merge automatically the code wont compile.
  • Even if the branch merge automatically and the code compile, some tests where broken.

With TFS Build you can achieve this result quite simply, first of all configure a build with a trigger for each push, then in the Source Settings specify that this build should be triggered whenever any branch that starts with feat got any push.

image

Figure 1: Source code configuration, only build when a branch that starts with feat got a code increment

This build is intended to monitor the status of all feature branches. All you need to do is including a PowerShell script in source code that will be run pre-build. This script basically merge with master with command line git (Build Agent where that build runs should have git installed and available in PATH environment variable). This is a possible PowerShell script to accomplish the merge

$teststring = $env:TF_BUILD_SOURCEGETVERSION

$branchName = ""
$teststring | where { $_ -match "/(?[^/]*?):" } |
    foreach { $branchName = $Matches["bname"] }

Write-Host "BranchName = $branchName"
$rootDir = "$env:TF_BUILD_BUILDDIRECTORY\src"
Write-Host "Source folder is $rootDir"

#be sure to checkout the right branch 
$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "git"
$ps.StartInfo.WorkingDirectory = $rootDir
$ps.StartInfo.Arguments = "checkout $branchName"
$ps.StartInfo.RedirectStandardOutput = $True
$ps.StartInfo.RedirectStandardError = $True
$ps.StartInfo.UseShellExecute = $false
$ps.start()

[string] $Out = $ps.StandardOutput.ReadToEnd();
[string] $ErrOut = $ps.StandardError.ReadToEnd();
Write-Host "Output git checkout $branchName is:" $Out
Write-Host "Return value: " $ps.ExitCode
if ($ps.ExitCode -ne 0) 
{
    $Host.ui.WriteErrorLine("Branch cannot be checked out")
}

#try to merge


$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "git"
$ps.StartInfo.WorkingDirectory = $rootDir
$ps.StartInfo.Arguments = " merge master"
$ps.StartInfo.RedirectStandardOutput = $True
$ps.StartInfo.RedirectStandardError = $True
$ps.StartInfo.UseShellExecute = $false
$ps.start()

[string] $Out = $ps.StandardOutput.ReadToEnd();
[string] $ErrOut = $ps.StandardError.ReadToEnd();
Write-Host "Output git merge master:" $Out
Write-Host "Return value: " $ps.ExitCode

if ($ps.ExitCode -ne 0) 
{
    $Host.ui.WriteErrorLine("Branch does not merge correctly")
}

This is really a first tentative in this direction, there probably better way to do it, but it solves the problem without any great complexity. It just uses a regular expression to find branch name from the environment variable TF_BUILD_SOURCEGETVERSION. Once you have name of the branch just checkout it and then try to “git merge master” to merge the branch with master. Quite all git command return 0 if the operation is successful, so if the ExitCode of “git merge master” command is different from 0 it means that the merge operation did not complete correctly (conflicts occurred). If the merge is unsuccessful the script use UI.WriteErrorLine to write error on the UI and this has two effects: first the message will be reported on the Build Summary, then then build will be marked as partially failed.

Now even if your branch compile correctly and every test is green, after you push to TFS your Build Server will try to merge and run the build, if the branch does not automatically merge with master the build will partially fail and you got this:

image

Figure 2: Code does not automatically merge after the latest push.

This is a signal that is time to do a forward integration from mainline to your branch, because the amount of conflicts will be probably small. Another advantage is that you can merge the code while your modification is fresh in your mind. Nothing is more painfully than merge code you have written a month ago.

If the merge is successfully it does not mean that the code is in good health. Suppose you rename a method in your branch, but in the mainline another one has pushed an increment with another class that uses that method. This is the perfect example of code that gave no merge conflicts, but it does not even compile. Since the script actually does the merge before the build, the rest of the build works on the code result of the merge, now the build fails.

image

Figure 3: Merge was successfully, but the result of the merge does not compile

In this situation you surely have a standard build that simply works on the feature branch that is green, but the Build System can run for you this another special build that runs on the result of the merge with mainline that is Red. This means: In your branch code everything is green, but when you will merge to mainline you will have problem, so it is probably better to look at that problems now!. 

The very same happens if the result of the merge breaks some unit test. In conclusion, thanks to few PowerShell lines, you can instruct your TFS build to automatically try to merge each increment of code in your feature branches and then compile, run test and do whatever any operation you usually do on your build on the result of the merge between the feature branch and mainline. Thanks to this you can immediately merge from mainline to your branch to fix the problem as soon as you have introduced it.

This approach has some disadvantages. First of all a PowerShell script cannot make the build fails, so if the code does not merge, usually you will have a failing build because the build system try to build source code during a merge and find invalid characters in source code. Second, this special build runs only when there is a push on some feature branch, but nothing happens if someone pushes in the mainline. In that case you should run this special build for every feature branch you have in your repository.

Gian Maria.