Monitor if your branch will generate merge conflicts with TFS Build

One of the greatest advantage in using Git over a centralized Version Control System is branching system. It is quite common for developers to start branching whenever they need to add a new feature, work on that branch and finally merge back to mainline when the feature is finished. One of the most famous workflow is called GitFlow, a way to work in Git that is implemented even in some GUI tool like SourceTree.

One of the main problem when you heavily use feature branches is the moment you merge back to mainline. One of the technique to easy the pain is periodically do a forward integration from mainline to feature branch, because it is usually easier doing several little merge than doing a big merge when the feature is complete.

Assuming that the mainline contains only stable code, because all feature branch merge to mainline only when they are stable, it is usually safe to periodically merge from mainline to branch to avoid a big merge at the end. In this scenario the question usually is: how frequently I have to merge? One possible solution is to take advantage of your Continuous Integration system to warn you when a modification you did on a branch introduces problem because:

  • Does not merge automatically with mainline but it generates conflicts
  • Even if the branch merge automatically the code wont compile.
  • Even if the branch merge automatically and the code compile, some tests where broken.

With TFS Build you can achieve this result quite simply, first of all configure a build with a trigger for each push, then in the Source Settings specify that this build should be triggered whenever any branch that starts with feat got any push.

image

Figure 1: Source code configuration, only build when a branch that starts with feat got a code increment

This build is intended to monitor the status of all feature branches. All you need to do is including a PowerShell script in source code that will be run pre-build. This script basically merge with master with command line git (Build Agent where that build runs should have git installed and available in PATH environment variable). This is a possible PowerShell script to accomplish the merge

$teststring = $env:TF_BUILD_SOURCEGETVERSION

$branchName = ""
$teststring | where { $_ -match "/(?[^/]*?):" } |
    foreach { $branchName = $Matches["bname"] }

Write-Host "BranchName = $branchName"
$rootDir = "$env:TF_BUILD_BUILDDIRECTORY\src"
Write-Host "Source folder is $rootDir"

#be sure to checkout the right branch 
$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "git"
$ps.StartInfo.WorkingDirectory = $rootDir
$ps.StartInfo.Arguments = "checkout $branchName"
$ps.StartInfo.RedirectStandardOutput = $True
$ps.StartInfo.RedirectStandardError = $True
$ps.StartInfo.UseShellExecute = $false
$ps.start()

[string] $Out = $ps.StandardOutput.ReadToEnd();
[string] $ErrOut = $ps.StandardError.ReadToEnd();
Write-Host "Output git checkout $branchName is:" $Out
Write-Host "Return value: " $ps.ExitCode
if ($ps.ExitCode -ne 0) 
{
    $Host.ui.WriteErrorLine("Branch cannot be checked out")
}

#try to merge


$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "git"
$ps.StartInfo.WorkingDirectory = $rootDir
$ps.StartInfo.Arguments = " merge master"
$ps.StartInfo.RedirectStandardOutput = $True
$ps.StartInfo.RedirectStandardError = $True
$ps.StartInfo.UseShellExecute = $false
$ps.start()

[string] $Out = $ps.StandardOutput.ReadToEnd();
[string] $ErrOut = $ps.StandardError.ReadToEnd();
Write-Host "Output git merge master:" $Out
Write-Host "Return value: " $ps.ExitCode

if ($ps.ExitCode -ne 0) 
{
    $Host.ui.WriteErrorLine("Branch does not merge correctly")
}

This is really a first tentative in this direction, there probably better way to do it, but it solves the problem without any great complexity. It just uses a regular expression to find branch name from the environment variable TF_BUILD_SOURCEGETVERSION. Once you have name of the branch just checkout it and then try to “git merge master” to merge the branch with master. Quite all git command return 0 if the operation is successful, so if the ExitCode of “git merge master” command is different from 0 it means that the merge operation did not complete correctly (conflicts occurred). If the merge is unsuccessful the script use UI.WriteErrorLine to write error on the UI and this has two effects: first the message will be reported on the Build Summary, then then build will be marked as partially failed.

Now even if your branch compile correctly and every test is green, after you push to TFS your Build Server will try to merge and run the build, if the branch does not automatically merge with master the build will partially fail and you got this:

image

Figure 2: Code does not automatically merge after the latest push.

This is a signal that is time to do a forward integration from mainline to your branch, because the amount of conflicts will be probably small. Another advantage is that you can merge the code while your modification is fresh in your mind. Nothing is more painfully than merge code you have written a month ago.

If the merge is successfully it does not mean that the code is in good health. Suppose you rename a method in your branch, but in the mainline another one has pushed an increment with another class that uses that method. This is the perfect example of code that gave no merge conflicts, but it does not even compile. Since the script actually does the merge before the build, the rest of the build works on the code result of the merge, now the build fails.

image

Figure 3: Merge was successfully, but the result of the merge does not compile

In this situation you surely have a standard build that simply works on the feature branch that is green, but the Build System can run for you this another special build that runs on the result of the merge with mainline that is Red. This means: In your branch code everything is green, but when you will merge to mainline you will have problem, so it is probably better to look at that problems now!. 

The very same happens if the result of the merge breaks some unit test. In conclusion, thanks to few PowerShell lines, you can instruct your TFS build to automatically try to merge each increment of code in your feature branches and then compile, run test and do whatever any operation you usually do on your build on the result of the merge between the feature branch and mainline. Thanks to this you can immediately merge from mainline to your branch to fix the problem as soon as you have introduced it.

This approach has some disadvantages. First of all a PowerShell script cannot make the build fails, so if the code does not merge, usually you will have a failing build because the build system try to build source code during a merge and find invalid characters in source code. Second, this special build runs only when there is a push on some feature branch, but nothing happens if someone pushes in the mainline. In that case you should run this special build for every feature branch you have in your repository.

Gian Maria.

Git submodule update error: Permission Denied Publickey

It could happens when you clone a Git Repository with submodules, issuing a git submodule update command, you are prompted with this error error.

Cloning into ‘src/xxxx’…
Warning: Permanently added the RSA host key for IP address xxx.xxx.xxx.xxx to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

If you search in the internet for the cause of errors, you can find some people suggesting that the url specified in .gitmodules file is wrong and should be changed, here is my .gitmodule

[submodule “src/CQRS”]
     path = src/CQRS
     url = git@github.com:xxxxxx/cqrs.git
     branch = master
    

You could change the url configuration to https url and everything works, but this is not the perfect solution, because the address git@github.com is perfectly valid, but probably there is some problem with your RSA keys stored in Github (or you never configured RSA Keys for your account). In my situation, my RSA Keys had some problem and I needed to recreate another one. If you do not know what a RSA key is and how to create a RSA Key to connect to github I strongly suggest you reading the guide: Generating SSH Keys.

Once you configure a valid certificate in github your submodule should word without problem.

Gian Maria.

Git showing file as modified even if it is unchanged

This is one annoying problem that happens sometimes to git users: the symptom is: git status command shows you some files as modified (you are sure that you had not modified that files), you revert all changes with a git checkout — . but the files stills are in modified state if you issue another git status.

This is a real annoying problem, suppose you want to switch branch with git checkout branchname, you will find that git does not allow you to switch because of uncommitted changes.

This problem is likely caused by the end-of-line normalization (I strongly suggest you to read all the details in Pro Git book or read the help of github). I do not want to enter into details of this feature, but I only want to help people to diagnose and avoid this kind of problem.

To understand if you really have a Line Ending Issue you should run git diff -w command to verify what is really changed in files that git as modified with git status command. The -w options tells git to ignore whitespace and line endings, if this command shows no differences, you are probably victim of problem in Line Ending Normalization.

This is especially true if you are working with git svn, connecting to a subversion repository where developers did not pay attention to line endings and it happens usually when you have files with mixed CRLF / CR / LF. If you work in mixed environment (Unix/Linux, Windows, Macintosh) it is better to find files that are listed as modified and manually (or with some tool) normalize Line Endings. If you do not work in mixed environment you can simply turn off eol normalization for the single repository where you experience the problem. To do this you can issue a git config –local core.autocrlf false  but it works only for you and not for all the other developers that works to the project. Moreover some people reports that they still have problem even with core.autocrlf to false.

Remember that git supports .gitattributes files, used to change settings for a single subdirectory. If you set core.autocrlf to false and still have line ending normalization problem, please search for .gitattribuges files in every subdirectory of your repository, and verify if it has a line where autocrlf is turned on:

* text=auto

now you can turn off in all .gitattributes files you find in your repository

* text=off

To be sure that every developer of the team works with autocrlf turned off, you should place a .gitattributes file in repository root with autocrlf turned off. Remember that it is a better option to normalize files and leave autocrlf turned on, but if you are working with legacy code imported from another VCS, or you work with git svn, git-tf or similar tools, probably it is better turn autocrlf to off if you start experiencing that kind of problems.

Gian Maria.

Moving between different VCS to Team Foundation Server

In an old post I’ve already demonstrated how you can move a project from a standard TFS Version Control System to a Git repository hosted on TF Service (or wherever else). This is especially useful if you are working with TFS but you really need features of a Distributed Version Control System. The original post is here: move source from a TFVCS based Team Project to a TF Service Git Based Team Project.

Now that you have support for Git even on on-premise installation of TFS 2013, you can use the very same procedure to move source control from a team project based on TFVC to another one based on Git. But there is more: thanks to support of git-tf, a java based bridge between Git and TFVC you can support even more scenarios. You can use Git-Svn to convert from a Subversion Repository and move code to a Team Project based on Git or TFVC. With a little setup you can also handle external companies that works with Subversion and you can import their work in your TFS maintaining even the connection between Subversion commit and Work Items. I’ve described this process in a post called Git as a bridge between Subversion and TFS. You can use Hg-git to convert from Mercurial to Git or TFVC, and in general you can move code to your TFS from any VCS that supports conversion to Git.

The primary drawbacks of this approach are:

Time compression: if you move to a TFVC based Team Project you are not able to maintain the original timestamp of commit / check-in, and the timestamp will reflect the moment you checking from git to TFVC with git-tf.

User Mapping: checkin to the system done with git-tf are done with the credential of the person that is doing the check-in.

Since Git is a real good tool, I strongly suggest to move code to a Git based Team Project, so you can maintain full history. Then if you really need to work with TFVC and you cannot use Git, you can move only the tip to a TFCV based team project, and the history will be maintained with full fidelity in the Git Repository.

Gian Maria.

Git as a bridge between Subversion and TFS

This is the scenario: Nablasoft company uses TFS with standard TFVC (TFS Version Control System) and needs to assign some of the work to some external Company (lets call it Acme), as an example developing for Android or iOs. The simplest solution is giving Acme developers access to TFS server, but then Nablasoft should pay the CAL for Acme developers, moreover, Acme uses subversion and they do not want to use another VCS unless it is strictly necessary.

Since part of the contract requires Acme to give Nablasoft full code history, using Git-svn and Git-Tf can easily solves this scenario.

Setting everything up

 

First Step: Nablasoft managers can create another team inside their Team Project to create a backlog that contains all the work that will be given to Acme Company. This will permits to Manager to schedule User Stories for Acme and monitor the progress. Only Nablasoft managers should access this team, no people from Acme company will ever use it.

After the backlog is created, you can simply email it to Acme developers with Email, or import in excel and send the excel.

image

Figure 1: Send a mail to Acme company with all details on the feature we want them to work on

This can be useful if Acme people can access the TFS of Nablasoft, but if we assume that Acme people has not access to Nablasoft Tfs, an addin of word like Word To Tfs can be useful to load all definition of your specification into a single Word file that can be sent to Acme. The overall goal is being able to send specification to acme as well as the original Id of the Work Item in the original Nablasoft Tfs.

Acme developers will develop on their own Subversion server, they should give to Nablasoft manager access to this Svn repository and should include in every subversion commit a comment containing an hastag followed by the id of the Feature related to that commit, Es #884

Nablasoft Manager will create a git clone of Acme Subversion repository using standard Git-Svn features of msysgit distribution with a simple command like.

git svn clone https://acmedev.googlecode.com/svn/trunk c:\temp\AcmeCode

This creates a full clone of subversion repository, then Nablasoft’s manager can simply download all modification with a simple git svn rebase, but the interesting part is that he can simply connect the local git repository to standard TFS repository with the command

git-tf configure https://gianmariaricci.visualstudio.com /DefaultCollection $/NablasoftCompany/Main/ReactiveSite

You can specify any path in TFVC source control, in previous command line I’ve decided to map my local git repository to the $/NablasoftCompany/Main/ReactiveSite but you can choose whatever folder you want. Once you have connected the same Git repository to remote Svn server and to your TFVC based repository you are able to grab modification from subversion and move them to TFS. With a

Git svn rebase

You download all new modification from SVN repository, and when you want to move everything into TFVC repository you can issue a

git-tf checkin –deep

and all modifications will be transferred to TFS. The nice stuff is that you can use a bunch of API call to automatically associate check-ins to Work Items based on comment content.

Example

Suppose Acme did a copule of commit on SVN Repository

image

Fifure 2: Log of subversion repository showing the two commits with code related to a couple of features.

Now if you issue a git svn rebase you can download those two commits into local git repository. If you view log of Git repository you realize that all subversion checkins are cloned in local repository.

image

Figure 3: Commits cloned from subversion repository to a local Git repository

Now a git-tf checkin with –deep option will transfer those commits to TFVC. The first commit (the one with comment initial directory structure) is the first one used to create the trunk directory, so it will not cause any check-in in the final repository, but the other two were transferred.

image

Figure 4: Check-ins created in TFVC from local Git repository thanks to git-tf tool.

The last part is running the little utility to associate Check-Ins and Work Items with hashtag in the comment. Thanks to this workflow, Nablasoft company is able to let Acme work with Subversion but they can keep all their work with history in a subfolder of Company’s TFS and maintaining association with Work Item. The only drawback of this approach is “Time Compression”, because all the commits that are transferred in TFS do not maintain the original date of Subversion Repository, because all changesetId should maintain sequential order of timestamp. To minimize this problem, you can script the import process manually and run several time a day.

image

Figure 5: Association between check-in and Work Item based on content of Comment.

Summary

Thanks to git you can actually create a collaboration bridge between an external company that works with subversion and the internal TFS. The steps are

1) Send to external company the list of task/feature to do thanks to TFS Work Item Emailling system
2) Let the external company works with their subversion, but ask them to include #workitemId in subversion comments
3) Clone subversion repository of External Company in a local Git Repository
4) Connect that repository to TFS with git-tf
5) Move code from TFS and subversion using git as a bridge

If you use Git as a back end of your TFS Team Project the overall process will be even simpler, because you do not need to use git-tf, the association between code and Work Items will be automatically done by TFS and finally you will not suffer Time Compression issue.

Gian Maria.