Keep Git repository in sync between VSTS / TFS and Git

Scenario: you have a repository in Git, both open source or in private repository and you want to keep a synchronized mirror in VSTS / TFS.

There are some legitimate reason to have a mirrored repository between Github or some external provider and an instance of VSTS / TFS, probably the most common one is keeping all development of a repository private and publish in open source only certain branches. Another reason is having all the code in Github completely in open source, but internally use VSTS Work Item to manage work with all the advanced tooling VSTS has to offer.

The solution to this problem is really simple, just use a build in VSTS that push new commits from GitHub to VSTS or the opposite. Lets suppose that you have a GitHub repository and you want it to be mirrored in VSTS.

Step 1 – install extension to manipulate variables

Before creating the build you should install Variable Toolbox extension from the marketplace. This extension allows you to manipulate build variable and it is necessary if you use GitFlow.

From the list of Build Variables available in the build system there are two variables that contains information about the branch that is to be build. They are  called Build.SourceBranch and Build.SourceBranchName, but noone of them contains the real name of the branch. The SourceBranch contains the full name refs/heads/branchname while SourceBranchName contains the last path segment in the ref. If you use gitflow and have a branch called hotfix/1.2.3 the full name of the branch is refs/heads/hotfix/1.2.3 and the variable SourceBranchName contains the value 1.2.3 …. not really useful.

Thanks to the Variable Toolbox Extension you can simple configure the task to replace the refs/heads part with null string, so you can have a simple way to have a variable that contains the real name of the build even if it contains a slash character.

Step 2 – configure the build

The entire build is composed by three simple task, the very first is a Transform Value task (from Variable Toolbox ) followed by two simple command line.

SNAGHTML3a85c0

Figure 1: The entire build is three simple tasks.

The first task is used to remove the refs/heads/ part from the $(Build.SourceBranch) and copy the result to the GitBranchName variable (you should have it defined in the variables tab).

image

Figure 2: Transformation variable configured to remove refs/heads

Now we need a first command line task that checkout the directory, because the build does not issue a checkout in git, but it simple works in detatched HEAD.

image

Figure 3: Checkout git through commandline

As you can see in Figure 3 this operation is really simple, you can invoke git in command line, issuing the command checkout $(GitBranchName) created in precedent step, finally you should specify that this command should be executed in $(Build.SourcesDirectory).

The last command line pushes the branch to a local VSTS Repository.

image

Figure 4: Git command line to push everything on VSTS

The configuration is really simple, I decided to push to address https://$(token)@myaddress.visualstudio.com. Token variable (2) is a custom secret variable where I store a valid Personal Access Token that has right to access the code. To push on remote repository the syntax $(GitBranchName):$(GitBranchName) to push local branch on remote repository with –force option to allow forcing the push.

Do not forget to make your token variable as a secret variable and configure the continuous integration to keep syncronized only the branch you are interested to.

image

Figure 5: Configure branches you want to keep syncronized

If you need also to keep tags syncronized between builds you can just add another command line git invokation that pushes all tags with the push –tags option.

The result

Thanks to this simple build, whenever we push something on GitHub, a build starts that automatically replicate that branch in VSTS without any user intervention.

image

Figure 5: Build result that shows command line in action during a build.

Thanks to the free build minutes on the hosted build, we have a complete copy in VSTS of a GitHub repository with automatic sync running in few minutes.

The very same configuration can be reversed to automatically push to GitHub some branches of your VSTS account, useful if you want to publish only some branches in open source, automatically.

Gian Maria.

Import a Git Project with REST API between VSTS Team Projects

I’ve got an interesting question about the possibility to import via REST API a Git Repository between Team Projects of VSTS. Actually the problem is: you want to import a private git repository from a Source repository (in this situation is another VSTS git repository but it could be hosted everywhere) to a VSTS Target  repository using only REST API.

The operation is quite simple thanks to the new api described here (https://www.visualstudio.com/en-us/docs/integrate/api/git/import-requests#create-a-request-to-import-a-repository) and in this post I’ll give you all the details.

Step 1 – create a PAT

To access VSTS through REST API you have many option to authenticate the call, but the easiest one is using PAT (Personal Access Token). If you do no already have a valid PAT you can create one using security page of your account.

image

Figure 1: Open security page of your account

Creating a PAT is really simple, you should only select Personal Access Token (1), then give a description, an expiration time, and the account where PAT is valid into. Since I have more than one VSTS Account I have a combo where all of my account are listed (2).

Finally you should select only the permission you want to give to the token. The default option is All Scopes, and this will imply that the token can do pretty much anything you can do. If you need this token to manage import of repositories you can simply select only code related permission.

image

Figure 2: Create a PAT to access your account.

Personal Access Token are the most secure way to authenticate an application in VSTS because they can be revoked, you can choose the permission you want to give to the token and they have an automatic expiration.

If your Source Account is on a different account from the Target Account you need to create PAT both in Source Account VSTS Instance and in Target Account VSTS Instance. In this example VSTS instance is the very same, so I need only one PAT.

Step 2 – Create endpoint to access Source Repository

My target repository is called ImportTest, and it is important that this repository is created empty. This is my Target Repository, the repository where I want to import the Source Repository.

image

Figure 3: Create Target Repository with standard Web Interface

The import routine should be able to access Source Repository and this imply that it needs to be authenticated. To maximize security you need to create an Endpoint that point to the Source Repository in the Team Project of Target Repository. This can be easily done from the administration page of the Team Project that contains the Target Repository. The team project that contains my ImportTest repository is contained in GitMiscellaneous Team Project, and I can proceed to manually create the endpoint.

image

Figure 4: Create an endpoint of type External Git

image

Figure 5: Specify endpoint details

In Figure 5 you can see all the options needed, you should specify a connection name, then the URL parameter is the url of the Source Repository, the same url you use to clone the repository. Finally you need to use the PAT as username, then you can press OK.

This service endpoint should be created in the Team Project that contains the Target Repository, because it will be used by the import routine to authenticate to the Source Repository to take data to import.

An endpoint is basically an URL and an authentication that is used by the server to access an external service

If you need to automate the whole process, the endpoint can be created easily with REST API  (https://www.visualstudio.com/en-us/docs/integrate/api/endpoints/endpoints) here is a simple call in Postman.

image

Figure 6: Creation of the endpoint with REST API

This does not need any explanation because it is a simple call with the very same option that you specify on the UI.

Step 3 – Create the call to import repository

To create the call to start repository import routine you need some parameters: first of all you need the id of the Endpoint you created in step 2. If you created the endpoint through REST API this is not a problem, because the Id is present in the response

image

Figure 7: Response of the request shown in Figure 6 contains endpoint Id

If you created the endpoint through Web UI the id can be grabbed by the url in the administration page of the endpoints, but a simpler and better method is to list all endpoint of the Team Project through REST API. In my situation is a simple GET call to this url https://gianmariaricci.VisualStudio.com/GitMiscellaneous/_apis/distributedtask/serviceendpoints?api-version=3.0-preview.1

The answer is the very same of Figure 7, and this gives me the id of the endpoint that points to the Source Repository: df12f2e3-7c40-4885-8dbd-310f1781369a

Now I need to create the import request, as described here (https://www.visualstudio.com/en-us/docs/integrate/api/git/import-requests#create-a-request-to-import-a-repository). And the only information I’m missing is the Id of the Target Repository

image

Figure 8: Repository part of the url in the call should be replaced by repository ID

As shown in Figure8 the only annoying part of the request is the Id of the Target Repository because it is the GUID of the repository not the name. Obtaining this value is not difficult, because with REST API this is a simple GET call to this url: https://gianmariaricci.VisualStudio.com/DefaultCollection/GitMiscellaneous/_apis/git/repositories?api-version=1.0. From the answer of this call the ID of the ImportTest repository is: 3037268a-0c91-4fe1-8435-a76e9b731f5e

Now I have everything to create the import request, just forge the request in Postman or similar tool and fire the request.

image

 Figure 9: The import request where 1 is the ID of Target Repository and 2 is the ID of the endpoint.

If you are quick enough and refresh the page of Target Repository while the import routine is running, you should be able to see this image

image

Figure 10: Importing is running

After a little bit (depending on the source of Source Repository) the Target Repository will be a perfect clone of the Source Repository .

If there are errors during the import process in the source code page of Target Repository  you are warned with the error, as shown in Figure 11.

image

Figure 11: Error in the importing routing were shown to source  code page of Target Repository

As an example the error in the above image is due to a misconfiguration of the Endpoint (done in part 2), as an example if you created the endpoint with wrong credentials.

Gian Maria

Git for windows, getting Invalid username or password with Wincred

If you use Https to communicate with your git repository, Es, Github or VisualStudioOnline, you usually setup credential manager to avoid entering credential for each command that contact the server. With latest versions of git you can configure wincred with this simple command.

git config --global credential.helper wincred

This morning I start getting error while I’m trying to push some commits to GitHub.

$ git push
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/ProximoSrl/Jarvis.DocumentS
tore.git/'

If I remove credential helper (git config –global credential.helper unset) everything works, git ask me for user name and password and I’m able to do everything, but as soon as I re-enable credential helper, the error returned. This problem is probably originated by some corruption of stored credentials, and usually you can simply clear stored credentials and at the next operation you will be prompted for credentials and everything starts worked again. The question is, where are stored credential for wincred?

If you use wincred for credential.helper, git is storing your credentials in standard windows Credential Manager

You can simple open credential manager on your computer,

image

Figrue 1: Credential manager in your Control Panel settings

Opening Credential manager you can manage windows and web credentials. Now simply have a look to both web credentials and windows credentials, and delete everything related to GitHub or the server you are using. The next time you issue a git command that requires authentication, you will be prompted for credentials again and the credentials will be stored again in the store.

Gian Maria.

Monitor if your branch will generate merge conflicts with TFS Build

One of the greatest advantage in using Git over a centralized Version Control System is branching system. It is quite common for developers to start branching whenever they need to add a new feature, work on that branch and finally merge back to mainline when the feature is finished. One of the most famous workflow is called GitFlow, a way to work in Git that is implemented even in some GUI tool like SourceTree.

One of the main problem when you heavily use feature branches is the moment you merge back to mainline. One of the technique to easy the pain is periodically do a forward integration from mainline to feature branch, because it is usually easier doing several little merge than doing a big merge when the feature is complete.

Assuming that the mainline contains only stable code, because all feature branch merge to mainline only when they are stable, it is usually safe to periodically merge from mainline to branch to avoid a big merge at the end. In this scenario the question usually is: how frequently I have to merge? One possible solution is to take advantage of your Continuous Integration system to warn you when a modification you did on a branch introduces problem because:

  • Does not merge automatically with mainline but it generates conflicts
  • Even if the branch merge automatically the code wont compile.
  • Even if the branch merge automatically and the code compile, some tests where broken.

With TFS Build you can achieve this result quite simply, first of all configure a build with a trigger for each push, then in the Source Settings specify that this build should be triggered whenever any branch that starts with feat got any push.

image

Figure 1: Source code configuration, only build when a branch that starts with feat got a code increment

This build is intended to monitor the status of all feature branches. All you need to do is including a PowerShell script in source code that will be run pre-build. This script basically merge with master with command line git (Build Agent where that build runs should have git installed and available in PATH environment variable). This is a possible PowerShell script to accomplish the merge

$teststring = $env:TF_BUILD_SOURCEGETVERSION

$branchName = ""
$teststring | where { $_ -match "/(?[^/]*?):" } |
    foreach { $branchName = $Matches["bname"] }

Write-Host "BranchName = $branchName"
$rootDir = "$env:TF_BUILD_BUILDDIRECTORY\src"
Write-Host "Source folder is $rootDir"

#be sure to checkout the right branch 
$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "git"
$ps.StartInfo.WorkingDirectory = $rootDir
$ps.StartInfo.Arguments = "checkout $branchName"
$ps.StartInfo.RedirectStandardOutput = $True
$ps.StartInfo.RedirectStandardError = $True
$ps.StartInfo.UseShellExecute = $false
$ps.start()

[string] $Out = $ps.StandardOutput.ReadToEnd();
[string] $ErrOut = $ps.StandardError.ReadToEnd();
Write-Host "Output git checkout $branchName is:" $Out
Write-Host "Return value: " $ps.ExitCode
if ($ps.ExitCode -ne 0) 
{
    $Host.ui.WriteErrorLine("Branch cannot be checked out")
}

#try to merge


$ps = new-object System.Diagnostics.Process
$ps.StartInfo.Filename = "git"
$ps.StartInfo.WorkingDirectory = $rootDir
$ps.StartInfo.Arguments = " merge master"
$ps.StartInfo.RedirectStandardOutput = $True
$ps.StartInfo.RedirectStandardError = $True
$ps.StartInfo.UseShellExecute = $false
$ps.start()

[string] $Out = $ps.StandardOutput.ReadToEnd();
[string] $ErrOut = $ps.StandardError.ReadToEnd();
Write-Host "Output git merge master:" $Out
Write-Host "Return value: " $ps.ExitCode

if ($ps.ExitCode -ne 0) 
{
    $Host.ui.WriteErrorLine("Branch does not merge correctly")
}

This is really a first tentative in this direction, there probably better way to do it, but it solves the problem without any great complexity. It just uses a regular expression to find branch name from the environment variable TF_BUILD_SOURCEGETVERSION. Once you have name of the branch just checkout it and then try to “git merge master” to merge the branch with master. Quite all git command return 0 if the operation is successful, so if the ExitCode of “git merge master” command is different from 0 it means that the merge operation did not complete correctly (conflicts occurred). If the merge is unsuccessful the script use UI.WriteErrorLine to write error on the UI and this has two effects: first the message will be reported on the Build Summary, then then build will be marked as partially failed.

Now even if your branch compile correctly and every test is green, after you push to TFS your Build Server will try to merge and run the build, if the branch does not automatically merge with master the build will partially fail and you got this:

image

Figure 2: Code does not automatically merge after the latest push.

This is a signal that is time to do a forward integration from mainline to your branch, because the amount of conflicts will be probably small. Another advantage is that you can merge the code while your modification is fresh in your mind. Nothing is more painfully than merge code you have written a month ago.

If the merge is successfully it does not mean that the code is in good health. Suppose you rename a method in your branch, but in the mainline another one has pushed an increment with another class that uses that method. This is the perfect example of code that gave no merge conflicts, but it does not even compile. Since the script actually does the merge before the build, the rest of the build works on the code result of the merge, now the build fails.

image

Figure 3: Merge was successfully, but the result of the merge does not compile

In this situation you surely have a standard build that simply works on the feature branch that is green, but the Build System can run for you this another special build that runs on the result of the merge with mainline that is Red. This means: In your branch code everything is green, but when you will merge to mainline you will have problem, so it is probably better to look at that problems now!. 

The very same happens if the result of the merge breaks some unit test. In conclusion, thanks to few PowerShell lines, you can instruct your TFS build to automatically try to merge each increment of code in your feature branches and then compile, run test and do whatever any operation you usually do on your build on the result of the merge between the feature branch and mainline. Thanks to this you can immediately merge from mainline to your branch to fix the problem as soon as you have introduced it.

This approach has some disadvantages. First of all a PowerShell script cannot make the build fails, so if the code does not merge, usually you will have a failing build because the build system try to build source code during a merge and find invalid characters in source code. Second, this special build runs only when there is a push on some feature branch, but nothing happens if someone pushes in the mainline. In that case you should run this special build for every feature branch you have in your repository.

Gian Maria.

Git submodule update error: Permission Denied Publickey

It could happens when you clone a Git Repository with submodules, issuing a git submodule update command, you are prompted with this error error.

Cloning into ‘src/xxxx’…
Warning: Permanently added the RSA host key for IP address xxx.xxx.xxx.xxx to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

If you search in the internet for the cause of errors, you can find some people suggesting that the url specified in .gitmodules file is wrong and should be changed, here is my .gitmodule

[submodule “src/CQRS”]
     path = src/CQRS
     url = git@github.com:xxxxxx/cqrs.git
     branch = master
    

You could change the url configuration to https url and everything works, but this is not the perfect solution, because the address git@github.com is perfectly valid, but probably there is some problem with your RSA keys stored in Github (or you never configured RSA Keys for your account). In my situation, my RSA Keys had some problem and I needed to recreate another one. If you do not know what a RSA key is and how to create a RSA Key to connect to github I strongly suggest you reading the guide: Generating SSH Keys.

Once you configure a valid certificate in github your submodule should word without problem.

Gian Maria.