Update GitVersion for large repositories

As you know I’m a fanatic user of GitVersion in builds, and I’ve written a simple task to use it in a TFS Build automatically. This is the typical task that you write and forget, because it just works and you usually not have the need to upgrade it. But there is a build where I start to see really high execution timing for the task, as an example GitVersion needs 2 minutes to run.

image

Figure 1: GitVersion task run time it is too high, 2 minutes

This behavior is perfectly reproducible on my local machine, that repository is quite big, it has lots of tags, feature branches and seems that GitFlow needs lot of time to determine semantic versioning.

Looking in the GitHub page of the project, I read some issue regarding performance problem, the good part is that all performance problem seems to disappear with the newer version, so it is time to upgrade the task.

Thanks to the new build system, updating a task in VSTS / TFS is really simple, I just deleted the old GitVersion executable and libraries from the folder where the task was defined, I copied over the new Gitversion.exe and libraries and with a tfx build tasks upload command I can immediately push the new version on my account.

Since I changed GitVersion from version 2 to version 3, it is a good practice to update the Major number of the task, to avoid all build to automatically use the new GitVersion executables, because it could break something. What I want is all build to show me that we have a new version of the task to use, so you can try and stick using the old version if Gitversion 3.6 gives you problems.

Whenever you do major change on a TFS / VSTS Build task it is a good practice to upgrade major number to avoid all builds to automatically use the new version of the task.

Thanks to versioning, you can decide if you want to try out the new version of the task for each distinct build.

image 

Figure 2: Thanks to versioning the owner of the build can choose if  the build should be upgraded to use the new version of GitVersion task.

After upgrading the build just queue a new build and verify if the task runs fine and especially if the execution time is reduced.

image

Figure 3: New GitVersion executable have a real boost in performance.

With few minutes of work I upgraded my task and I’m able to use new version of GitVersion.exe in all the builds, reducing the execution time significantly. Comparing this with the old XAML build engine and you understand why you should migrate all of your XAML Build to the new Build System as soon as possible.

Gian Maria.

Bash pornography

Suppose you have two Git repositories that are in sync with a VSTS build, after some time the usual problem is that, branches that are deleted from the “main” repository are not deleted from mirrored repository.

The reason is, whenever someone does a push on main repository, VSTS build push the branch to the other repository, but when someone deleted a branch, there is no build triggered, so the mirrored repository still maintain all branches.

This is usually not a big problem, I can accept that cloned repository still maintain deleted branches for some time after a branch is deleted from the original repository and I can accept that this kind of cleanup could be scheduled or done manually periodically. My question is:

In a git repository with two remotes configured, how can I delete branch in remote B that are not present in remote A?

I am pretty sure that there are lots of possible solutions using node.js or PowerShell or other scripting language. After all we are programmer and it is quite simple to issue a git branch –r command, parsing the output to determine branches in repository B that are not in repository A, then issue a git push remote –delete command.

Since I use git with Bash even in Windows, I really prefer a single line bash instruction that can do everything for me.

The solution is this command and I must admit that it is some sort of Bash Pornography :D.

diff --suppress-common-lines <(git branch -r | grep origin/ | sed 1d | sed "s/origin\///")  " | sed "s/>\s*//" | xargs git push --delete vsts

This is some sort of unreadble line, but it works. It is also not really complex, you only need to consider the basic constituent, first of all I use the diff command to diff two input stream. The general syntax is diff <(1) <(2) where 1 and 2 are output of a bash command. In the above example (1) is:

(git branch -r | grep origin/ | sed 1d | sed "s/origin\///")

This command list all the branches, then keep only lines that start with origin/ (grep origin/ ), then removes the first line (sed 1d) because it contains origin HEAD pointer, and finally remove the trailing origin/ from each line (sed “s/origin\///”). If I run this I got something like this.

SNAGHTML16d0ce1

Figure 1: Output of the command to list all branches in a remote

This is a simple list of all branches that are present on remote origin. If you look at the big command line, you can notice that the diff instruction diffs the output of this command and another identical command that is used to list branches in VSTS.

The output of the diff command is one line for each branch that is in VSTS and it is not in origin, so you can pipe this list to xargs, to issue a push –delete for each branch.

Now you can use this command simply issuing a git fetch for each remotes, then paste the instruction in your bash, press return and the game is done.

image

Figure 2: Results of the command, two branches are deleted from VSTS remote because they were deleted from the origin remote

It is amazing what you can do with bash + awk + sed in a single line 😛

Gian Maria.

Keep Git repository in sync between VSTS / TFS and Git

Scenario: you have a repository in Git, both open source or in private repository and you want to keep a synchronized mirror in VSTS / TFS.

There are some legitimate reason to have a mirrored repository between Github or some external provider and an instance of VSTS / TFS, probably the most common one is keeping all development of a repository private and publish in open source only certain branches. Another reason is having all the code in Github completely in open source, but internally use VSTS Work Item to manage work with all the advanced tooling VSTS has to offer.

The solution to this problem is really simple, just use a build in VSTS that push new commits from GitHub to VSTS or the opposite. Lets suppose that you have a GitHub repository and you want it to be mirrored in VSTS.

Step 1 – install extension to manipulate variables

Before creating the build you should install Variable Toolbox extension from the marketplace. This extension allows you to manipulate build variable and it is necessary if you use GitFlow.

From the list of Build Variables available in the build system there are two variables that contains information about the branch that is to be build. They are  called Build.SourceBranch and Build.SourceBranchName, but noone of them contains the real name of the branch. The SourceBranch contains the full name refs/heads/branchname while SourceBranchName contains the last path segment in the ref. If you use gitflow and have a branch called hotfix/1.2.3 the full name of the branch is refs/heads/hotfix/1.2.3 and the variable SourceBranchName contains the value 1.2.3 …. not really useful.

Thanks to the Variable Toolbox Extension you can simple configure the task to replace the refs/heads part with null string, so you can have a simple way to have a variable that contains the real name of the build even if it contains a slash character.

Step 2 – configure the build

The entire build is composed by three simple task, the very first is a Transform Value task (from Variable Toolbox ) followed by two simple command line.

SNAGHTML3a85c0

Figure 1: The entire build is three simple tasks.

The first task is used to remove the refs/heads/ part from the $(Build.SourceBranch) and copy the result to the GitBranchName variable (you should have it defined in the variables tab).

image

Figure 2: Transformation variable configured to remove refs/heads

Now we need a first command line task that checkout the directory, because the build does not issue a checkout in git, but it simple works in detatched HEAD.

image

Figure 3: Checkout git through commandline

As you can see in Figure 3 this operation is really simple, you can invoke git in command line, issuing the command checkout $(GitBranchName) created in precedent step, finally you should specify that this command should be executed in $(Build.SourcesDirectory).

The last command line pushes the branch to a local VSTS Repository.

image

Figure 4: Git command line to push everything on VSTS

The configuration is really simple, I decided to push to address https://$(token)@myaddress.visualstudio.com. Token variable (2) is a custom secret variable where I store a valid Personal Access Token that has right to access the code. To push on remote repository the syntax $(GitBranchName):$(GitBranchName) to push local branch on remote repository with –force option to allow forcing the push.

Do not forget to make your token variable as a secret variable and configure the continuous integration to keep syncronized only the branch you are interested to.

image

Figure 5: Configure branches you want to keep syncronized

If you need also to keep tags syncronized between builds you can just add another command line git invokation that pushes all tags with the push –tags option.

The result

Thanks to this simple build, whenever we push something on GitHub, a build starts that automatically replicate that branch in VSTS without any user intervention.

image

Figure 5: Build result that shows command line in action during a build.

Thanks to the free build minutes on the hosted build, we have a complete copy in VSTS of a GitHub repository with automatic sync running in few minutes.

The very same configuration can be reversed to automatically push to GitHub some branches of your VSTS account, useful if you want to publish only some branches in open source, automatically.

Gian Maria.

Import a Git Project with REST API between VSTS Team Projects

I’ve got an interesting question about the possibility to import via REST API a Git Repository between Team Projects of VSTS. Actually the problem is: you want to import a private git repository from a Source repository (in this situation is another VSTS git repository but it could be hosted everywhere) to a VSTS Target  repository using only REST API.

The operation is quite simple thanks to the new api described here (https://www.visualstudio.com/en-us/docs/integrate/api/git/import-requests#create-a-request-to-import-a-repository) and in this post I’ll give you all the details.

Step 1 – create a PAT

To access VSTS through REST API you have many option to authenticate the call, but the easiest one is using PAT (Personal Access Token). If you do no already have a valid PAT you can create one using security page of your account.

image

Figure 1: Open security page of your account

Creating a PAT is really simple, you should only select Personal Access Token (1), then give a description, an expiration time, and the account where PAT is valid into. Since I have more than one VSTS Account I have a combo where all of my account are listed (2).

Finally you should select only the permission you want to give to the token. The default option is All Scopes, and this will imply that the token can do pretty much anything you can do. If you need this token to manage import of repositories you can simply select only code related permission.

image

Figure 2: Create a PAT to access your account.

Personal Access Token are the most secure way to authenticate an application in VSTS because they can be revoked, you can choose the permission you want to give to the token and they have an automatic expiration.

If your Source Account is on a different account from the Target Account you need to create PAT both in Source Account VSTS Instance and in Target Account VSTS Instance. In this example VSTS instance is the very same, so I need only one PAT.

Step 2 – Create endpoint to access Source Repository

My target repository is called ImportTest, and it is important that this repository is created empty. This is my Target Repository, the repository where I want to import the Source Repository.

image

Figure 3: Create Target Repository with standard Web Interface

The import routine should be able to access Source Repository and this imply that it needs to be authenticated. To maximize security you need to create an Endpoint that point to the Source Repository in the Team Project of Target Repository. This can be easily done from the administration page of the Team Project that contains the Target Repository. The team project that contains my ImportTest repository is contained in GitMiscellaneous Team Project, and I can proceed to manually create the endpoint.

image

Figure 4: Create an endpoint of type External Git

image

Figure 5: Specify endpoint details

In Figure 5 you can see all the options needed, you should specify a connection name, then the URL parameter is the url of the Source Repository, the same url you use to clone the repository. Finally you need to use the PAT as username, then you can press OK.

This service endpoint should be created in the Team Project that contains the Target Repository, because it will be used by the import routine to authenticate to the Source Repository to take data to import.

An endpoint is basically an URL and an authentication that is used by the server to access an external service

If you need to automate the whole process, the endpoint can be created easily with REST API  (https://www.visualstudio.com/en-us/docs/integrate/api/endpoints/endpoints) here is a simple call in Postman.

image

Figure 6: Creation of the endpoint with REST API

This does not need any explanation because it is a simple call with the very same option that you specify on the UI.

Step 3 – Create the call to import repository

To create the call to start repository import routine you need some parameters: first of all you need the id of the Endpoint you created in step 2. If you created the endpoint through REST API this is not a problem, because the Id is present in the response

image

Figure 7: Response of the request shown in Figure 6 contains endpoint Id

If you created the endpoint through Web UI the id can be grabbed by the url in the administration page of the endpoints, but a simpler and better method is to list all endpoint of the Team Project through REST API. In my situation is a simple GET call to this url https://gianmariaricci.VisualStudio.com/GitMiscellaneous/_apis/distributedtask/serviceendpoints?api-version=3.0-preview.1

The answer is the very same of Figure 7, and this gives me the id of the endpoint that points to the Source Repository: df12f2e3-7c40-4885-8dbd-310f1781369a

Now I need to create the import request, as described here (https://www.visualstudio.com/en-us/docs/integrate/api/git/import-requests#create-a-request-to-import-a-repository). And the only information I’m missing is the Id of the Target Repository

image

Figure 8: Repository part of the url in the call should be replaced by repository ID

As shown in Figure8 the only annoying part of the request is the Id of the Target Repository because it is the GUID of the repository not the name. Obtaining this value is not difficult, because with REST API this is a simple GET call to this url: https://gianmariaricci.VisualStudio.com/DefaultCollection/GitMiscellaneous/_apis/git/repositories?api-version=1.0. From the answer of this call the ID of the ImportTest repository is: 3037268a-0c91-4fe1-8435-a76e9b731f5e

Now I have everything to create the import request, just forge the request in Postman or similar tool and fire the request.

image

 Figure 9: The import request where 1 is the ID of Target Repository and 2 is the ID of the endpoint.

If you are quick enough and refresh the page of Target Repository while the import routine is running, you should be able to see this image

image

Figure 10: Importing is running

After a little bit (depending on the source of Source Repository) the Target Repository will be a perfect clone of the Source Repository .

If there are errors during the import process in the source code page of Target Repository  you are warned with the error, as shown in Figure 11.

image

Figure 11: Error in the importing routing were shown to source  code page of Target Repository

As an example the error in the above image is due to a misconfiguration of the Endpoint (done in part 2), as an example if you created the endpoint with wrong credentials.

Gian Maria

Git for windows, getting Invalid username or password with Wincred

If you use Https to communicate with your git repository, Es, Github or VisualStudioOnline, you usually setup credential manager to avoid entering credential for each command that contact the server. With latest versions of git you can configure wincred with this simple command.

git config --global credential.helper wincred

This morning I start getting error while I’m trying to push some commits to GitHub.

$ git push
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/ProximoSrl/Jarvis.DocumentS
tore.git/'

If I remove credential helper (git config –global credential.helper unset) everything works, git ask me for user name and password and I’m able to do everything, but as soon as I re-enable credential helper, the error returned. This problem is probably originated by some corruption of stored credentials, and usually you can simply clear stored credentials and at the next operation you will be prompted for credentials and everything starts worked again. The question is, where are stored credential for wincred?

If you use wincred for credential.helper, git is storing your credentials in standard windows Credential Manager

You can simple open credential manager on your computer,

image

Figrue 1: Credential manager in your Control Panel settings

Opening Credential manager you can manage windows and web credentials. Now simply have a look to both web credentials and windows credentials, and delete everything related to GitHub or the server you are using. The next time you issue a git command that requires authentication, you will be prompted for credentials again and the credentials will be stored again in the store.

Gian Maria.