SonarQube UTF-8 error after upgrading

I’ve upgraded a SonarQube instance, and then, suddently I realyze that some builds start failing due to to a strange error in the SonarQube complete analysis task.

2016-12-09T11:41:00.8695497Z ##[error]WARN:
 Invalid character encountered in file C:\vso\_work\3\s\src\Jarvis.ConfigurationService.Client.CastleIntegration\Properties\AssemblyInfo.cs at line 25 for encoding UTF-8.
 Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.

Thiserror in turns made the entire task execution failing with a really strange error

2016-12-09T11:41:03.5135467Z ##[error]ERROR: Error during SonarQube Scanner execution
2016-12-09T11:41:03.5145430Z ##[error]java.lang.IllegalArgumentException: 5 is not a valid line offset for pointer. File [moduleKey=Jarvis.ConfigurationService:Jarvis.ConfigurationService:893ED554-3D86-47C8-B529-965329DB32AF, relative=Properties/AssemblyInfo.cs, basedir=C:\vso\_work\3\s\src\Jarvis.ConfigurationService.Client.CastleIntegration] has 1 character(s) at line 2
2016-12-09T11:41:03.5145430Z ##[error]at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
2016-12-09T11:41:03.5145430Z ##[error]at org.sonar.api.batch.fs.internal.DefaultInputFile.checkValid(DefaultInputFile.java:215)
2016-12-09T11:41:03.5145430Z ##[error]at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:206)

This is really annoying, but actually I start investigating the error logging to the build server and checking the file that generates the error. When you have strange encoding erorr I strongly suggested you to visualize the file with some Hex editor, and not a standard editor.

I immediately realized that the error is due to my GitVersion task script, because it manipulates assemblyinfo.cs files to change version numbers and save back the assemblyinfo.cs in UTF-16 encoding. If you do not know Byte Order Mark I strongly suggest you to take a look in wikipedia, because this is an important concepts for Unicode files.

I immediately checked with an hex editor, and verified that the original assemblyinfo.cs has a BOM for UTF-8 but the PowerShell script modified it converting to UTF-16 but the BOM is correct. The annoying stuff is that the build worked perfectly until I updated SonarQube (server + analyzers) this means that for some reason, the most recent version of Sonar somewhat does not check the BOM to understand file encoding.

I’ve solved the problem simply changing the Set-Content call to force UTF8 and the problem is gone.

Set-Content -Encoding UTF8 -Path $file.FullName -Force

I really like SonarQube, but you should always verify that everything works after every upgrade.

Gian Maria.

Enable new Work Item Form in TFS “15”

If you installed TFS 15 Preview, one of the news you expected to see is the new Work Item Layout (already available in VSTS). You could get disappointed that actually your existing Work Items still are shown with the old interface, as you can see in Figure 1

image

Figure 1: After upgrade TFS still shows the old Work Item Form

The new Work Item Form is installed with an opt-in method so it is disabled by default. To enable it you should navigate to the Project Collection administration page. From here you should see that this feature is actually disabled (Figure 2), but you have the link to Enable it.

image

Figure 2: New Work Item form is disabled by default after upgrade

If you click the “Enable the new work item form” link, you are informed that this operation will create a new layout for the Work Item, but you can choose, after the creation of the new layout, if you want to use the new model, and the opt-in model, as you can see in Figure 3.

image

Figure 3: Enabling the new Work Item layout starts with the creation of the new Layout.

Thanks to the opt-in method, you are not forced to use the new layout, but you can activate it only if you want to use it

After the creation of the Layout you should configure the Opt-in model, or you can disable entirely the new Work Item Form.

image

Figure 4: Options to enable the new Work Item form with the opt-in model

The opt-in model basically allows you to decide who can view the new Work Item form layout. You have three options, as shown in Figure 5, you can give the ability to use the new layout only to administrators, to all user, or you can force everyone to use the new Layout, disabling the old layout entirely.

image

Figure 5: Opt-in model options to enable the new Layout form

Opt-in model for the new Layout Form is really flexible because you can leave the decision up to each single user.

The central option is usually the less impacting, because each member of the team can choose to evaluate the new layout or stick with the old one. A new link appears on the head of the Work Item Form, in the far right part, as you can see in Figure 6.

image

Figure 6: Opt in model allows each user to choose to evaluate the new form.

If the user choose to preview the new form, the page refresh and the Work Item is rendered with the new layout. The user has the option to return to the old form if he do not like the new form, giving the whole team the time to evaluate the feature and decide if using it or not.

SNAGHTML7928e8

Figure 8: New form is active and the user can back to old form if the do not like it.

This settings is “per collection” so you can decide different opt-in model for each collection of your TFS.

Gian Maria.

Bulk insert in MongoDb with C# driver

There are situation where you need to save a lot of documents inside a collection in MongoDb. My scenario is a migration of documents from a collection to another database, with in-memory manipulation of the documents.

The most common error in these situation is to read the documents from the original collection, then execute a function that modify the document in-memory, and finally issuing an insert in destination collection. This is wrong because you have a roundrip against MongoDb for each document you are saving.

Whenever you are calling Insert or Save function, you are paying the penality of a call to MongoDb process, network latency, etc, whenever possible you should reduce the number of calls to database engine.

In such a scenario MongoDb driver has a function called InsertBatch that allows you to insert document in batches and the fun part is that it simply accepts an IEnumerable. As an example, I have a function that manipulate a BsonDocument stored in a variable called Action, I have source and dest database where I need to copy documents with manipulation and this is the code that does everything.

 var sourceQueue = source.GetCollection(queue);
var destQueue = dest.GetCollection(queue);

if (sourceQueue.Count() == 0) return;

//migrate counterCollection
Console.WriteLine("Migrating Queue " + queue);
var allElement = sourceQueue
	.FindAll()
	.AsEnumerable()
	.Select(document => {
		Action(document);
		return document;
	});
destQueue.InsertBatch(allElement);

The name of the collection is contained in queue variable (actually I’m transforming a software that manage jobs), and as you can verify I can simply enumerate all source documents with FindAll (this code uses old 1.10 driver), for each object I’m calling the Action function that manipulate the document, and finally I can simply use the InsertBatch to insert documents in batches.

This function runs really faster than saving each document with a separate call, event if the MongoDb instance runs on the very same machine, so you do not pay the network latency.

If you use latest version of the drivers, you have the InsertMany method that offers even more options and basically does the very same operation than InsertBatch.

Gian Maria.

Keep Git repository in sync between VSTS / TFS and Git

Scenario: you have a repository in Git, both open source or in private repository and you want to keep a synchronized mirror in VSTS / TFS.

There are some legitimate reason to have a mirrored repository between Github or some external provider and an instance of VSTS / TFS, probably the most common one is keeping all development of a repository private and publish in open source only certain branches. Another reason is having all the code in Github completely in open source, but internally use VSTS Work Item to manage work with all the advanced tooling VSTS has to offer.

The solution to this problem is really simple, just use a build in VSTS that push new commits from GitHub to VSTS or the opposite. Lets suppose that you have a GitHub repository and you want it to be mirrored in VSTS.

Step 1 – install extension to manipulate variables

Before creating the build you should install Variable Toolbox extension from the marketplace. This extension allows you to manipulate build variable and it is necessary if you use GitFlow.

From the list of Build Variables available in the build system there are two variables that contains information about the branch that is to be build. They are  called Build.SourceBranch and Build.SourceBranchName, but noone of them contains the real name of the branch. The SourceBranch contains the full name refs/heads/branchname while SourceBranchName contains the last path segment in the ref. If you use gitflow and have a branch called hotfix/1.2.3 the full name of the branch is refs/heads/hotfix/1.2.3 and the variable SourceBranchName contains the value 1.2.3 …. not really useful.

Thanks to the Variable Toolbox Extension you can simple configure the task to replace the refs/heads part with null string, so you can have a simple way to have a variable that contains the real name of the build even if it contains a slash character.

Step 2 – configure the build

The entire build is composed by three simple task, the very first is a Transform Value task (from Variable Toolbox ) followed by two simple command line.

SNAGHTML3a85c0

Figure 1: The entire build is three simple tasks.

The first task is used to remove the refs/heads/ part from the $(Build.SourceBranch) and copy the result to the GitBranchName variable (you should have it defined in the variables tab).

image

Figure 2: Transformation variable configured to remove refs/heads

Now we need a first command line task that checkout the directory, because the build does not issue a checkout in git, but it simple works in detatched HEAD.

image

Figure 3: Checkout git through commandline

As you can see in Figure 3 this operation is really simple, you can invoke git in command line, issuing the command checkout $(GitBranchName) created in precedent step, finally you should specify that this command should be executed in $(Build.SourcesDirectory).

The last command line pushes the branch to a local VSTS Repository.

image

Figure 4: Git command line to push everything on VSTS

The configuration is really simple, I decided to push to address https://$(token)@myaddress.visualstudio.com. Token variable (2) is a custom secret variable where I store a valid Personal Access Token that has right to access the code. To push on remote repository the syntax $(GitBranchName):$(GitBranchName) to push local branch on remote repository with –force option to allow forcing the push.

Do not forget to make your token variable as a secret variable and configure the continuous integration to keep syncronized only the branch you are interested to.

image

Figure 5: Configure branches you want to keep syncronized

If you need also to keep tags syncronized between builds you can just add another command line git invokation that pushes all tags with the push –tags option.

The result

Thanks to this simple build, whenever we push something on GitHub, a build starts that automatically replicate that branch in VSTS without any user intervention.

image

Figure 5: Build result that shows command line in action during a build.

Thanks to the free build minutes on the hosted build, we have a complete copy in VSTS of a GitHub repository with automatic sync running in few minutes.

The very same configuration can be reversed to automatically push to GitHub some branches of your VSTS account, useful if you want to publish only some branches in open source, automatically.

Gian Maria.

Re-indexing source in TFS “15” preview

If you installed TFS “15” preview you should give a try to code search, because it is surely one of the coolest feature introduced in this new release.

If for some reason the indexing went wrong, or code is not indexed, you can try reindexing using some powershell scripts that are described in this post.

Happy TFS.