Exclude Folders from SonarQube analysis

Creating a build that is capable of perform a SonarQube analysis on a VSTS /  TFS is a really simple task, thanks to the two tasks that are present out-of-the box.

image

Figure 1: Build that uses Sonarqube tasks to perform analysis

The problem in a project that was alive for more than a couple of years is that you usually have a really bad report when you do your first analysis. This happens because, without a constant analysis, the code have surely some smells.

Sometimes you get really discouraged, because the number of issue is really high, but before losing any hope, check if the errors are really part of your code. In a project where I’m working, we got really bad numbers and I was 100% sure that it is not a problem of our code.

When you analyze your project for the first time, sometimes the number of issue is so high that you really are discouraged. Before giving up, check if the errors are really part of your code.

To diagnostic the problem, simply login to the project, then go to Code view (something like http://build:9000/code/?id=projectName), then you will see a summary of all the bugs, but unfortunately you cannot order for the number of the bug, so Just scroll down to see the part of the code with the most errors.

SNAGHTML139609

Figure 2: 185 bugs are located in scripts folder

In this situation, we have 185 bugs signaled by the javascript analyzer under the scripts folder, but in that folder we have our code but also third party code. This is not the very first analysis, because the first analysis shows a folder where there are more than 3k of errors and it was the folder that stores all third party javascript libraries.

If you do not use npm it is quite normal to have third party javascript code in your source code and if you are using SonarQube you absolutely need to configure it to exclude that directories. Just go to the administration page of the project and in the Analysis Scope you find the Source File Exclusions section that allows you to exclude some directories from the analysis.

SNAGHTML16d611

Figure 3: Exclude folder for analysis

In your situation the vast majority of errors comes from the angular library, from all the script of the skin we use and for third party libraries stored under the /app/scripts/lib folder. After exclusion, the number of bugs dropped from almost 7k to 500.

If you add Sonarqube analysis to existsting project that have third party javascript code in source code repository, please always check where the errors are and exclude folder accordingly.

Gian Maria.

SonarQube UTF-8 error after upgrading

I’ve upgraded a SonarQube instance, and then, suddently I realyze that some builds start failing due to to a strange error in the SonarQube complete analysis task.

2016-12-09T11:41:00.8695497Z ##[error]WARN:
 Invalid character encountered in file C:\vso\_work\3\s\src\Jarvis.ConfigurationService.Client.CastleIntegration\Properties\AssemblyInfo.cs at line 25 for encoding UTF-8.
 Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.

Thiserror in turns made the entire task execution failing with a really strange error

2016-12-09T11:41:03.5135467Z ##[error]ERROR: Error during SonarQube Scanner execution
2016-12-09T11:41:03.5145430Z ##[error]java.lang.IllegalArgumentException: 5 is not a valid line offset for pointer. File [moduleKey=Jarvis.ConfigurationService:Jarvis.ConfigurationService:893ED554-3D86-47C8-B529-965329DB32AF, relative=Properties/AssemblyInfo.cs, basedir=C:\vso\_work\3\s\src\Jarvis.ConfigurationService.Client.CastleIntegration] has 1 character(s) at line 2
2016-12-09T11:41:03.5145430Z ##[error]at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
2016-12-09T11:41:03.5145430Z ##[error]at org.sonar.api.batch.fs.internal.DefaultInputFile.checkValid(DefaultInputFile.java:215)
2016-12-09T11:41:03.5145430Z ##[error]at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:206)

This is really annoying, but actually I start investigating the error logging to the build server and checking the file that generates the error. When you have strange encoding erorr I strongly suggested you to visualize the file with some Hex editor, and not a standard editor.

I immediately realized that the error is due to my GitVersion task script, because it manipulates assemblyinfo.cs files to change version numbers and save back the assemblyinfo.cs in UTF-16 encoding. If you do not know Byte Order Mark I strongly suggest you to take a look in wikipedia, because this is an important concepts for Unicode files.

I immediately checked with an hex editor, and verified that the original assemblyinfo.cs has a BOM for UTF-8 but the PowerShell script modified it converting to UTF-16 but the BOM is correct. The annoying stuff is that the build worked perfectly until I updated SonarQube (server + analyzers) this means that for some reason, the most recent version of Sonar somewhat does not check the BOM to understand file encoding.

I’ve solved the problem simply changing the Set-Content call to force UTF8 and the problem is gone.

Set-Content -Encoding UTF8 -Path $file.FullName -Force

I really like SonarQube, but you should always verify that everything works after every upgrade.

Gian Maria.

Upgrading SonarQube from 5.1 to 6.0

SonarQube is a really nice software, but for what I experienced it does not play well with Sql Server. Even if Sql Server is fully supported, there are always some little problem in setting everything up, and this is probably due to the fact that most of the people using SonarQube are using MySql as Database Engine.

Today I was upgrading a test instance from version 5.1 to 6.0, I’ve installed the new version, launched database upgrade procedure, and after some minutes the upgrade procedure stopped with a bad error

image

Figure1: Databse error during upgrade.

Remember that SonarQube upgrade procedure does not have a rollback procedure, so it is mandatory that you take a full backup of the system before performing the upgrade.

From what I’ve learned in previous situation, whenever something does not work in SonarQube, you need to look at the log. The above error message is really misleading, because it seems that there were some connection problems. Reading the log I discovered that the error is really different.

2016.08.11 19:24:00 ERROR web[o.s.s.d.m.DatabaseMigrator] Fail to execute database migration: org.sonar.db.version.v60.CleanUsurperRootComponents
com.microsoft.sqlserver.jdbc.SQLServerException: Cannot resolve the collation conflict between "SQL_Latin1_General_CP1_CS_AS" and "Latin1_General_CS_AS" in the equal to operation.

Whenever you have problem with SonarQube do not forget to read the log, because only in the log you can understand the real cause of errors.

Collation problem are quite common with SonarQube, documentation is somewhat not correct, because it tell you that you need to use an Accent and Case sensitive collation, but does not specify the collation. My SonarQube 5.1 instance worked perfectly with SQL_Latin1_General_CP1_CS_AS, but sadly enough, script to upgrade db to version 6.0 fails because it is expecting Latin1_General_CS_AS.

If you install SonarQube with Sql Server, it is better to chose Latin1_General_CS_AS as collation to avoid problems.

Luckly enough you can change database collation for an existing database, the only caveat is that you need to set the database in Single User Mode to perform this operation.

USE master;  
GO  
  
ALTER DATABASE Sonar SET SINGLE_USER WITH ROLLBACK IMMEDIATE; 
GO 

ALTER DATABASE Sonar  COLLATE Latin1_General_CS_AS ;  
GO 

ALTER DATABASE Sonar SET MULTI_USER; 
GO  

Clearly you should have a backup of your database before the migration, or you will end with a corrupted database and nothing to do. So I restored the database from the backup, run the above script, restart SonarQube and try to perform the upgrade again.

image

Figure 2: Database is upgraded correctly

My instance is now running 6.0 version.

Gian Maria.

No agent could be found with the following capabilities

In TFS 2015 / VSTS new build system each task contains a series of requirements that needs to be matched by agents capabilities for the task to run. Usually you install Visual Studio in the machine with the build agent and you can schedule standard .NET builds without problem, but what happens when the build starts to evolve?

When you start creating more complex build, you can find that your agent does not meets requirements because it miss some of the required capabilities. As an example, in TFS 2015 Build I’ve added task to run Sonar Qube Analysis on my code.

Build definition with Sonar Qube analysis enabled

Figure 1: A build with SonarQube analysis enabled

Now if I queue a build manually, TFS warned me that it is not able to find a suitable agent, and if you ignore that warning and queue the build here is the result.

The build failed because no agent with required capabilities was found in the pool.

Figure 2: Build failed because no suitable agent was find

The build failed because there are no agent capable to run it. Now you should go to the Project Collection administration page and verify all the agents.

Thanks to Capabilities tab in the administration page I can see a complete list of all agent capability.

figure 3: View agents capabilities in administration page

From that picture I verified that the only agent I’ve configured missed the Java capability, so I simply remote desktop to the server, installed java on the machine and then restart the agent service (VSO-Agemt-TFS2013Preview). After agent is restarted the build rans just fine.

Thanks to the new build system, TFS Build Agents can automatically determines some known capabilities (such as Java installed) and this is atuomatically matched with all the requirements that are contained in tasks you use in the build, so TFS can choose to run the build only in the agent that satisfies requirements, and if no agent is found it warns you with a clear error.

Gian Maria.

Sonar Upgrade from 5.1 to 5.3

I really like SonarQube, but I need to admit that upgrade procedures are not the best experience I’ve had. Upgrading from 5.1 to 5.3 should be a simple task, but I had a couple of problems.

I’ve followed the official upgrade instruction: http://docs.sonarqube.org/display/SONAR/Upgrading but the first problem was my connection string to SQL Server, that needs to be changed. Original connection string was

sonar.jdbc.url=jdbc:jtds:sqlserver://localhost/Sonar;SelectMethod=Cursor;instance=sqlexpress

But it seems that with the new version of Solr it needs to be changed because jtds is not anymore used. The string was changed to

sonar.jdbc.url=jdbc:sqlserver://localhost;databaseName=Sonar;instance=sqlexpress

And the service is able to connect to my database. Then I start upgrade procedure but I got an error, and here what I found in the log

sonr Fail to execute DROP INDEX unique_schema_migrations ON schema_migrations

Luckily enough, another person got this error  and I was able to find the solution on StackOverflow Post that explained how to create the missing index. Then I restart the service, re-execute again migration procedure and everything now runs fine.

I cannot avoid to compare this experience to an upgrade of TFS on-premises, that is a fully automated procedure, a little more of next, next, next, and it is a product really more complex than SonarQube. 🙂

Gian Maria