Sonar Analysis of Python with Azure DevOps pipeline

Once you have test and Code Coverage for your build of Python code, last step for a good build is adding support for Code Analysis with Sonar/SonarCloud. SonarCloud is the best option if your code is open source, because it is free and you should not install anything except the free addin in Azure Devops Marketplace.

From original build you need only to add two steps: PrepareAnalysis onSonarCloud and Run SonarCloud analysis, in the same way you do analysis for a .NET project.

image

Figure 1: Python build in Azure DevOps

You do not need to configure anything for a standard analysis with default options, just follow the configuration in Figure 2.:

image

Figure 2: Configuration of Sonar Cloud analysis

The only tricks I had to do is deleting the folder /htmlcov created by pytest for code coverage results. Once the coverage result was uploaded to Azure Devops server I do not needs it anymore and I want to remove it from sonar analysis. Remember that if you do not configure anything special for Sonar Cloud configuration it will analyze everything in the code folder, so you will end up with errors like these:

image

Figure 3: Failed Sonar Cloud analysis caused by output of code coverage.

You can clearly do a better job simply configuring Sonar Cloud Analysis to skip those folder, but in this situation a simple Delete folder task does the job.

To avoid cluttering SonarCloud analysis with unneeded files, you need to delete any files that were generated in the directory and that you do not want to analyze, like code coverage reports.

Another important settings is the Advances section, because you should specify the file containing code coverage result as extended sonar property.

image

Figure 4: Extra property to specify location of coverage file in the build.

Now you can run the build and verify that the analysis was indeed sent to SonarCloud.

image

Figure 5: After the build I can analyze code smells directly in sonar cloud.

If you prefer, like me, YAML builds, here is the complete YAML build definition that you can adapt to your repository.

queue:
  name: Hosted Ubuntu 1604

trigger:
- master
- develop
- features/*
- hotfix/*
- release/*

steps:

- task: UsePythonVersion@0
  displayName: 'Use Python 3.x'

- bash: |
   pip install pytest 
   pip install pytest-cov 
   pip install pytest-xdist 
   pip install pytest-bdd 
  displayName: 'Install a bunch of pip packages.'

- task: SonarSource.sonarcloud.14d9cde6-c1da-4d55-aa01-2965cd301255.SonarCloudPrepare@1
  displayName: 'Prepare analysis on SonarCloud'
  inputs:
    SonarCloud: SonarCloud
    organization: 'alkampfergit-github'
    scannerMode: CLI
    configMode: manual
    cliProjectKey: Pytest
    cliProjectName: Pytest
    extraProperties: |
     # Additional properties that will be passed to the scanner, 
     # Put one key=value per line, example:
     # sonar.exclusions=**/*.bin
     sonar.python.coverage.reportPath=$(System.DefaultWorkingDirectory)/coverage.xml

- bash: 'pytest --junitxml=$(Build.StagingDirectory)/test.xml --cov --cov-report=xml --cov-report=html' 
  workingDirectory: '.'
  displayName: 'Run tests with code coverage'
  continueOnError: true

- task: PublishTestResults@2
  displayName: 'Publish test result /test.xml'
  inputs:
    testResultsFiles: '$(Build.StagingDirectory)/test.xml'
    testRunTitle: 010

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage.xml'
    reportDirectory: '$(System.DefaultWorkingDirectory)/htmlcov'
    additionalCodeCoverageFiles: '$(System.DefaultWorkingDirectory)/**'

- task: DeleteFiles@1
  displayName: 'Delete files from $(System.DefaultWorkingDirectory)/htmlcov'
  inputs:
    SourceFolder: '$(System.DefaultWorkingDirectory)/htmlcov'
    Contents: '**'

- task: SonarSource.sonarcloud.ce096e50-6155-4de8-8800-4221aaeed4a1.SonarCloudAnalyze@1
  displayName: 'Run Sonarcloud Analysis'

The only settings you need to adapt is the name of the SonarCloud connection (in this example is called SonarCloud) you can add/change in Project Settings > Service Connections.

image

Figure 6: Service connection settings where you can add/change connection with Sonar Cloud Servers.

A possible final step is adding the Build Breaker extension to your account that allows you to made your build fails whenever the Quality Gate of SonarCloud is failed.

Thanks to Azure DevOps build system, creating a build that perform tests and analyze your Python code is extremely simple.

Happy Azure Devops.

Gian Maria

Run code coverage for Python project with Azure DevOps

Creating a simple build that runs Python tests written with PyTest framework is really simple, but now the next step is trying to have code coverage. Even if I’m pretty new to Python, having code coverage in a build is really simple, thanks to a specific task that comes out-of-the-box with Azure DevOps: Publish Code Coverage.

In Azure DevOps you can create build with Web Editor or with simple YAML file, I prefer YAML but since I’ve demonstrated in the old post YAML build for Python, now I’m creating a simple build with standard Web Editor

Instead of creating a Yaml Build, this time I’m going to demonstrate a classic build: here is the core part of the build.

image

Figure 1: Core build to run tests and have code coverage uploaded to Azure DevOps

As you can see, I decided to run test with a Bash script running on Linux, here is the task configuration where I’ve added Pytest options to have code coverage during test run.

image

Figure2: Configuration of Bash script to run Pytests

The task is configured to run an inline script (1), command line (2) contains –cov options to specify Python modules I want to monitor for code coverage, then a couple of –cov-report options to have output in xml and HTML format. Finally I’ve specified the subfolder that contains the module I want to test (3) and finally I’ve configured the task con Continue on Error (4), so if some of the tests fails the build will be marked as Partially failed.

Thanks to Pytest running code coverage is just a matter of adding some options to command line

After a build finished you can find in the output how Pytest generates Code Coverage reporting, it create a file called coverage.xml then an entire directory called htmlcov that contains a report for code coverage.

image

Figure 3: Result of running tests with code coverage.

If you look at Figure 1 you can see that the build final task is a Publish Code Coverage Task, whose duty is to grab output of the Pytest run and upload to the server. Configuration is really simple, you need to choose Cobertura as Code coverage tool (the format used by Pytest) and the output of test run. Looking at output of Figure 3 you can double check that the summary file is called coverage.xml and all the report directory is in htmlcov subdirectory.

image

Figure 4: Configuration for Publish Code Coverage task.

Once you run the build, you can find Code Coverage result on the summary page, as well as Code Coverage Report published as Build artifacts, the whole configuration will take you no more than 10 minutes.

image

Figure 5: Artifacts containing code coverage reports as well as code coverage percentage are accessible from Build Summary page.

Finally you have also a dedicated tab for Code Coverage, showing the HTML summary of the report

image

Figure 6: Code coverage HTML report uploaded in a dedicated Code Coverage tab in build result

Even if the code coverage output is not perfectly formatted you can indeed immediately verify percentage of code coverage of your test.

Gian Maria.