Azure DevOps is now 150 sprints old

I remember old days when Azure DevOps was still in private preview, and yet it was really a good product, now 150 sprints passed, and the product is better than ever. Not everything is perfect, but, as users, we can expect new feature to being deployed each 3 weeks, the duration of Microsoft Sprint.

This means that now the product is 450 Weeks old, and finally we got a little nice feature that shows up news in the front page.

image

Figure 1: Widget with new feature of newest deployed sprint

This allows users to immediately being notified of new feature in their accounts, with a nice summary with key new features. In this sprint we have the new Task Assistant to help editing YAML pipelines, and many new feature, like the new agent administration ui.

image

Figure 1: New administration page in action.

The new page is more consistent with the look and feel of the rest of the service, also it shows wait time and build duration when you drill down in a pool.

As always I cannot stress out how good is to have all of your project administration tool in the Cloud, no time spent to upgrade, no time spent to verify and check backup policies, and, completely free for the first 5 users.

Gian Maria.

Troubleshoot YAML Build first run

Scenario: You create a branch in your git repository to start with a new shiny YAML Build definition for Azure Devops, you create a yaml file, push the branch in Azure Devops and Create a new Build based on that YAML definition. Everything seems ok, but when you press the run button you got and error

Could not find a pool with name Default. The pool does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.

image

Figure 1: Error running your new shiny pipeline

Ok this is frustrating and following the link gives you little clue on what really happened. The problem is that, with the new editor experience, when you navigate to the pipeline page, all you see is the editor of YAML build and nothing more.

image

Figure 2: New Editor page of YAML pipeline, advanced editor and nothing more.

The new editor is fantastic, but it somewhat hides standard configuration parameters page, where the default branch can be set. As you can see from Figure 2 you can specify pool name (default) and triggers directly in YAML build so you think that this is everything you need, but there is more. Clicking on the three buttons in the right upper corner you can click on the trigger menu to open the old editor.

image

Figure 3: Clicking on the Triggers menu item will bring on the old UI

This is where the YAML pipeline experience still needs some love, you are surely puzzled why you need to click triggers menu item if you already specified triggers directly in the YAML definition, but the reason is simple, it will open the old pipeline editor page.

The new editor page with YAML editor is fantastic, but you should not forget that there are still some parameters, like default branch, that are editable from the old interface

Trigger page is not really useful, it only gives you the ability to override the YAML configuration, but the important aspect is that we can now access the first tab of the YAML configuration to change default branch.

image

Figure 4: Trigger page is not useful, but now we can access default configuration for the pipeline.

image

Figure 5: Default configuration tab where you can edit default branch

In Figure 5 you can now understand what went wrong, the wizard created my pipeline using master as default branch, but clearly my buid YAML file does not exists in master, but exists only in my feature branch. Yust change the default build to the branch that contains your build definition file, save and queue again; now everything should word again.

This trick works also when you got errors not being authorized to use endpoints, like sonar endpoint, nuget endpoint etc.

Happy YAML Building experience.

Gian Maria.

YAML Build in Azure DevOps

I’ve blogged in the past about YAML build in azure DevOps, but in that early days, that kind of build was a little bit rough and many people still preferred the old build based on visual editing in a browser. One of the main complaint was that the build was not easy to edit and there were some glitch, especially when it is time to access external services.

After months from the first version, the experience is really improved and I strongly suggest you to start trying to migrate existing build to this new system, to take advantage of having definition of build directly in the code, a practice that is more DevOps oriented and that allows you to have different build tasks for different branches.

YAML Build is now a first class citized in Azure DevOps and it is time to plan switching old build to the new engine.

You can simply start with an existing build, just edit it, select one of the phases (or the entire process) then press View YAML button to grab generated YAML build.

SNAGHTML2fecbc

Figure 1: Generate YAML definition from an existing build created with the standard editor

Now you can simply create a yaml file in any branch of your repository, paste the content in the file, commit to the branch and create a new build based on that file. I can clearly select not only AzDO repositories, but I can build also GitHub and GitHub enterprise

image

Figure 2: I can choose GitHub as source repository, not only azure repos

Then I can choose the project searching in all the project I have access to with my access token used to connect GitHub

image

Figure 3: Accessing GitHub repositories is simple, once you connected the acount with an access token AzDO can search in repositories

Just select a repository and select the option Existing Azure Pipelines, if you are starting from scratch you can create a starter pipeline.

image

Figure 4: Choose the option to use an existing pipeline.

You are ready to go, just choose branch and yaml file and the game is done.

image

Figure 5: You can directly specify the build file in the pipeline creation wizard.

Converting an existing build pipeline to YAML it is matter of no more than 10 minutes of your time.

Now you can simply run and edit your build directly from the browser, the whole process took no more than 10 minutes, including connecting my AzDO account to GitHub

image

Figure 6: Your build is ready to run inside Azure DevOPS.

Your build is now ready to run without any problem. If you specified triggers as in Figure 6 you can just push to the repository to have the build automatically kicks in and being executed. You can also directly edit the build in the browser, and pressing the Run button (Figure 6) you can trigger a run of the build without the need to push anything.

But the coolness of actual YAML build editor starts to shine when you start editing your build in the web editor, because you have intellisense, as you can see in Figure 7.

image

Figure 7: YAML build web editor has now intellisense.

As you can see the YAML build editor allows you to edit with full intellisense support, if you want to add a task, you can simply start writing task followed by a semicolon and the editor will suggest you all the tasks-available. When it is time to edit properties, you have intellisense and help for each task parameters, as well as help for the entire task. This is really useful because it immediately spots deprecated tasks (Figure 9)

image

Figure 8: All parameters can be edited with fully intellisense and help support

image

Figure 9: Helps gives you helps for the deprecation of old task that should not be used.

With the new web editor with intellisense, maintaining a YAML build is now easy and not more difficult than the standard graphical editor.

Outdated tasks are underlined in green, so you can immediately spot where the build definition is not optimal, as an example if I have a task that have a new version, the old version is underlined in green, and the intellisense suggests me that the value is  not anymore supported. This area still need some more love, but it works quite well.

There is no more excuses to still use the old build definition based on web editor, go and start converting everything to YAML definition, your life as bulid maintainer will be better :)

Gian Maria.

Change Work Item Type in a fresh installation of Azure DevOps server

If you want to use Azure DevOps I strongly suggest you to use cloud version https://dev.azure.com, but if you really need to have it on premise, you can install Team Foundation Server, now renamed to Azure DevOps Server.

One of the most waited feature for the on-premise version is the ability to change work item Type and to move work item between project, a feature present in Azure DevOps Server, but that needs a complete disable of Reporting Services to work, as I discussed in an old Post.

In that very post I had a comment telling me that after a fresh installation of Azure DevOps Server, even if he did not configured reporting services, the option to move a Work Item Between Team Project is missing, as well as the option to change Work Item Type. The problem is, until you do not explicitly disable reporting on the TFS instance those two options are not available. This is probably due to avoiding using these feature, then subsequently enable Reporting ending with incorrect data in the warehouse.

First of all we need to clarify some radical change in Azure DevOps 2019 respect former version TFS 2018.

Azure DevOps Server has a couple of different type of Project Collection, one is the classic one with an XML process, the new one is the one based on process inheritance.

image 

Figure 1: Different type of Project Collection in Azure Devops

If you check Figure 1, you can verify that an  inheritance based project collection does not use with Sql Server Anlysis services and reporting; thus you can always change Team Project or type because reporting is not used in these type of collection. As you can see in Figure 2, if I have a project collection based on Inheritance model, I can change work item type even if Reporting is configured.

image

Figure 2: Project collection based on inheritance model are not affected by reporting services configuration.

If you instead create a new collection using the old XML process model, even if you have not configured reporting services, the ability to Change Type or Move Between team project is not present. This happens because, even if you had not configured reporting, you must explicitly disable that feature, to prevent it to be reactivated in the future and have some erratic report.

image

Figure 3: Even if you did not configure reporting for Azure DevOps server, the option to change Team Project and Change type are not available

To enable Move Between Team Project and Change Work Item Type you really need to explicitly disable reporting, as shown in Figure 3 and Figure 4

If you disable reporting the system is warning you that the reporting options could not be enabled anymore.

image

Figure 4: A confirmation dialog warn that disabling Reporting is an option that cannot be undone

As soon reporting is disabled, you can change Type and Move to other Team Project.

image

Figure 5: When reporting is explicitly disabled, you immediately have the two options enabled.

Happy Azure Devops.

Gian Maria.

Sonar Analysis of Python with Azure DevOps pipeline

Once you have test and Code Coverage for your build of Python code, last step for a good build is adding support for Code Analysis with Sonar/SonarCloud. SonarCloud is the best option if your code is open source, because it is free and you should not install anything except the free addin in Azure Devops Marketplace.

From original build you need only to add two steps: PrepareAnalysis onSonarCloud and Run SonarCloud analysis, in the same way you do analysis for a .NET project.

image

Figure 1: Python build in Azure DevOps

You do not need to configure anything for a standard analysis with default options, just follow the configuration in Figure 2.:

image

Figure 2: Configuration of Sonar Cloud analysis

The only tricks I had to do is deleting the folder /htmlcov created by pytest for code coverage results. Once the coverage result was uploaded to Azure Devops server I do not needs it anymore and I want to remove it from sonar analysis. Remember that if you do not configure anything special for Sonar Cloud configuration it will analyze everything in the code folder, so you will end up with errors like these:

image

Figure 3: Failed Sonar Cloud analysis caused by output of code coverage.

You can clearly do a better job simply configuring Sonar Cloud Analysis to skip those folder, but in this situation a simple Delete folder task does the job.

To avoid cluttering SonarCloud analysis with unneeded files, you need to delete any files that were generated in the directory and that you do not want to analyze, like code coverage reports.

Another important settings is the Advances section, because you should specify the file containing code coverage result as extended sonar property.

image

Figure 4: Extra property to specify location of coverage file in the build.

Now you can run the build and verify that the analysis was indeed sent to SonarCloud.

image

Figure 5: After the build I can analyze code smells directly in sonar cloud.

If you prefer, like me, YAML builds, here is the complete YAML build definition that you can adapt to your repository.

queue:
  name: Hosted Ubuntu 1604

trigger:
- master
- develop
- features/*
- hotfix/*
- release/*

steps:

- task: UsePythonVersion@0
  displayName: 'Use Python 3.x'

- bash: |
   pip install pytest 
   pip install pytest-cov 
   pip install pytest-xdist 
   pip install pytest-bdd 
  displayName: 'Install a bunch of pip packages.'

- task: SonarSource.sonarcloud.14d9cde6-c1da-4d55-aa01-2965cd301255.SonarCloudPrepare@1
  displayName: 'Prepare analysis on SonarCloud'
  inputs:
    SonarCloud: SonarCloud
    organization: 'alkampfergit-github'
    scannerMode: CLI
    configMode: manual
    cliProjectKey: Pytest
    cliProjectName: Pytest
    extraProperties: |
     # Additional properties that will be passed to the scanner, 
     # Put one key=value per line, example:
     # sonar.exclusions=**/*.bin
     sonar.python.coverage.reportPath=$(System.DefaultWorkingDirectory)/coverage.xml

- bash: 'pytest --junitxml=$(Build.StagingDirectory)/test.xml --cov --cov-report=xml --cov-report=html' 
  workingDirectory: '.'
  displayName: 'Run tests with code coverage'
  continueOnError: true

- task: PublishTestResults@2
  displayName: 'Publish test result /test.xml'
  inputs:
    testResultsFiles: '$(Build.StagingDirectory)/test.xml'
    testRunTitle: 010

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage'
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/coverage.xml'
    reportDirectory: '$(System.DefaultWorkingDirectory)/htmlcov'
    additionalCodeCoverageFiles: '$(System.DefaultWorkingDirectory)/**'

- task: DeleteFiles@1
  displayName: 'Delete files from $(System.DefaultWorkingDirectory)/htmlcov'
  inputs:
    SourceFolder: '$(System.DefaultWorkingDirectory)/htmlcov'
    Contents: '**'

- task: SonarSource.sonarcloud.ce096e50-6155-4de8-8800-4221aaeed4a1.SonarCloudAnalyze@1
  displayName: 'Run Sonarcloud Analysis'

The only settings you need to adapt is the name of the SonarCloud connection (in this example is called SonarCloud) you can add/change in Project Settings > Service Connections.

image

Figure 6: Service connection settings where you can add/change connection with Sonar Cloud Servers.

A possible final step is adding the Build Breaker extension to your account that allows you to made your build fails whenever the Quality Gate of SonarCloud is failed.

Thanks to Azure DevOps build system, creating a build that perform tests and analyze your Python code is extremely simple.

Happy Azure Devops.

Gian Maria