Strange Error uploading artifacts in Azure DevOps pipeline

I have a pipeline that worked perfectly for Years, but yesterday a build failed while uploading artifacts, I queued it again and it still failed, so it does not seems to be an intermittent error (network could be unreliable). I was really puzzled because from the last good build we changed 4 C# files, nothing really changed that can justify the failing and also we have no network problem that can justify problem uploading artifacts to Azure DevOps.

Well, the error was also not telling me anything, it is a simple statement that file upload failed and since it told me that there were a retry, definitely this is not an intermittent network error.

SNAGHTML156bf49

Figure 1: Upload file error, nothing is suggesting me the real error

Digging deeper in the execution logs I found something really strange, an error after a GetTempFileName() function call.

image

Figure 2: The real error reported in the detail log

Ok, this is definitely weird, it seems that a task is calling GetTempFileName() method, but the call failed with a strange error “The file already exists”; digging around in the internet it seems that the error can be generated by a temp directory with too much files (someone claims this limit to be 65535).

Temp directories tends to grow really big and if you have plenty of space in your HDD, usually they are not cleared, but you can incur in problems if the number of files becomes too high.

The trick part is knowing which is the real temp directory used by the agent that runs the build, so I’ve added an Execute Command task to simply echo “%TEMP% variable.

image

Figure 3: Dumping temp variable inside a build

Checking the build server, I verified that temp folder contains more than 100k files for several gigabytes of space. After deleting all those temp files, the build started working again and the team was happy again. Yesterday was a strange day, because this is not the weirdest bug I had to solve :P. (another post will follow).

Gian Maria.

Azure DevOps Pipeline template steps and .NET Core 3 local tools

I’m a strong fan of Azure DevOps templates for pipelines because it is a really good feature to both simplify Pipeline authoring and avoid proliferation of too many way to do the same things. In some of my previous examples I’ve always used a template that contains full Multi Stage pipeline definition, this allows you to create a new pipeline with easy, reference repository with the template, choose right template, set parameters and you are ready to go.

Using a template file that contains all stages allows you to define in a single place an entire pipeline to reuse with simple parameters definition

Today my dear friend Giulio Vian told me that he was investigating the use of a cool feature of .NET core 3.0, called Local Tools. Basically with Local Tools you are able to create a special file called dotnet-tools.json that contains all the tools you need for your project. Since I’m an heavy fan of GitVersion, it seems to me standard to include in every project such file with the actual version of GitVersion used in my project. Here is an example of the file.

{
  "version": 1,
  "isRoot": true,
  "tools": {
    "gitversion.tool": {
      "version": "5.2.4",
      "commands": [
        "dotnet-gitversion"
      ]
    }
  }
}

Once you have this file in place, you can simply issue a dotnet tool restore command and all referenced tools will be automatically installed locally and ready to use. This makes me extra simple to use GitVersion in my pipelines, because a simple dotnet tool restore will make GitVersion available on my pipeline (given that I’ve previously created a dotnet-tools.json in my project).

To experiment this new feature I want to give you another approach in using Azure DevOps templates, shared steps. Lets have a look at this template file:

parameters:
  buildName: 'Specify name'
  dotNetCoreVersion: '2.2.301'

steps:
  - task: DotNetCoreInstaller@2
    displayName: 'Use .NET Core sdk ${{parameters.dotNetCoreVersion}}'
    inputs:
      version: ${{parameters.dotNetCoreVersion}}

  - task: DotNetCoreCLI@2
    displayName: 'install if needed dotnet gitversion tool'
    inputs:
      command: 'custom'
      custom: 'tool'
      arguments: 'restore'
  
  - script: |
      dotnet tool run dotnet-gitversion $(Build.Repository.LocalPath) /output buildserver
    name: Run_dotnet_gitversion

  - powershell: |
      Write-Host "##vso[build.updatebuildnumber]${{parameters.buildName}}-$env:GITVERSION_FULLSEMVER"

      $var = (gci env:*).GetEnumerator() | Sort-Object Name
      $out = ""
      Foreach ($v in $var) {$out = $out + "`t{0,-28} = {1,-28}`n" -f $v.Name, $v.Value}

      write-output "dump variables on $env:BUILD_ARTIFACTSTAGINGDIRECTORY\test.md"
      $fileName = "$env:BUILD_ARTIFACTSTAGINGDIRECTORY\test.md"
      set-content $fileName $out

      write-output "##vso[task.addattachment type=Distributedtask.Core.Summary;name=Environment Variables;]$fileName"
    name: Update_build_number

  - powershell: |
      echo "[task.setvariable variable=GITVERSION_ASSEMBLYSEMVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMVER)"
      echo "[task.setvariable variable=GITVERSION_ASSEMBLYSEMFILEVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMFILEVER)"
      echo "[task.setvariable variable=GITVERSION_SHA;isOutput=true]$(GITVERSION.SHA)"
      echo "[task.setvariable variable=GITVERSION_FULLSEMVER;isOutput=true]$(GITVERSION.FULLSEMVER)"
      echo "##vso[task.setvariable variable=GITVERSION_ASSEMBLYSEMVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMVER)"
      echo "##vso[task.setvariable variable=GITVERSION_ASSEMBLYSEMFILEVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMFILEVER)"
      echo "##vso[task.setvariable variable=GITVERSION_SHA;isOutput=true]$(GITVERSION.SHA)"
      echo "##vso[task.setvariable variable=GITVERSION_FULLSEMVER;isOutput=true]$(GITVERSION.FULLSEMVER)"
    name: 'SetGitVersionVariables'

It is a quite long template but basically it begins with usual parameters section, followed by a series of steps, not an entire multi stage definition. The sequence of steps are used to: runs dotnet tool restore, run GitVersion and finally does some PowerShell dumping of the variables. This kind of template was more similar to a Task Group because it is basically just a sequence of steps with parameters.

With this approach you are defining small pieces of an entire pipeline, allowing for a more granular reuse. This other template contains steps to build and run tests for a solution in .NET core.

parameters:
  dotNetCoreVersion: '3.1.201'
  buildConfiguration: release
  solution: ''
  continueOnTestErrors: true
  GitVersionFullSemVer: ''
  GitVersionAssemblyVer: ''
  GitVersionAssemblySemFileVer: ''
  SkipInstallDotNetCore: false

steps:

  - task: DotNetCoreInstaller@2
    displayName: 'Use .NET Core sdk ${{parameters.dotNetCoreVersion}}'
    inputs:
      version: ${{parameters.dotNetCoreVersion}}
    condition: ne(${{parameters.SkipInstallDotNetCore}}, 'true')

  - task: DotNetCoreCLI@2 
    displayName: 'dotnet restore'
    inputs:
      command: restore
      projects: '${{parameters.solution}}'
      feedsToUse: config
      nugetConfigPath: src/NuGet.Config

  - task: DotNetCoreCLI@2
    displayName: 'dotnet build'
    inputs:
      command: build
      projects: '${{parameters.solution}}'
      configuration: '${{parameters.buildConfiguration}}'
      arguments: /p:AssemblyVersion=${{parameters.GitVersionAssemblyVer}} /p:FileVersion=${{parameters.GitVersionAssemblySemFileVer}} /p:InformationalVersion=${{parameters.GitVersionAssemblyVer}}_$(Build.SourceVersion)

  - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test
      nobuild: true
      projects: '${{parameters.solution}}'
      arguments: '--configuration ${{parameters.buildConfiguration}} --collect "Code coverage" --logger trx' 
    continueOnError: ${{parameters.continueOnTestErrors}}

This steps declare as parameters some of semVer numbers extracted by GitVersion, so this sequence of steps are based on the fact that you should have run GitVersion in some preceding steps. Finally I’ve the last template that is used to publish with NuGet.

parameters:
  dotNetCoreVersion: '3.1.201'
  buildConfiguration: release
  nugetProject: ''
  GitVersionFullSemVer: ''
  GitVersionAssemblyVer: ''
  GitVersionAssemblySemFileVer: ''
  SkipInstallDotNetCore: false

steps:

  - task: DotNetCoreInstaller@2
    displayName: 'Use .NET Core sdk ${{parameters.dotNetCoreVersion}}'
    inputs:
      version: ${{parameters.dotNetCoreVersion}}
    condition: ne(${{parameters.SkipInstallDotNetCore}}, 'true')

  - task: DotNetCoreCLI@2
    displayName: NuGet Pack
    inputs:
      command: custom
      custom: pack
      projects: ${{parameters.nugetProject}}
      arguments: -o "$(Build.ArtifactStagingDirectory)\NuGet" -c ${{parameters.buildConfiguration}} /p:PackageVersion=${{parameters.GitVersionFullSemVer}} /p:AssemblyVersion=${{parameters.GitVersionAssemblyVer}} /p:FileVersion=${{parameters.GitVersionAssemblySemFileVer}} /p:InformationalVersion=${{parameters.GitVersionAssemblyVer}}_$(Build.SourceVersion)

  - task: NuGetCommand@2
    displayName: NuGet Push
    inputs:
      command: push
      packagesToPush: '$(Build.ArtifactStagingDirectory)\NuGet\*.nupkg'
      nuGetFeedType: internal
      publishVstsFeed: '95a01998-aa90-433c-8077-41da981289aa'
    continueOnError: true

Now that I have these three distinct template files in a repository of my Azure DevOps account I can refer them to pipelines in another repository.

Here is the real pipeline file of final project that is actually using all those three steps template defined in another repository.

trigger:
  branches:
    include:
      - master
      - develop
      - release/*
      - hotfix/*
      - feature/*

resources:
  repositories:
  - repository: templatesRepository
    type: git
    name: Jarvis/BuildScripts
    ref: refs/heads/develop

jobs:

- job: net_build_test

  pool:
    name: '$(Pool)'
    demands:
      - vstest
      - msbuild
      - visualstudio

  steps:
  - template: 'steps/GitVersionDotnetCoreLocal.yml@templatesRepository' # Template reference
    parameters:
      dotNetCoreVersion: '3.1.201'
      buildName: 'License Manager'
  
  - template: 'steps/BuildAndTestCore.yml@templatesRepository' # Template reference
    parameters:
      dotNetCoreVersion: '3.1.201'
      solution: 'src/LicenseManager.sln'
      SkipInstallDotNetCore: true
      GitVersionFullSemVer: '$(GITVERSION.FULLSEMVER)'
      GitVersionAssemblyVer: '$(GITVERSION.ASSEMBLYSEMVER)'
      GitVersionAssemblySemFileVer: '$(GITVERSION.ASSEMBLYSEMFILEVER)'

  - template: 'steps/PublishNuget.yml@templatesRepository' # Template reference
    parameters:
      dotNetCoreVersion: '3.1.201'
      nugetProject: 'src/LicenseManager.Client/LicenseManager.Client.csproj'
      GitVersionFullSemVer: '$(GITVERSION.FULLSEMVER)'
      GitVersionAssemblyVer: '$(GITVERSION.ASSEMBLYSEMVER)'
      GitVersionAssemblySemFileVer: '$(GITVERSION.ASSEMBLYSEMFILEVER)'
      SkipInstallDotNetCore: true

As you can see, with steps template I decide on final build how many stage I need, in this example I’m perfectly confortable with a single stage. The cool part of this approach is that I can mix standard steps and steps template files, giving me more flexibility in how the pipeline is constructed. Clearly this pipeline is more complex than one that use a full template file, because we need to pass parameter to every template. Running the pipeline gives you a standard run.

image

Figure 1: Steps are expanded during execution.

As you can verify from Figure 1 all steps templates are expanded in basic steps, allowing you to verify the output of every single step. Thanks to local tool feature I can simply run dotnet tool restore to have GitVersion automatically installed

image

Figure 2: Restoring tooling with dotnet tool restore automatically restore gitversion

This will greatly simplify agent requirements, because all requirements are automatically restored by the pipeline.

Gian Maria.

Azure DevOps pipeline template for build and release .NET core project

Some days ago I’ve blogged on how to release projects on GitHub with actions, now it is time to understand how to do a similar thing in Azure DevOps to build / test / publish a .NET core library with nuget. The purpose is to create a generic template that can be reused on every general that needs to build an utility dll, run test and publish to a Nuget feed.

The ability to create template pipeline in Azure DevOps is a great opportunity to define a standard way to build / test /  deploy projects in your organization

Everything starts with a dedicated repository where I store a single build template file to create a MultiStage pipeline, where the first stage is a .NET core build test, and the second stage is publishing with nuget. Such simple build could be done with a single stage, but creating it with MultiStage gives me the opportunity to explain some interesting aspect of Azure DevOps pipelines.

Everything starts with parameters declaration.

parameters:
  buildName: 'Specify name'
  solution: ''
  buildPlatform: 'ANY CPU'
  buildConfiguration: 'Release'
  nugetProject: ''
  nugetProjectDir: ''
  dotNetCoreVersion: '2.2.301'
  pool: 'Default'
  nugetPublish: true

Every single parameter can have a default option, and can be overridden, after parameter first stage starts, build and test .NET core project.

jobs:

- job: 'Build_and_Test'
  pool:
    name: ${{parameters.pool}}

  steps:
  - task: DotNetCoreInstaller@2
    displayName: 'Use .NET Core sdk ${{parameters.dotNetCoreVersion}}'
    inputs:
      version: ${{parameters.dotNetCoreVersion}}

  - task: DotNetCoreCLI@2
    displayName: 'install if needed dotnet gitversion tool'
    inputs:
      command: 'custom'
      custom: 'tool'
      arguments: 'update GitVersion.Tool --tool-path $(Agent.ToolsDirectory)/gitversion/5.1.3 --version 5.1.3'
  
  - script: |
      $(Agent.ToolsDirectory)/gitversion/5.1.3/dotnet-gitversion $(Build.Repository.LocalPath) /output buildserver

  - powershell: |
      Write-Host "##vso[build.updatebuildnumber]${{parameters.buildName}}-$env:GITVERSION_FULLSEMVER"

      $var = (gci env:*).GetEnumerator() | Sort-Object Name
      $out = ""
      Foreach ($v in $var) {$out = $out + "`t{0,-28} = {1,-28}`n" -f $v.Name, $v.Value}

      write-output "dump variables on $env:BUILD_ARTIFACTSTAGINGDIRECTORY\test.md"
      $fileName = "$env:BUILD_ARTIFACTSTAGINGDIRECTORY\test.md"
      set-content $fileName $out

      write-output "##vso[task.addattachment type=Distributedtask.Core.Summary;name=Environment Variables;]$fileName"

  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: restore
      projects: '${{parameters.solution}}'
      feedsToUse: config
      nugetConfigPath: src/NuGet.Config

  - task: DotNetCoreCLI@2
    displayName: 'dotnet build'
    inputs:
      command: build
      projects: '${{parameters.solution}}'
      configuration: '$(BuildConfiguration)'
      arguments: /p:AssemblyVersion=$(GITVERSION.ASSEMBLYSEMVER) /p:FileVersion=$(GITVERSION.ASSEMBLYSEMFILEVER) /p:InformationalVersion=$(GITVERSION.SHA)

  - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test
      nobuild: true
      projects: '${{parameters.solution}}'
    continueOnError: true

  - powershell: |
      echo "[task.setvariable variable=GITVERSION_ASSEMBLYSEMVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMVER)"
      echo "[task.setvariable variable=GITVERSION_ASSEMBLYSEMFILEVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMFILEVER)"
      echo "[task.setvariable variable=GITVERSION_SHA;isOutput=true]$(GITVERSION.SHA)"
      echo "[task.setvariable variable=GITVERSION_FULLSEMVER;isOutput=true]$(GITVERSION.FULLSEMVER)"
      echo "##vso[task.setvariable variable=GITVERSION_ASSEMBLYSEMVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMVER)"
      echo "##vso[task.setvariable variable=GITVERSION_ASSEMBLYSEMFILEVER;isOutput=true]$(GITVERSION.ASSEMBLYSEMFILEVER)"
      echo "##vso[task.setvariable variable=GITVERSION_SHA;isOutput=true]$(GITVERSION.SHA)"
      echo "##vso[task.setvariable variable=GITVERSION_FULLSEMVER;isOutput=true]$(GITVERSION.FULLSEMVER)"
    name: 'SetGitVersionVariables'

You can recognize in this script many of the techniques already discussed in previous GitHub Action post, it is just declined for Azure DevOps. The main difference is that Actions are directed toward a simple way to execute a “script” composed by a series of commandline istruction and tasks while Pipelines are more structured to create a workflow, but everything is really similar.

Since this pipeline will run on Windows, I can simply use PowerShell task to execute inline script. The most peculiar part of the script is the last PowerShell script, that contains a series of Pipeline commands echoing ##vso in the output stream. The purpose of that step is to save some variable values to reuse in subsequent stages. This is a killer feature, in this example I runs GitVersion on first stage only, then pass all the output to be reused by subsequent Stages.

The ability to pass variable values between stages opens a wide range of opportunities, where you can run special tools on special agents, then reuse output in all other stages.

This is really handy if you need to execute subsequent stages, in different operating system / environment and you want to simply reuse some variable values that was calculate in some previous stage. Suppose you have a tool that runs only on windows, you can run in a stage, then reuse output in subsequent stages that runs in linux.

Publish stage is really simple, the only really interesting part is the declaration.

- job: 'Pack_Nuget'
  dependsOn: 'Build_and_Test'
  condition: eq(${{parameters.nugetPublish}}, true)

  pool:
    name: ${{parameters.pool}}

  variables:
    GITVERSION_ASSEMBLYSEMVER: $[ dependencies.Build_and_Test.outputs['SetGitVersionVariables.GITVERSION_ASSEMBLYSEMVER'] ]
    GITVERSION_ASSEMBLYSEMFILEVER: $[ dependencies.Build_and_Test.outputs['SetGitVersionVariables.GITVERSION_ASSEMBLYSEMFILEVER'] ]
    GITVERSION_SHA: $[ dependencies.Build_and_Test.outputs['SetGitVersionVariables.GITVERSION_SHA'] ]
    GITVERSION_FULLSEMVER: $[ dependencies.Build_and_Test.outputs['SetGitVersionVariables.GITVERSION_FULLSEMVER'] ]

The stage starts with a name and a dependency declaration on previous Build_and_test stage, this implies that this stage can run only if the previous stage run successfully. The execution is also dependent on a parameter called nugetPublish, that should be true for this stage to execute. This allows the pipeline that uses this template to choose if publish stage should run.

The ability to conditionally execute stages allows for complex workflow execution, where each stage can decide on following stages execution.

Following the declaration we can find a variables section, where I actually load variable from previous stage in this stage. In this specific example I’m retrieving all GitVersion output value that I need to build NugetPackage.

The stage ends with standard pack and publish of NugetPackage, using SemVer numbers that were passsed from previous stage.

steps:

  - task: DotNetCoreInstaller@2
    displayName: 'Use .NET Core sdk ${{parameters.dotNetCoreVersion}}'
    inputs:
      version: ${{parameters.dotNetCoreVersion}}

  - powershell: |
      echo "GITVERSION_ASSEMBLYSEMVER $(GITVERSION_ASSEMBLYSEMVER)"
      echo "GITVERSION_ASSEMBLYSEMFILEVER $(GITVERSION_ASSEMBLYSEMFILEVER)"
      echo "GITVERSION_SHA $(GITVERSION_SHA)"
      echo "GITVERSION_FULLSEMVER $(GITVERSION_FULLSEMVER)"
    name: 'Dumpvariables'

  - task: DotNetCoreCLI@2
    displayName: NuGet Pack
    inputs:
      command: custom
      custom: pack
      projects: ${{parameters.nugetProject}}
      arguments: -o "$(Build.ArtifactStagingDirectory)\NuGet" -c ${{parameters.BuildConfiguration}} /p:PackageVersion=$(GITVERSION_FULLSEMVER) /p:AssemblyVersion=$(GITVERSION_ASSEMBLYSEMVER) /p:FileVersion=$(GITVERSION_ASSEMBLYSEMFILEVER) /p:InformationalVersion=$(GITVERSION_SHA)

  - task: NuGetCommand@2
    displayName: NuGet Push
    inputs:
      command: push
      packagesToPush: '$(Build.ArtifactStagingDirectory)\NuGet\*.nupkg'
      nuGetFeedType: internal
      publishVstsFeed: '95a01998-aa90-433c-8077-41da981289aa'
    continueOnError: true

Once this template file is checked in in AzureDevOps repository, you can refer it from another project in the same Organization. This is the real power of templates, I wrote the definition one time in a dedicated repository and every other project that needs to declare a pipeline to build / test / publish can simply refers to this template. With a few lines of YAML you can create a pipeline for your new project.

trigger:
  branches:
    include:
      - master
      - develop
      - release/*
      - hotfix/*
      - feature/*

resources:
  repositories:
  - repository: templatesRepository
    type: git
    name: Jarvis/BuildScripts
    ref: refs/heads/develop

jobs:

- template: 'NetStandardTestAndNuget.yaml@templatesRepository'
    
  parameters:
    buildName: 'LicenseManager'
    solution: 'src/LicenseManager.sln'
    pool: '$(pool)'
    dotNetCoreVersion: '3.1.100'
    nugetPublish: true
    nugetProject: 'src/LicenseManager.Core/LicenseManager.Core.csproj'

Look at how simple it is, just define triggers, add repository that contains the build script in the resources section, and simple populate the parameters, and, BAM, your project has a pipeline for build / test / publish.

The ref parameter of reference section allows you to choose which branch use to grab script template, in this project I want the latest trunk version, so I’ve choose develop, other project can stay on master to have a better stable version.

The template once uses an old version of task of mine to perform GitVersion, that is become really obsolete and it is not useful to maintain anymore. I’ve decided to upgrade the template to use dotnet-gitversion command line tool, I’ve upgraded the template in a feature branch, using a project as test, then I’ve merged in develop and when I’ll finally close it on master, every project that uses this template, will use the new definition, without any user intervention.

Thanks to template I can upgrade the template in dedicated branch, test with actual project, then promote the upgrade through standard develop, release and master branch to automatically upgrade pipeline of all projects that uses this template.

How cool is that.

It is actually superfluous telling you how important  is to have an automatic build / test pipeline, as an example it seems that yesterday night I’ve broke the test Smile, shame on me.

image

Figure 1: Build results showing me that some test fails

The nice aspect of Azure DevOps pipeline is that they have a dedicated section to examine test failures, that gives me immediate insight on what went wrong. It seems that I’ve messed something in exception handling.

image

Figure 2: Dedicated pane to view test result.

Actually Azure DevOps pipelines are more complex than GitHub actions, but they can also solve more complex problems and are (at date of today) probably better suited for an enterprise, especially with the ability to define template to make a standard in how to build projects of the company. Another key value is the ability to immediately explore failed tests and code coverage for your build, not to mention multi stage pipeline to create complex build / release workflows.

Actually we have a superposition between Azure DevOps and GitHub pipelines, both of them now owned by Microsoft. My advice is just look at what are the capabilities as today and choose what better suites you.

Remember also that you can easily use Azure DevOps pipeline to build a GitHub project, just point the build to a GitHub repository after you connected your GitHub organization  / account to Azure DevOps. The only limitation is that build template file should still reside on AzureDevOps account (a limitation that probably will expire soon).

Remember that you can freely use Azure DevOps pipeline to build GitHub projects without any problem, just choose the product that better suites your need.

Gian Maria.

Azure DevOps YAML pipeline authorization problem

It could happen, sometimes, that when you create a pipeline in Azure Devops at first run you got the following error.

##[error]Pipeline does not have permissions to use the referenced pool(s) Default. For authorization details, refer to https://aka.ms/yamlauthz.

There are more than one kind of this error, the most common one is the build using some external resource that requires authorization, but in this specific error message, pipeline has no permission to run on default pool.

If you look at documentation, you can verify that you are expected to be presented with a nice button that allows you to authorize the pipeline with current pool, but sadly enough, sometime you have no button.

The reason is probably the inability of the system to determine the pool used by the build during authorization phase, this happens as an example when the poll is passed as variable to the pipeline. Look at the following definition

image

Figure 1: Pool specified as variable in the definition.

As you can see in Figure 1 pool name is specified through a standard variable, this allows the user to choose the pool with manual queue, while leaving standard continuous integration to target a pool chosen in the definition.

This kind of flexibility is necessary, as an example, to avoid priority problems. We usually have a dedicated pool with fast machines and left one pipeline license free so we can schedule a build in pool fast to have it being executed immediately.

How can you solve the problem? The simplest solution is going to the administration page of the project, select the pool you want to use for the build and finally grant permission to every pipeline.

image

Figure 2: Giving access to all pipelines to a pool solve authorization problems

If you prefer being able to give permission to single pipelines, the only available approach is editing the build, pushing yaml definition with a specific pool (not using a variable) and wait for the build to fail. If the pool is static you should have a nice button to authorize the build. Once authorized, you can edit the definition again with variable pool and the game is done. Remember to do the game for each pool that the build should use.

Gian MAria.

Windows Docker Container for Azure Devops Build agent

Thanks to Docker Compose, I can spin off an agent for Azure Devops in mere seconds (once you have all the images). Everything I need is just insert the address of my account a valid token and an agent is ready.

With .NET core everything is simple, because we have a nice build task that automatically install .NET Core SDK in the agent, the very same for node.js. This approach is really nice, because it does not require to preinstall too much stuff in your agent, everything is downloaded and installed on the fly when a build needs that specific tooling.

.NET core Node.Js and other SDK can be installed directly from build definition, without any requirement on the machine where the agent is running

In my projects I need also to build solution based on Full Framework SDK and clearly I want to use the very same Docker approach for Agents. Everything I need is a machine with Windows Server2019 with docker enabled and looking in the internet for needed Dockerfile or pre build images on public registry.

I’ve started with a nice article that explain how to install Build Tools inside a container. This articles gives you a dockerfile that basically is doing everything, just docker build and the container is ready to be used.

The only real modification I need is excluding most of the packages I’m not using in my project; I’ve also changed base image to use .NET Framework 4.8 and not 4.7.2.

image

Figure 1: RUN instruction to install build tools inside a Docker Image.

Be prepared to wait a litlle bit for the image to be built, because it downloads all the packages from network and then install into the machine. To build the container image from dockerfile you can simply type.

docker build -t buildtools2019:1.0 -m 4GB .

Now I need to modify my Azure Devops Agent dockerfile built following MSDN instructions  and change base image to FROM buildtools2019:1.0 this will base the agent image on the new image built with all .NET sdk for full framework and all build tools. With this image the agent can build Full Framework projects.

Creating a Docker Images with all needed tools for full framework SDK was really easy.

The other challenge on creating your build agent in Docker is having all the requirements in other docker images, for SQL Server I’ve rebuild the container on top of Windows Server Core 2019 image and it runs just fine, for MongoDb I’ve enabled experimental feature to mix Linux and Windows container so I’ve used a Linux based mongodb image with no problem.

My only remaining dependency was ElasticSearch, because I’m using an old version, 2.4 and I’d like to have that image running in Windows Container. It is not difficult to find some nice guy that published dockerfile for Elasticsearch, here https://github.com/StefanScherer/dockerfiles-windows you can find lots of interesting images for Windows. In my situation I only need Elasticsearch, so I’ve gotten the image, modified to install the version I need and the game is done.

Now I need to have some specific plugins installed in Elasticsearch for my integration tests to run; these situations are the ones where Docker shows its full powers. Since I’ve already an image with elasticsearch, I only need to create a DockerFile based on that image that install required plugin.

FROM elasticsearch_custom:2.4.6.1

RUN "C:\elasticsearch\bin\plugin.bat" install delete-by-query

Wow, just two lines of code launch docker build command and I have my ElasticSearch machine ready to be used.

image

Figure 2: All my images ready to be used in a Docker compose file

Now everything you need to do is create the docker compose file, here is an example

version: '2.4'

services:
  agent:
    image: azdoagent:1.0
    environment:
      - AZP_TOKEN=****q
      - AZP_POOL=docker
      - AZP_AGENT_NAME=TestAlkampfer
      - AZP_URL=https://dev.azure.com/prxm
      - NSTORE_MONGODB=mongodb://mongo/nstoretest
      - NSTORE_MSSQL=Server=mssql;user id=sa;password=sqlPw3$secure
      - TEST_MONGODB=mongodb://mongo/
      - TEST_ES=http://es:9200

  mssql:
    image: mssqlon2019:latest
    environment:
      - sa_password=sqlPw3$secure
      - ACCEPT_EULA=Y
    ports:
      - "1433:1433"

  mongo:
    platform: linux
    image: mongo
    ports:
      - "27017:27017"

  es:
    image: elasticsearch_jarvis:1.0
    ports: 
      - "9200:9200"

Once the file is ready just start an agent with

docker-compose -f docker-compose.jarvis.yml  up –d

and your agent is up and running.

image

Figure 3: All docker images up and running

Then I go to my C# project and included all the docker file needed to create images as well as docker compose file directly in source repository. This allows everyone to locally build docker image and spin out an agent with all the dependency needed to build the project.

If your organization uses azure or any other container registry (you can also push to a public registry because all these images does not contains sensitive data) spinning the agent is only a matter of starting docker composer instance.

After a couple of minutes (the time needed by azure devops agent to download everything and configure itself) my builds start running.

image

Figure 4: A build running a Full Framework solution with integration tests on MongoDB and Elasticsearch is running in my docker container.

From now on, every time I need to have another agent to build my project, I can simply spin out another docker-compose instance and in less than one minute I have another agent up and running.

Gian Maria.