Azure DevOps agent with Docker Compose

I’ve dealt in the past on using Docker for your Azure DevOps Linux Build Agent in a post called Configure a VSTS Linux agent with docker in minutes and also I’ve blogged on how you can use Docker inside a build definition to have some prerequisite for testing (like MongoDb and Sql Server), now it is time to move a little step further and leverage Docker compose.

Using Docker commands in pipeline definition is nice, but has some drawbacks: First of all this approach suffers in speed of execution, because the container must start each time you run a build (and should be stopped at the end of the build). Is indeed true that if the docker image is already present in the agent machine startup time is not so high, but some images, like MsSql, are not immediately operative, so you need to wait for them to be ready for Every Build. The alternative is leave them running even if the build is finished, but this could lead to resource exaustion.

Another problem is dependency from Docker engine. If I include docker commands in build definition, I can build only on a machine that has Docker Installed. If most of my projects uses MongoDb, MsSql and Redis, I can simple install all three on my build machine maybe using a fast SSD as storage. In that scenario I’m expecting to use my physical instances, not waiting for docker to spin new container.

Including Docker Commands in pipeline definition is nice, but it tie the pipeline to Docker and can have a penalty in execution speed

What I’d like to do is leverage docker to spin out an agent and all needed dependencies once, then use that agent with a standard build that does not require docker. This gives me the flexibility of setting up a build machine with everything preinstalled, or to simply use Docker to spin out in seconds an agent that can build my code. Removing Docker dependency from my pipeline definition gave user the most flexibility.

For my first experiment I want also use Docker in Windows 2109 to leverage Windows Container.

First of all you can read the nice MSDN article about how to create a Windows Docker image that downloads, install and run an agent inside a Windows server machine with Docker for Windows. This allows you to spin out a new Docker Agent based on Windows image in minutes (just the time to download and configure the agent).

Thanks to Windows Containers, running an Azure DevOps agent based on Windows is a simple Docker Run command.

Now I need that agent to being able to use MongoDb and MsSql to run integration tests. Clearly I can install both db engine on my host machine and let docker agent to use them, but since I’ve already my agent in Docker I wish for dependencies to run also in Docker; so… welcome Docker Compose.

Thanks to Docker Compose I can define a YAML file with a list of images that are part of a single sceanrio so I specified an Agent image followed by a Sql Server and a MongoDb images. The beauty of Docker-compose is the ability to refer to other container machines by name. Lets do an example: here is my complete docker compose YML file.

version: '2.4'

    image: dockeragent:latest
      - AZP_TOKEN=efk5g3j344xfizar12duju65r34llyw4n7707r17h1$36o6pxsa4q
      - AZP_AGENT_NAME=mydockeragent
      - AZP_URL=
      - NSTORE_MONGODB=mongodb://mongo/nstoretest
      - NSTORE_MSSQL=Server=mssql;user id=sa;password=sqlPw3$secure

    image: mssqlon2019:latest
      - sa_password=sqlPw3$secure

      - "1433:1433"

    platform: linux
    image: mongo
      - "27017:27017"

To simplify everything all of my integration tests that needs a connection string to MsSql or MongoDb grab the connection string by environment variable. This is convenient so each developer can use db instances of choice but also this technique makes super easy to configure a Docker agent specifying database connection strings as seen in Figure 1. I can specify in environment variables connection string to use for testing and I can simply use other docker service names directly in connection string.


Figure 1: Environment variable to specify connection string.

As you can see (1) connection strings refers to other containers by name, nothing could be easier.

The real advantage of using Docker Compose is the ability to include Docker Compose file (as well as dockerfiles for all custom images that you could need)  inside your source code. With this approach you can specify build pipelines leveraging YAML build of Azure DevOps and also the configuration of the agent with all dependencies.

Since you can configure as many Agent you want for Azure DevOps (you actually pay for number of concurrent executing pipeline) thanks to Docker Compose you can setup an agent suitable for your project in less than one minutes. But this is optional, if you do not like to use Docker compose, you can simply setup an agent manually, just as you did before.

Including a docker compose file in your source code allows consumer of the code to start a compatible agent with a single command.

Thanks to docker compose, you pay the price of downloading pulling images once, also you are paying only once the time needed for any image to become operative (like MsSql or other databases that needs a little bit before being able to satisfy requests). After everything is up and running, your agent is operative and can immediately run your standard builds, no docker reference inside your YAML build file, no time wasted in waiting your images to become operational.

Thanks to experimental feature of Windows Server2019, I was able to specify a docker-compose file that contains not only windows images, but also Linux images. The only problem I had is that I did not find a Windows 2019 Image for Sql Server. I started getting error using standard MsSql images (build for windows 2016); So I decided to download official Docker file, change reference image and recreate the image and everything worked like a charm.


Figure 2: Small modification to Sql Server docker windows image file, targeting Windows Server 2019.

Since it is used only for test, I’m pretty confident that it should work and indeed my build runs just fine.


Figure 3: My build result, ran on docker agent.

Actually my agent created with Docker Compose is absolutely equal to all other agentw, from the point of view of Azure DevOps it has nothing different, but I’ve started it with a single line of Docker-Compose command


Figure 4: Agent running in docker it is treated as standard agents.

That’s all, with a little effort I’m able to include in my source code both YAML build definition as YAML Docker Compose file to specify the agent with all prerequisites to ran the build. This is especially useful for Open Source projects, where you want to fork a project then activate CI with no effort.

Gian Maria.

Analyze your GitHub project for free with Azure DevOps and SonarCloud

I’ve blogged some weeks ago on how to analyze OS code with SonarCloud, but it is time to update the post, because if you want to use SonarCloud you have a dedicated extension in the marketplace.


Figure 1: Official SonarCloud extension in the marketplace.

One of the great feature of Azure DevOps is its extendibility, that allows people external to Microsoft to create extensions to expand the possibility of the tool. Once you’ve added the SonarCloud extension to your account, you have a whole bunch new build templates you can use:


Figure 2: Build template based on Sonar Cloud

Having a template make super easy to create a build, you just choose .NET Desktop with SonarCloud and you are ready to go. As you can see in Figure 2 you can also use Azure DevOps pipeline to build with Gradle, maven or .NET core, so you are not confined to microsoft tooling.

In Figure 3 there is the build created by .NET desktop project template (remember that this template can be used also for web application, and for every .NET application).


Figure 3: .NET Sonar Cloud analysis template.

The only task you need to configure for Sonar Cloud analysis is the Prepare analysis on Sonar Cloud. As you can see in Figure 4, you should first create an endpoint that connect Azure DevOps to your SonarCloud account.


Figure 4: In task configuration you have a nice button to create the connection to your SonarCloud account

Configuring the connection is really simple, just give a name to the connection and specify the access token (you should first generate a token in SonarCloud). Then, as shown in Figure 5, press Verify Connection to check that everything is ok.


Figure 5: Configuration and test of the connection between Azure DevOps and SonarCloud.

Thanks to the concept of external services, you can configure one or more connection to SonarCloud and having it available in the build without disclosing tokens.

Once you’ve selected the connection, just specify name and key of the project, and other optional parameters if you need to do a custom analysis. In less than a couple of minutes you have a build up and running. Just configure the agent to use Hosted VS2017 pipeline and queue a first build to verify that everything is ok.

Once you have configured the build with the visual web designer, you can convert to Yaml build with few steps.

Clearly I prefer to have a YAML build for a lot of reasons, once the build is up and running simply press the YAML button in the build definition to have your build converted to YAML.

# .NET Desktop
# Build and run tests for .NET Desktop or Windows classic desktop solutions.
# Add steps that publish symbols, save build artifacts, and more:

  vmImage: 'VS2017-Win2016'

- master
- develop
- release/*
- hotfix/*
- feature/*

  solution: 'migration/MigrationPlayground.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'


- task: GitVersion@1
  displayName: GitVersion 
    BuildNamePrefix: 'MigrationCI'

- task: SonarSource.sonarcloud.14d9cde6-c1da-4d55-aa01-2965cd301255.SonarCloudPrepare@1
  displayName: 'Prepare analysis on SonarCloud'
    SonarCloud: 'SonarCloud'
    organization: 'alkampfergit-github'
    projectKey: MigrationPlayground
    projectName: MigrationPlayground
    projectVersion: '$(AssemblyVersion)'

- task: NuGetToolInstaller@0

- task: NuGetCommand@2
    restoreSolution: '$(solution)'

- task: VSBuild@1
    solution: '$(solution)'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: VSTest@2
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: SonarSource.sonarcloud.ce096e50-6155-4de8-8800-4221aaeed4a1.SonarCloudAnalyze@1
  displayName: 'Run Code Analysis'

- task: SonarSource.sonarcloud.38b27399-a642-40af-bb7d-9971f69712e8.SonarCloudPublish@1
  displayName: 'Publish Quality Gate Result'

Finally, if you still have not installed Azure Devops Pipeline in your GitHub account, I strongly suggest you to do so, just follow the instruction of this article, it is free and gives you free hosted pipelines to run your build for free.

Gian Maria

Welcome Azure DevOps

Yesterday Microsoft announced a change in naming for VSTS, now branded as Azure DevOps. You can read most of the details in this blog post and if you are using VSTS right now you will not have a big impact in the future. Event is this is just a rebranding of the service, there are a couple of suggestion I’d like to give you to have a smoother transition.

Visual  Studio Team Services was rebranded in Azure DevOps, this will not impact your existing VSTS projects, but it is wise to start planning for a smooth transition.

First of all, if you still don’t use the new navigation, I strongly suggests to enable it, because it will become the default navigation in the future, and it is best to gain familiarity with it, before it will become the only UI available.


Figure 1: Enable the new navigation for the account

The nice aspect is that you can enable new navigation only for your account, then enable for all accounts in the instance. This will make the transition smoother, you can find key member of your teams that wants to try new features, let them explore it and after some time let everyone use the new interface, knowing that at least some core members of the team are well used to it. Planning for a smooth transition instead of having big bang day when everyone can only use the new UI it is a wise approach.

Another suggestion is starting to use the new links right now, if your account is, your new URI will be and it is already available for all of your accounts. You can expect that the old URI will work for a really long time, but it is better starting to use the new URI as soon as possible, to avoid having link in the old format that maybe will cease to work some years from now.

Another part of the service that is affected by change of uri is remote address of git repositories. Microsoft assures that the old url will remain valid for a long time, but it is good to spend 1 minute updating remotes to never worrying that some day in the future remotes uri can break.


Figure 2: Change the url of origin to adapt to the new uri of Azure DevOps Repositories.

Updating git remote address is a good practice to immediately start using the new and official link.

Thanks to git, the only thing you need to do is grab the new link using the new UI, and use the command git remote set-url origin newlink to update uri of the remote to the new one, and you can continue work as ever (the first time you will be prompted by a login because you never authenticated git to domain).

Happy VSTS oops :) Happy Azure Devops

Gian Maria.

TFS 2018 is out, time to upgrade

Some days are passed, but it is good to remind you that TFS 2018 is finally out. Some people are surprised because after TFS 2015 we had TFS 2017 and we are still in 2017 and we already have version 2018, but this is the good part of the ALM tools in Microsoft, they are really shipping tons of new goodness each year :).

Release note page contains all the details about the new version, from that link you have a small 13 minute video that explain what is new in this version and as usual in the page you have a detailed list of all the news with detailed information about each of the new features.


I strongly suggest you to start verifying system requirements and planning for the upgrade, because, as usual, it is a good habit to install the latest bit if possible, to avoid having to do a big bang upgrade after years of not upgrading.

It is always a good practice not to skip a single major version, the upgrade process will be smoother than doing a big jump (like people migrating from TFS 2008 to 2015/2017

Apart new features, the above link informs you on all the features that are actually removed from this version, because they were deprecated in the old version. This can be an update blocker, but I strongly suggest you to start thinking to a remediation pattern, instead of being stuck forever in the 2017 version.

From removed features, Team Room is probably the least impacting, very few people are using it, and you can use Slack or other tools. Tfs Extension for SharePoint were also removed, this is also a feature that very few people will miss. The Lab Center in Microsoft Test Manager was also removed, but probably the most important missing feature is the XAML Build support. In TFS 2018 you can only use the new build introduced with TFS 2015, no excuses guys, you really need to migrate every XAML build to the new format, as soon as possible.

Happy upgrading.

Gian Maria

Configure a VSTS Linux agent with docker in minutes

It is really simple to create a build agent for VSTS that runs in Linux and is capable of building and packaging your DotNetCore project, I’ve explained everything in a previous post, but I want to remind you that, with docker, the whole process is really simple.

Anyone knows that setting up a build machine often takes time. VSTS makes it super simple to install the Agent , just download a zip, call a script to configure the agent and the game is done. But this is only one side of the story. Once the agent is up, if you fire a build, it will fail if you did not install all the tools to compile your project (.NET Framework) and often you need to install the whole Visual Studio environment because you have specific dependencies. I have also code that needs MongoDB and Sql Server to run tests against those two databases, this will usually require more manual work to setup everything.

In this situation Docker is your lifesaver, because it allowed me to setup a build agent in linux in less than one minute.

Here are the steps: first of all unit tests use an Environment Variable to grab the connection string to Mongodb, MsSql and every external service they need. This is a key part, because each build agent can setup those environment variable to point to the right server. You can think that 99% of the time the connection are something like mongodb://localhost:27017/, because the build agent usually have mongodb installed locally to speedup the tests, but you cannot be sure so it is better to leave to each agent the ability to change those variables.

With this prerequisite, I installed a simple Ubuntu machine and then install Docker . Once Docker is up and running I just fire up three Docker environment, first one is the mongo database

sudo docker run -d -p 27017:27017 --restart unless-stopped --name mongommapv1 mongo

Than, thanks to Microsoft, I can run Sql Server in linux in a container, here is the second Docker container to run MSSQL

sudo docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=my_password' -p 1433:1433 --name msssql --restart=unless-stopped -d microsoft/mssql-server-linux

This will start a container with Microsoft Sql Server, listening on standard port 1433 and with sa user and password my_password. Finally I start the docker agent for VSTS

sudo docker run \
  -e VSTS_ACCOUNT=prxm \
  -e VSTS_TOKEN=xxx\
  -e TEST_MONGODB=mongodb:// \
  -e TEST_MSSQL='Server=;user id=sa;password=my_password' \
  -e VSTS_AGENT='schismatrix' \
  -e VSTS_POOL=linux \
  --restart unless-stopped \
  --name vsts-agent-prxm \
  -it microsoft/vsts-agent

Thanks to the –e option I can specify any environment variable I want, this allows me to specify TEST_MSSQL and TEST_MONGODB variables for the third docker container, the VSTS Agent. The ip of mongodb and MSSql are on a special interface called docker0, that is a virtual network interfaces shared by docker containers.


Figure 1: configuration of docker0 interface on the host machine

Since I’ve configured the container to bridge mongo and SQL port on the same port of the host, I can access MongoDB and MSSQL directly using the docker0 interface ip address of the host. You can use docker inspect to know the exact ip of the docker container on this subnet but you can just use the ip of the host.


Figure 2: Connecting to mongodb instance

With just three lines of code my agent is up and running and is capable of executing build that require external databases engine to verify the code.

This is the perfect technique to spinup a new build server in minutes (except the time needed for my network to download Docker images :) ) with few lines of code and on a machine that has no UI (clearly you want to do a minimum linux installation to have only the thing you need).

Gian Maria.