Continuous Integration in GitHub Actions, deploy in AzureDevops

My dear friend Matteo just published an interesting article on integration between GitHub actions and Azure Devops Pipeline here. I have a different scenario where I’ve already published a GitHub release from a GitHub action, but I have nothing ready in GitHub to release in my machines.

While GitHub is really fantastic for source code and starts having a good support for CI with Actions, in the release part it still miss a solution. Usually this is not a problem, because we have Azure DevOps or other products that can fill the gap.

This specific project is a firewall that closes every port in a machine and listen on UDP ports for a specific message to open other ports, thus, a machine where the service is installed has no way to be contacted unless you use the appropriate client to ask for port opening. I want the deploy to be automatic, no way I’m going to all my machines, login in RDP and then update the service, everything should happen automatically.

The really nice aspect of Azure DevOps release pipelines, is that, once you installed agents in one or more machines, those machines will contact Azure DevOps and pull works to do without the need for the machine to be contacted from outside world.

This is a key point of Azure DevOps release pipeline, you do not need to have any special setup in deploy target, you should simply let the target to be able to contact Azure DevOps site (https://dev.azure.com).

Another nice aspect of Azure DevOps release pipeline, is that it can use many sources for artifacts, not only those produced by Azure DevOps CI pipeline. When you add an artifacts, you can choose a GitHub release as well as Jenkins and other CI providers like Azure Artifacts (check Matteo’s article to see how to publish in azure artifacts from a GitHub Action)

image

Figure 1: Choose GitHub release as artifact source

To use GitHub as source you should have already connected your azure DevOps to GitHub with a service connection, another cool feature of Azure DevOps. As an administrator you can connect Azure DevOps account to GitHub, then give permission to specific people to use that service connection, without requiring them to know the real credentials to connect to the service (Github in this example). Once you have one or more connection active you can simply choose the repository to use. In Figure 2 You can see the configuration I choose for my project.

image

Figure 2: Configure GitHub release as artifact source.

Settings are: Repository (1), default version of the release to use (2) and finally alias you use for that specific artifact in your release (3). Remember that a release can have more than a single artifact as source, if you have a simple project like this, probably you have a single artifact.

Now you have the full power of Azure DevOps pipeline at your fingertips, in this specific example I just need to deploy a Windows Service and this is the pipeline to release in my stages.

image

Figure 3: Release pipeline for a Windows Service

This is a standard  four phase for a service release, first step is needed to stop the service if it is running, then I extract the artifacts coming from GitHub as 7zipped files, then I overwrite the directory where I’ve installed the service and finally I install the service if needed and restart it.

Before launching the release, you need to be sure that you have at least one release associated to that repository, in this example I have release 0.4.1 and others available.

image

Figure 4: Available releases for my GitHub repository

When you create a release (if the release is launched manually) you can choose GitHub release you want to use (if the release is automatic it will use release configured in the artifact configuration, usually latest one), the connection is done by Azure DevOps for you, no need to know credentials of GitHub, just choose the version you want to install and Bam, you are ready.

image

Figure 5: Choose the release you want to use directly from Azure DevOps

When the release starts, your target will download the workflow, it will instruct the agent to download artifacts from GitHub and then your scripts will run releasing the software.

image

Figure 6: Release completed, version 0.4.1 is now released on my machines.

As you can verify from detail page, artifacts are indeed downloaded by a GitHub standard release.

image

Figure 7: Artifacts downloaded directly from GitHub.

If everything runs successfully, you will have the new version installed on all machines part of deployment group used.

image

Figure 8: All steps executed successfully.

As you can see, Azure DevOps has a real powerful way to connect other services like GitHub and this is ideal to compensate the gap that other tools has at the moment. This leaves you free to compose your tooling chain, using the service that is best for the specific part.

Gian Maria.

GitHub security Alerts

I really love everything about security and I’m really intrigued by GitHub security tab that is now present on you repository. In your project usually it is disabled by default.

image

Figure 1: GitHub Security tab on your repository

If you enable it you start receiving suggestion based on code that you check in on the repository, as an example, GitHub will scan your npm packages source to find dependencies with libraries that are insecure.

When GitHub found something that require your attention, it will put a nice warning header on your project, so the alert cannot really pass unnoticed.

image

Figure 2: Security alert warning banner

If you go to the security tab you got a detailed list of the analysis, so you can put a remediation plan in motion, or you can simply dismiss if you believe that you can live with them.

image

Figure 3: Summary of security issues for the repository

Clearly you can click on any issue to have a detailed description of the vulnerability, so you can decide if you are going to fix it or simple dismiss because that issue is not relevant to you or you cannot in anyway bypass the problem.

image

Figure 4: Detailed report of security issue

If you noticed in Figure 4, you have also a nice button “Create Automated Security Fix” in the upper right part of the page, this means that not only GitHub is telling me where the vulnerability is, it sometimes can fix the code for me. Pressing the button will simply create a new Pull Request to fix that error, how nice.

image

Figure 5: Pull request with the fix for the security issue

In this specific situation it is simply a vulnerable package that is donwloaded by npm install, the change is simply bumping a library to a version that removed this vulnerability.

Actually GitHub perform a security scan on project dependencies and can present a remediation simply with nice pull requests

Using Pull request is really nice, really in the spirit of GitHub. The overall experience is really nice, the only annoying stuff is that actually the analysis seems to be done on master branch and proposed solution creates pull requests for master branch. While this is perfectly fine, the only problem I have is that, closing that pull request from the UI, it will merge this commit on the master branch, effectively bypassing GitFlow flow.

Since I’m a big fan of command line, I prefer to close that Pull request manually, so I simply issue a fetch, identify the new branch (it has annoying long name Smile) and simply checkout it as an hotfix branch

$ git checkout -b hotfix/0.3.1 remotes/origin/dependabot/npm_and_yarn/CoreBasicSample/src/MyWonderfulApp.Service/UI/tar-2.2.2
Switched to a new branch 'hotfix/0.3.1'
Branch 'hotfix/0.3.1' set up to track remote branch 'dependabot/npm_and_yarn/CoreBasicSample/src/MyWonderfulApp.Service/UI/tar-2.2.2' from 'origin'.

With this commands I simply checkout the remote branch as hotfix/0.3.1, so I can simply issue a git flow hotfix finish and pushing everything back to the repository.

If you have a specific flow for hotfixes, like GitFlow, it is quite easy closing Pull Requests locally, following your process, GitHub will automatically detect that the PR is closed after the push.

Now branch is correctly merged

image

Figure 6: Pull request manually merged.

If you really like this process, you can simply ask GitHub to automatically create pull requests without your intervention. As soon as a security fix is present, a PR will be created.

image

Figure 7: You can ask to receive automated pull request for all vulnerabilities

Et voilà, it is raining pull requests

image

Figure 8: A series of Pull requests made to resolve security risks

This raise another little issue, we have a single PR for each vulnerability, so, if I want to apply all of them in a unique big hotfix, I only need to manually start the hotfix, then fetch all those branches from the repo and finally cherry-pick all the commits. This operation is easy because each Pull Request contains a single commit that fixes a single vulnerability issue. Sequence of command is:

git flow hotifx start 0.3.2
git cherry-pick commit1
git cherry-pick commit2
git cherry-pick commit3
git flow hotfix finish

Final result is an hotfix resulted from cherry-picking of three distinct PR.

image

Figure 9: Three of pull requests were closed using simple cherry-pick

GitHub is really good in understanding that I’ve cherry-picked all commits in yellow from pull requests, because all pull requests were automatically closed after the push.

image

Figure 10: My pull requests are correctly closed even if I cherry-picked all commits manually.

Actually this functionality is really nice, in this simple repository I have really few lines of code but it helped me revealing some npm dependencies with vulnerabilities and, most important, it gave me the solution so I can immediately put a remediation in place.

Gian Maria.

First Experience with GitHub Actions

GitHub actions is the new CI/CD system created by GitHub that allows you to build and release your software with a simple workflow defined in YAML file. Actually it is in beta, but you can simply request to be enlisted and your account will be enabled so you can try it in preview.

Actions engine is based on a yaml definition that is stored directly in code, there are lots of predefined actions made by GitHub team as well as custom actions that can be developed by the community.The real power rely on the fact that you can use simply use command line and docker commands, making the creation of a release a simple and smooth process.

Adding a new workflow is really simple, just open the Actions tab of the repository, then ask to create a new worfklow:

image

Figure 1: Create new workflow for GitHub action directly from repository page.

This will simply create a new yml file in a directory called .github and you can immediately start editing the build. The syntax is really simple and it aims to simplicity rather than complexity. The vast majority of tasks can be simple accomplished inserting command line arguments.

My first impression is that the strongest point of GitHub actions is simplicity and easy to use.

Here is the first part of workflow definition:

name: NStore CI

on: [push]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        dotnet: [ '2.2.401', '3.0.100-preview9-014004'] 
        os: ['ubuntu-latest']
    name: Build for .NET ${{ matrix.dotnet }}
    steps:

You can find complete workflow syntax at this page, but here is the explanation of my workflow. First of all on: [push] directive asks for continuous integration (run action for each push), then a list of jobs follows.

First and only job for this example is called build and it could run on different operating system. This is a nice feature of actions called matrix: you can define array of values and use those arrays in workflow definition to have it run multiple time, once for each parameter combination. Array of values are defined inside the strategy.matrix section, where I defined two distinct set of parameters, dotnet (version of dotnet core used to build) and os (type of machine where my action should be run). For this example I’m going to use only framework version as matrix value.

Runs-on step define OS, for this example I’m using ubuntu-latest. Finally I give a name to the job: Build for .NET following the actual version of matrix.dotnet value. When I push the code I can verify that two distinct jobs are scheduled.

image

Figure 2: Two distinct job where scheduled, one for each matrix version.

This is a really nice feature because we can specify a single workflow and have GitHub action engine run it with different configuration.

Thanks to Matrix configuration a single job can be run for many different combination of input parameters.

A job is simply composed by different steps, for my solution, I wants only to build my solution and run some tests agains Microsoft Sql Server and MongoDb

steps:     
    - uses: actions/checkout@v1
    
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: ${{ matrix.dotnet }}
      
    - name: Build with dotnet
      run: dotnet build src/NStore.sln --configuration Release
    
    - name: Start Docker for MSSSql
      run: docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=sqlPw3$secure' -e 'MSSQL_PID=Developer' -p 1433:1433 --name msssql -d mcr.microsoft.com/mssql/server:2017-latest-ubuntu
      
    - name: Start Docker for Mongodb
      run: docker run -d -p 27017:27017 mongo
      
    - name: Dump mssql docker logs
      run: docker logs msssql
      
    - name: Run Tests - Core
      run: dotnet test src/NStore.Core.Tests/NStore.Core.Tests.csproj --configuration Release --no-build

    - name: Run Tests - Domain
      run: dotnet test src/NStore.Domain.Tests/NStore.Domain.Tests.csproj --configuration Release --no-build
    
    - name: Run Tests - MongoDb
      env:
        NSTORE_MONGODB: mongodb://localhost/nstore
      run: dotnet test src/NStore.Persistence.Mongo.Tests/NStore.Persistence.Mongo.Tests.csproj --configuration Release --no-build
           
    - name: Run Tests - MsSql
      env:
        NSTORE_MSSQL: Server=localhost;user id=sa;password=sqlPw3$secure
      run: dotnet test src/NStore.Persistence.MsSql.Tests/NStore.Persistence.MsSql.Tests.csproj --configuration Release --no-build
    
    - name: Dump mssql docker logs after tests
      run: docker logs msssql
      
    - name: Run Tests - Sql Lite
      run: dotnet test src/NStore.Persistence.Sqlite.Tests/NStore.Persistence.Sqlite.Tests.csproj --configuration Release --no-build 

Workflows starts with actions/checkout@v1 a really standard action that simply clone and checkout the code, it is followed by another action that ensure that a specific version of .NET core SDK is installed and configured in the system. It is declared with the syntax uses: actions/setup-dotnet@v1 and allows me to use a specific version of .NET core; this action supports parameters, and is followed by a with: section used to pass parameters. This is another strong point of GitHub actions, it is really simple to declare and use actions, there is no need to install or reference anything, just reference the action in the right repository and the game is done.

The rest of the repository is a series of steps composed only by a name and a command line instruction. This allows me to simply issue dotnet command to restore, build, test my solution.

Another cool aspect of Actions is that Docker is available inside the machine, this allows me to run a couple of containers: SQL Server and MongoDb, to run my tests during a build. This is super cool, because it allows me to use Docker to create all prerequisites that I need for my build.

Having Docker inside the machine that runs actions is a real blessing because it allows to run integration tests.

My first impression is quite positive, with just a bunch of Yaml code I was able to create a workflow to build and run tests for my project (I spent a quite good amount of time to have MsSql container work, but this is another story).

Another good aspect of Actions is the ability to see real-time log of your run directly from a browser, without the need of installing anything.

A final real nice aspect of Actions is that they are defined by conventions inside a special folder .github/workflows; I’ve developed this build in a fork of the original project, then I issued a pull request and when the pull request was accepted, this new workflows appears in the original repository.

image

Figure 3: After pull request was merged, immediately the workflow is up and running on target repository..

Clearly this is still a beta and there are still part that should be improved. First of all, if a test run fails, the build is marked as failed and you need to look at test logs to understand which tests failed.

image

Figure 3: Build failed, but to understand why it failed you need to check the logs.

This is the reason why I included a distinct test step for each test assembly, instead of a simple dotnet run on the entire solution. Using this little trick I can at least understand which test run failed.

image

Figure 4: Action run result, each failed step is marked with a red cross

Clicking on failed step, you can find the output log of the step, needed to understand which tests failed and why. For those of you used to Azure DevOps pipeline, you will surely miss the nice Test Result page, but I’m expecting GitHub actions to close the gap in this area.

image

Figure 5: Action step run detail.

Another problem I found (but I need to investigate more) is that docker seems not to be available on MacOS Machine. If I run previous build on MacOS I got a docker command not found.

You only need to enlist in beta and start playing with Actions, you will surely find a good use for them.

Gian Maria.

Git for windows, getting Invalid username or password with Wincred

If you use Https to communicate with your git repository, Es, Github or VisualStudioOnline, you usually setup credential manager to avoid entering credential for each command that contact the server. With latest versions of git you can configure wincred with this simple command.

git config --global credential.helper wincred

This morning I start getting error while I’m trying to push some commits to GitHub.

$ git push
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/ProximoSrl/Jarvis.DocumentS
tore.git/'

If I remove credential helper (git config –global credential.helper unset) everything works, git ask me for user name and password and I’m able to do everything, but as soon as I re-enable credential helper, the error returned. This problem is probably originated by some corruption of stored credentials, and usually you can simply clear stored credentials and at the next operation you will be prompted for credentials and everything starts worked again. The question is, where are stored credential for wincred?

If you use wincred for credential.helper, git is storing your credentials in standard windows Credential Manager

You can simple open credential manager on your computer,

image

Figrue 1: Credential manager in your Control Panel settings

Opening Credential manager you can manage windows and web credentials. Now simply have a look to both web credentials and windows credentials, and delete everything related to GitHub or the server you are using. The next time you issue a git command that requires authentication, you will be prompted for credentials again and the credentials will be stored again in the store.

Gian Maria.