Create Word document from Work Items

Post in the series:
1) API Connection
2) Retrieve Work Items Information
3) Azure DevOps API, Embed images into  HTML

Now we have all the prerequisites in place to connect to an Azure DevOps account, execute a query to grab all work items of a sprint and modifying HTML of Rich Edit fields to embed images. It is time to create a word document.

To have a better look and feel of exported document, the best approach is using the concept of Templates created by simple Word documents. With this technique we can use all the styles, formatting directly in Word, then use some placeholder to specify where you want to include fields of work Items.

image

Figure 1: A simple example of a Word Template used to export content of a Work Item

As you can see from Figure 1, a template is a simple word file where I have some special placeholder like {{title}} in the text to identify the point where I want to insert content taken from Work Items. This approach is really useful because Open XML format has a really nice feature that allows you to embed word documents inside other Word documents. This will allows me to open the template, perform substitution keeping all formatting, finally save everything to a temp file and append to the main document. With this approach I do not need to do any formatting in code, while giving the user of the tool the ability to decide the template of the output simply editing a word file.

The concept of template made extremely simple for a user to specify the formatting while keeping the code simple because it should only look for specific tokens and perform substitution.

I will really thanks Proximo S.r.L. a company I’m collaborating with for giving me the permission to share the code to manipulate Word Document, and to publish it open source. The whole code is in the example hosted in GitHub,

If take an high level look to the routine, I simply grab a reference to a list of WorkItems object, then proceed to generate a new Word Document with the help of an object called WordManipulator that contains all the routines I needs to generate a word starting from templates.

var fileName = Path.GetTempFileName() + ".docx";
using (WordManipulator manipulator = new WordManipulator(fileName, true))
{
    foreach (var workItem in workItems)
    {
        manipulator.InsertWorkItem(workItem, @"Templates\WorkItem.docx", true);
    }
}

WordManipulator class simply accept a name of a file, and a boolean value to specify if we need to create a new file, in this example I request for creation of a new file, then InsertWorkItem method will accept the template file and a boolean value that specify if you want to add a page break after the Work Item.

public void InsertWorkItem(WorkItem workItem, String workItemTemplateFile, Boolean insertPageBreak = true)
{
    //ok we need to open the template, give it a new name, perform substitution and finally append to the existing document
    var tempFile = Path.GetTempFileName();
    File.Copy(workItemTemplateFile, tempFile, true);
    using (WordManipulator m = new WordManipulator(tempFile, false))
    {
        m.SubstituteTokens(CreateDictionaryFromWorkItem(workItem));
    }

    AppendOtherWordFile(tempFile, insertPageBreak);
    File.Delete(tempFile);
}

As promised the routine is really simple, just create a temporary file name, copy the template file over it, then open with another instance of WordManipulator and call the SubstituteTokens function, passing a dictionary with all the fields of Work Items we want to export.

private Dictionary CreateDictionaryFromWorkItem(WorkItem workItem)
{
    var retValue = new Dictionary();
    retValue["title"] = workItem.Title;
    retValue["description"] = new HtmlSubstitution(workItem.EmbedHtmlContent(workItem.Description));
    retValue["assignedto"] = workItem.Fields["System.AssignedTo"].Value?.ToString() ?? String.Empty;
    retValue["createdby"] = workItem.Fields["System.CreatedBy"].Value?.ToString() ?? String.Empty;
    return retValue;
}

For this first example I export only four fields, but what it is interesting is that use an helper class called HmlSubstitution for the WorkItem.Description field, to specify to the substitution engine that I do not want a simple text substitution but I need a piece of HTML to be inserted into the document. The helper method EmbedHtmlContent was previously discussed and it is needed only to have an HTML with all the image embedded as base64.

Thanks to the concept of templates, creating a Word Document from Work Items it is just a series of  simple operations: open template, perform substitution and append to the main document.

The SubstituteTokens is a slightly more complex, because it scans all paragraphs of the document looking for keys of the substitution dictionary; when a key is found it will perform substitution using corresponding value. The code is complex because when you put a token like {{token}} inside a word file, it could be stored in XML format using more than one simple Run object (consult the ECMA for specifications). Given this premise, the code will try to find all run objects that contains the token, then perform substitution.

Even if a paragraphs seems really simple in Word, it could be saved with many Runs in OpenXml format, thus when you perform substitution you should never assume that a token will fit into an entire run.

Some of the routine are really interesting, the AppendOtherWordFile will simply append another file to the current one, and it is using the concept of the AltChunk an object in the SDK that allows me to embed one document into another. The trick about AltChunk is that it is a simple object where you can store complex data with the FeedData method, simply passing a stream as argument.

public WordManipulator AppendOtherWordFile(String wordFilePath, Boolean addPageBreak = true)
{
    MainDocumentPart mainPart = _document.MainDocumentPart;
    string altChunkId = "AltChunkId" + Guid.NewGuid().ToString();
    AlternativeFormatImportPart chunk = mainPart.AddAlternativeFormatImportPart(AlternativeFormatImportPartType.WordprocessingML, altChunkId);

    using (FileStream fileStream = File.Open(wordFilePath, FileMode.Open))
    {
        chunk.FeedData(fileStream);
        AltChunk altChunk = new AltChunk();
        altChunk.Id = altChunkId;
        mainPart.Document
            .Body
            .InsertAfter(altChunk, mainPart.Document.Body
            .Elements().LastOrDefault());
        mainPart.Document.Save();
    }
    if (addPageBreak)
    {
        _body.Append(
            new Paragraph(
            new Run(
                new Break() { Type = BreakValues.Page })));
    }
    return this;
}

The real magic is in the AddAlternativeFormatImportPart of the MainDocumentPart of the destination document, that allows you to specify the creation of a special chunk, containing a AlternativeFormatImportPartType.WorprocessingML (another word document). Thanks to this method we can create an alternate part, copying the entire content of the word document to attach and finally add this part to the original document (at the last position).

This method is so powerful that it can be used to create an alternate Import part of HTML type.

private AltChunk CreateChunkForHtmlPage(string htmlPage)
{
    var realHtml = $"{htmlPage}";
    string altChunkId = "myid" + Guid.NewGuid().ToString();
    using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(realHtml)))
    {
        // Create alternative format import part.
        AlternativeFormatImportPart formatImportPart = _document.MainDocumentPart.AddAlternativeFormatImportPart(
            AlternativeFormatImportPartType.Html,
            altChunkId);

        // Feed HTML data into format import part (chunk).
        formatImportPart.FeedData(ms);
    }
    var altChunk = new AltChunk();
    altChunk.Id = altChunkId;
    return altChunk;
}

OpenXML format is really fascinating and it is a real fantastic effort made by Microsoft to create a standard that is easy to use. As you can see from the above snippet, inserting HTML code inside a Word Document is done with a couple of calls.

All the rest of the code in the example is boilerplate, and here is the result of a test export. The code relating to this example is in GitHub with the tag 0.2.0. Here is an example of an exported document

image

Figure 2: An exported document with complex description

The original Work Item in Figure 2 was created with AIT WordToTfs tools, that allows bidirectional editing of Work Item in Word. As you can see, thanks to this tool I was able to change font in the Work Item description, and you can also verify to the Rigth that exported document maintains the formatting, and it is also using the Word template file.

Output Word Document still maintain all formatting of the template (color, bold, font etc) but also the Work Item description maintains its formatting, so the export is high fidelity.

To run this example I used this command line that allows me to specify all the information needed by the tool to export everything.

--address https://gianmariaricci.visualstudio.com 
--tokenfile C:\develop\Crypted\patOri.txt 
--teamproject "zoalord insurance" 
--iterationpath "zoalord insurance\Release 1\Sprint 6"
--areapath "zoalord insurance"

Happy new Year and Happy Azure DevOps.

Gian Maria.

Azure DevOps API, Embed images into html

Post in the series:
1) API Connection
2) Retrieve Work Items Information

Before going to generate a Word File from Work Item Data we need to solve a little problem with HTML content in Work Item fields. As you know Azure DevOps has a rich web editor that allows you to create complex text in some fields, like Description, the problem is: whenever you copy and paste images inside the Web Editor, those images were added as Work Item attachments and the real HTML content is just a reference to the attachmen Url. If you want to generate a consistent Word or export to whatever destination you want, you should manipulate html to embed the image, or the html will be not consistent.

Focus of this article will be: how I can download attachment of Work Items and how I can embed image attachment directly in HTML code.

Html content in Work Item support images, but images are usually a reference to attachment of  the Work Item itself, thus it is not consistent because it refers protected resources.

Here is an example, I have a work item, I embedded an image in the description and the HTML content of System.Description field is an <img> tag with this src value: https://gianmariaricci.visualstudio.com/3a600197-fa66-4389-aebd-620186063db0/_apis/wit/attachments?FileID=481805&amp;FileName=System.Description.0.png. Actually this could be seen as a no-problem, because if you copy this url into a browser the image will be correctly downloaded, but the problem rely in authentication. If you are going to embed this HTML into a Word Document no one will be able to visualize the image, because word is not authenticated to Azure DevOps, thus you need to download locally and embed into the html document.

A possible approach is reference the HtmlAgilityToolkit library, then build a routine that programmatically download every attachment, and finally embeds the image in src attribute value using Base64 encoding, here is the code.

public static String EmbedHtmlContent(this WorkItem workItem, String htmlContent)
{
    HtmlDocument doc = new HtmlDocument();
    doc.LoadHtml(htmlContent);

    var images = doc.DocumentNode.SelectNodes("//img");
    if (images != null)
    {
        foreach (var image in images)
        {
            //need to understand if it is in base 64 or no, if the answer is no, we need to embed image
            var src = image.GetAttributeValue("src", "");
            if (!String.IsNullOrEmpty(src))
            {
                if (src.Contains("base64")) // data:image/jpeg;base64,
                {
                    //image already embedded
                    Log.Debug("found image in html content that was already in base64");
                }
                else
                {
                    Log.Debug("found image in html content that point to external image {src}", src);
                    //is it a internal attached images?
                    var match = Regex.Match(src, @"FileID=(?\d*)");
                    if (match.Success)
                    {
                        var attachment = workItem.Attachments
                            .OfType()
                            .FirstOrDefault(_ =&gt; _.Id.ToString() == match.Groups["id"].Value);
                        if (attachment != null)
                        {
                            //ok we can embed in the image as base64
                            WorkItemServer wise = workItem.Store.TeamProjectCollection.GetService();
                            var downloadedAttachment = wise.DownloadFile(attachment.Id);
                            byte[] byteContent = File.ReadAllBytes(downloadedAttachment);
                            String base64Encoded = Convert.ToBase64String(byteContent);
                            var newSrcValue = $"data:image/{attachment.Extension.Trim('.')};base64,{base64Encoded}";
                            image.SetAttributeValue("src", newSrcValue);
                        }
                    }
                }
            }
        }
    }

    return doc.DocumentNode.OuterHtml;
}

This code is really simple, it is an extension method of the WorkItem type so you can simply use whenever you have a reference to a Work Item. The code will simply search in all HTML text img tags, for each img tag it will verify if it already contains string base64 (because the image could be already embedded), if the answer is no, we need to download the image locally and embed.

If you look at the attachment url you can notice a FileID=xxxx that points to the attachment of the work item. With a simple regex I can find if the url conform to this pattern, and if the answer is yes, I’ll search into WorkItem.Attachments collection for the right attachment.

Work Item object in C# library has a nice Attachments collection that allows you to iterate through all attachments to find any information you need

Having a reference to the Attachment is crucial, because I need to know the extension of the file. Once the attachment object is found, I can use Store property of the work item to grab a reference to the TfsTeamProjectCollection object that allows me to grab a reference to the WorkItemServer object, that is needed to download the file locally. Thanks to C# object model, if I have a simple reference to a Work Item I can still traverse properties to grab a reference to the original collection object that was still authenticated to the server.

Using Store property of Work Item allows you to access the original Collection object that is authenticated to the server, thus you can ignore authentication problems

Once I have a reference to WorkItemServer, its method DownloadFile will simply download attachment by id to a temp local file, then a simple conversion to Base64 will perform the trick. The result is a src attribute that embed the image.

image

Figure 1: Src attribute with image embedded

Now I can simply change the attribute of the image thanks to HtmlAgilityToolkit library, and finally return modified HTML to the caller.

Now I have html code that embed all images and has no reference to external resources in Azure DevOps, so I can embed it everywhere I want without any problem.

Gian Maria.

Run code coverage for Python project with Azure DevOps

Creating a simple build that runs Python tests written with PyTest framework is really simple, but now the next step is trying to have code coverage. Even if I’m pretty new to Python, having code coverage in a build is really simple, thanks to a specific task that comes out-of-the-box with Azure DevOps: Publish Code Coverage.

In Azure DevOps you can create build with Web Editor or with simple YAML file, I prefer YAML but since I’ve demonstrated in the old post YAML build for Python, now I’m creating a simple build with standard Web Editor

Instead of creating a Yaml Build, this time I’m going to demonstrate a classic build: here is the core part of the build.

image

Figure 1: Core build to run tests and have code coverage uploaded to Azure DevOps

As you can see, I decided to run test with a Bash script running on Linux, here is the task configuration where I’ve added Pytest options to have code coverage during test run.

image

Figure2: Configuration of Bash script to run Pytests

The task is configured to run an inline script (1), command line (2) contains –cov options to specify Python modules I want to monitor for code coverage, then a couple of –cov-report options to have output in xml and HTML format. Finally I’ve specified the subfolder that contains the module I want to test (3) and finally I’ve configured the task con Continue on Error (4), so if some of the tests fails the build will be marked as Partially failed.

Thanks to Pytest running code coverage is just a matter of adding some options to command line

After a build finished you can find in the output how Pytest generates Code Coverage reporting, it create a file called coverage.xml then an entire directory called htmlcov that contains a report for code coverage.

image

Figure 3: Result of running tests with code coverage.

If you look at Figure 1 you can see that the build final task is a Publish Code Coverage Task, whose duty is to grab output of the Pytest run and upload to the server. Configuration is really simple, you need to choose Cobertura as Code coverage tool (the format used by Pytest) and the output of test run. Looking at output of Figure 3 you can double check that the summary file is called coverage.xml and all the report directory is in htmlcov subdirectory.

image

Figure 4: Configuration for Publish Code Coverage task.

Once you run the build, you can find Code Coverage result on the summary page, as well as Code Coverage Report published as Build artifacts, the whole configuration will take you no more than 10 minutes.

image

Figure 5: Artifacts containing code coverage reports as well as code coverage percentage are accessible from Build Summary page.

Finally you have also a dedicated tab for Code Coverage, showing the HTML summary of the report

image

Figure 6: Code coverage HTML report uploaded in a dedicated Code Coverage tab in build result

Even if the code coverage output is not perfectly formatted you can indeed immediately verify percentage of code coverage of your test.

Gian Maria.

Set new Azure DevOps url for your account

Due to switching from the old url format organization.visualstudio.com to dev.azure.com/organization it is a good practice to start transition to the new url as soon as possible. The old url will continue to function for a long time, but the new official domain is going to become the default.

Every user can still use both the new or old domain name, but there is a settings in the general setting page of the account that globally enable the new url.

image

Figure 1: New url format enabled in global settings.

Actually there are small parts of the site that does not work perfectly if you browse Azure DevOps with the url not configured in that settings. This is often due to security restriction especially for Cross Site Origin. As an example I got problem with the new url in the release page, because some of the script will be still served with organization.visualstudio.com and they got blocked due to CORS

image

Figure 2: Cross origin request blocked due to security plugin.

This will prevent me to correctly use the new address. After you switch to the new url with the settings shown in Figure 1: The problem is gone.

This setting will also made a redirect to the new url, whenever one is typing organization.visualstudio.com they will be redirected to the new url, thus enforcing the usage of the new url automatically.

Gian Maria.

Analyze your GitHub project for free with Azure DevOps and SonarCloud

I’ve blogged some weeks ago on how to analyze OS code with SonarCloud, but it is time to update the post, because if you want to use SonarCloud you have a dedicated extension in the marketplace.

image 

Figure 1: Official SonarCloud extension in the marketplace.

One of the great feature of Azure DevOps is its extendibility, that allows people external to Microsoft to create extensions to expand the possibility of the tool. Once you’ve added the SonarCloud extension to your account, you have a whole bunch new build templates you can use:

image

Figure 2: Build template based on Sonar Cloud

Having a template make super easy to create a build, you just choose .NET Desktop with SonarCloud and you are ready to go. As you can see in Figure 2 you can also use Azure DevOps pipeline to build with Gradle, maven or .NET core, so you are not confined to microsoft tooling.

In Figure 3 there is the build created by .NET desktop project template (remember that this template can be used also for web application, and for every .NET application).

image

Figure 3: .NET Sonar Cloud analysis template.

The only task you need to configure for Sonar Cloud analysis is the Prepare analysis on Sonar Cloud. As you can see in Figure 4, you should first create an endpoint that connect Azure DevOps to your SonarCloud account.

SNAGHTML1d1ca3

Figure 4: In task configuration you have a nice button to create the connection to your SonarCloud account

Configuring the connection is really simple, just give a name to the connection and specify the access token (you should first generate a token in SonarCloud). Then, as shown in Figure 5, press Verify Connection to check that everything is ok.

image

Figure 5: Configuration and test of the connection between Azure DevOps and SonarCloud.

Thanks to the concept of external services, you can configure one or more connection to SonarCloud and having it available in the build without disclosing tokens.

Once you’ve selected the connection, just specify name and key of the project, and other optional parameters if you need to do a custom analysis. In less than a couple of minutes you have a build up and running. Just configure the agent to use Hosted VS2017 pipeline and queue a first build to verify that everything is ok.

Once you have configured the build with the visual web designer, you can convert to Yaml build with few steps.

Clearly I prefer to have a YAML build for a lot of reasons, once the build is up and running simply press the YAML button in the build definition to have your build converted to YAML.

# .NET Desktop
# Build and run tests for .NET Desktop or Windows classic desktop solutions.
# Add steps that publish symbols, save build artifacts, and more:
# https://docs.microsoft.com/azure/devops/pipelines/apps/windows/dot-net

pool:
  vmImage: 'VS2017-Win2016'

trigger:
- master
- develop
- release/*
- hotfix/*
- feature/*

variables:
  solution: 'migration/MigrationPlayground.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

steps:

- task: GitVersion@1
  displayName: GitVersion 
  inputs:
    BuildNamePrefix: 'MigrationCI'

- task: SonarSource.sonarcloud.14d9cde6-c1da-4d55-aa01-2965cd301255.SonarCloudPrepare@1
  displayName: 'Prepare analysis on SonarCloud'
  inputs:
    SonarCloud: 'SonarCloud'
    organization: 'alkampfergit-github'
    projectKey: MigrationPlayground
    projectName: MigrationPlayground
    projectVersion: '$(AssemblyVersion)'

- task: NuGetToolInstaller@0

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: SonarSource.sonarcloud.ce096e50-6155-4de8-8800-4221aaeed4a1.SonarCloudAnalyze@1
  displayName: 'Run Code Analysis'

- task: SonarSource.sonarcloud.38b27399-a642-40af-bb7d-9971f69712e8.SonarCloudPublish@1
  displayName: 'Publish Quality Gate Result'




Finally, if you still have not installed Azure Devops Pipeline in your GitHub account, I strongly suggest you to do so, just follow the instruction of this article, it is free and gives you free hosted pipelines to run your build for free.

Gian Maria