Installing on-premise component against TFS Service (TFS On Azure)

Tfs on azure is a a perfectly valid solution to have your TFS hosted by Microsoft, no worries about upgrading, maintaining, backup and most important, it is available from every computer that has an internet connection. This last factor can be really interesting for companies where people sometimes works from home, or from an hotel during transfers because the only solution with an On-Premise server is exposing the server as Https, or using a VPN, but with TFS on Azure all that you need is a simple internet connection.

The most important aspect is that TFS is not a single software, but it is an environment of services. As an example you can consider the section devoted to Continuous Integration, in such a scenario you can choose to use azure to host your build servers, or you can install a build environment on premise or you can create a mix of these.

Installing a Build environment to build solution on team project hosted on azure is really simple, and it is not different from configuring for an On-Premise TFS. First of all insert DVD of TFS and start a normal installation, after you installed TFS you are presented with a series of wizard to configure the various roles that compose a TFS environment. To install a Build Environment you should choose “Configure Team Foundation Build Service”

image_thumb2

Figure 1: First step, choose the wizard to configure the Build Service

The very first information is the Team Project Collection to bind the Build controller to, you can choose from the standard UI “Connect to Team Project Collection”, in my example gianmariaricci.tfspreview.com, login to the service and the game is done.

image_thumb5

Figure 2: Specify Project Collection used by the Build Server, remember each build server can be used for only a Project Collection.

Now the wizard check the Team Project Collection to verify if there are already some Build controlled configured. In Figure 3 I got a screenshot of my installation that has already a Configured Build Controller based on Dev11 Developer preview, so you have all three option enabled.

image_thumb8

Figure 3: The three options you got if another Build Controller is already configured for this Project Collection

The first option is the scale out, basically it adds this machine under control of another previously installed build controller and it is used to add more build power to a specific controller. You can verify from the screenshot that it suggests you to install 8 build agents, because this machine has 8 core. You can choose whatever number you want, as an example maybe this machine is used not only as a build agent, so you can choose to install a number of build agent less than the number of core. The combo permits you to choose the Build Controller to scale out, in case you have already more than one Build Controller bound to the same Project Collection.

The option number two basically is used to replace an existing build machine, this is the option I choose, because the old Build Controller named Vs11-PC, is based on dev11 developer preview and I want it to be replaced with this new installation based on beta bit.s As you can see even this option has a combo to specify the Build Controller to replace and in the combo you got also an indication on the status of the Build Controller (in Figure 3 you can see that it is offline).

Third option permits you to only install the bits, but leaving the configuration to a further moment, it can be useful to prepare template virtual machine to be used as “fast deploy” build agents. Now I need only to specify the user that will be used to run the service, in most situation I can leave the Local Service as suggested by the wizard.

image_thumb11

Figure 4:  Choose the user that will be used to run the Windows Service of the Build Controller, you can simply leave default Local Service

Now you should verify the configuration and proceed to the configuration. When the wizard finishes, congratulations, you have installed a build controller on-premise to build Team Project hosted in TFS Service.

image_thumb15

Figure 5: Congratulation, with few clicks you now have an on-premise build controller bounded to a Project Collection in Windows azure

Now you can simply manage your build controller from the standard Team Foundation Server Administration Console; the cool part is that installing a Build Server on-premise is a matter of pressing Next –> Next –>Next.

image_thumb21

Figure 6: You can monitor the status of the Build Controller directly from the Team Foundation Server administration Console.

If you prefer, Brian Harry confirmed in his blog that a Build Service for Team Foundation Service is now available on TFS Service, so you can simply rely on it to build your software. The only drawback is that you do not have control over the build machines, so if you need specific software installed on the build machines in order to build your project, having an on-premise build machine is a valid approach. Another option is the creation of a Build Service directly on Windows Azure, to have full control over the build controller, but avoiding having id on-premise.

Once you have your build controller up and running you need to be aware of a potential problem of this configuration: bandwidth usage. One of the drawbacks of using TFS Service could be bandwidth usage, especially in country like Italy, where having good connection can be difficult. For everything that regards the new Web UI Interface you will face no problem, because you can use it even with a standard 3G connection from your Cell Phone, because the interface is highly optimized, but for source code the situation is really different.

When developers or build server issue a GetLatest operation, the bandwidth consumption is determined by the size of the files that are requested to the server and even if source codes usually are little text file, it is standard to include in the source control all the libraries and third party control that are needed for the build. For Build server if you configured the build to grab source code for each build you can really have an high bandwidth usage and it can be a real problem in everyday work.

The optimal solution is using a TFS Proxt Server, because it will limit the bandwidth usage from your organization to TFS Service, because when you issue a Get Latest files will be returned by the proxy server and not the TFS Service instance. As for Build configuration, installing a proxy server on-premise is really simple.

After installing TFS 11 on a computer in your network, open Administration console and locate Proxy Server node, then select Configure Installed Features.

image_thumb2

Figure 7: To configure a proxy server just press “configure installed features”

Before starting configuration phase, the wizard shows you a remind of why and when you should use a proxy server; as you can verify the main reason is reducing Bandwidth in download, because each upload operation, like a Check-in, is done directly to the main TFS Server. This is not a problem because the main use of bandwidth is on downloading items.

image_thumb8

Figure 8: A simple reminder that explain the reason to use a proxy

The first question is the address of the Project Collection to proxy, you can choose your collection on TFS Service and proceed to the next step.

image_thumb11

Figure 9: The usual “Connect to Team Project Collection” windows that permits you the server to proxy

Now you should choose the user that will be used to run the proxy service, as usual you can leave the default.

image_thumb14

Figure 10: Specify user of the IIS Application that will run the proxy

Now you need to specify the port of the TFS proxy and the local folder where it will cache files.

image_thumb17

Figure 11: Specify the local folder where the proxy will store the cache.

Now you can finish the wizard and let it configure everything needed.

image_thumb21

Figure 12: Everything is ready, the wizard will install IIS (if needed) and will configure the proxy

This is really simple, because you need to specify only the project collection and you are ready to go. When configuration finished you should see a success windows likes this one.

image_thumb25

Figure 13: The proxy is ready to use, it even modify IIS configuration to optimize performances

As you can read from the summary, the configuration not only configured the proxy, but configure IIS to have optimum performance, like enabling dynamic compression and since this option is valid for every site, the wizard informs you on what he did. Now you can simply change Visual Studio settings to use the Proxy server to download files.

image_thumb1

Figure 14: Configure the use of the proxy in each machine of your organization.

To verify if everything is ok you can simply issue a Get Specific Version to download again the whole source control of a Team Project and verify in the proxy server if the cache folder has file in it. After the Get Specific operation finished you can try it again to verify that this time it sould be really faster, because files are now get from the proxy.

When the proxy is up and running you should go to every on-premise build machines, launch Team Explorer and configure the proxy as in Figure 14, this permits to Build Agents to use proxy during build. This is really important because build server usually do a lot of traffic against TFS Server, especially if you choose to reset the workspace at each build.

If you have not installed Team Explorer on build machine, you can configure proxy with direct manipulation of the windows registry, simply open the HKCU\Software\Microsoft\VisualStudio\10.0\TeamFoundation\SourceControl\Proxy and add a couple of keys called Enabled and Url and the game is done.

image_thumb2

Figure 15: Registry keys to configure TFS proxy for Build Agent machines that does not have Team Explorer Installed.

Now you have configured build and proxy to have maximum benefit from your TFS service account.

Gian Maria

Logging in Custom Build action for Tfs Build 2010

In a previous post I dealt with the creation of a Custom Activity to use in TFS2010 builds, in that example I did not dealt about logging. Logging is a vital task to do in custom action, because it is quite difficult to attach a debugger to the Build Agent, and if a build fails, it is really important to be able to understand what is gone wrong.

If you want to log from a custom action you can use this simple function

 private void LogMessage(String message, CodeActivityContext context)
        {
           BuildInformationRecord<BuildMessage> record =
             new BuildInformationRecord<BuildMessage>()
             {
                 Value = new BuildMessage()
                    {
                        Importance = BuildMessageImportance.High,
                        Message = message,
                    },
             };

            context.Track(record);
        }

Thanks to this function I can use with my custom activity, here is how I use it in my XmlPoke activity, used to change content of an xml file.

LogMessage("XPoke on the file " + filePath, context);

Thanks to this message I can find information on the build log.

image

This is really useful because I can, in this example, verify witch file was changed by my action simply looking at the build log.

alk.

Tags:

Automatic deployment of a web application with TFS Build

When you develop web applications you usually have X developers solving bugs and implementing features, and a series of testers that test application during developing process. A must to have requirement is that

  1. Modifications to the trunk are visible as soon as possible to testers.
  2. Data in test database gets preserved

Point 2 is especially important, testers usually work with the site and fills database with data. Suppose that tester John find a bug that occurs only with specific data, a developer correct the bug, then a deploy is done, all test data are wiped away, and the tester is not able to verify if the bug is gone.

Moreover testers usually fill the database with real data useful for testing, then you need to deploy update to web application while updating database schema preserving data. The optimum solution is the one represented by this schema

image

When a developer does a check-in or at a scheduled time the build server gets latest bit of the tip, then compile it, and update the web server pointing IIS to the new version and updating schema of the database in the meanwhile, preserving all the data inside the test db.

It is absolutely not a complex stuff to do, here is the only modification I need to do into the msbuild file of the build.

<UsingTask
    TaskName="DotNetMarche.MsBuildExtensions.Administrative.IISChangePhisicalDirTask"
    AssemblyFile="..\sources\libs\MsBuildCustomTasks\DotNetMarche.MsBuildExtensions.dll"/>
    
<UsingTask
    TaskName="DotNetMarche.MsBuildExtensions.Xml.XmlPokeTask"
    AssemblyFile="..\sources\libs\MsBuildCustomTasks\DotNetMarche.MsBuildExtensions.dll"/>
    
<Target Name="AfterDropBuild">
    <Message Text="Deploy web application for $(BuildNumber)" />

    <XmlPokeTask
        FilePath="$(DropLocation)\$(BuildNumber)\debug\_PublishedWebsites\NorthwindWeb\web.config"
        XPath="/connectionStrings/add[@name='Northwind']/@connectionString"
        NewValue='Data Source=localhost\sql2008;Initial Catalog=NorthwindCiTest;user=sa;pwd=Pa$$w0rd'
        FailIfError='false' />
    
    <IISChangePhisicalDirTask
        Username="administrator"
        Password="xxxxxxxxxxx"
        MachineName="WS2008V1"
        SiteName="NorthWindTest"
        NewPhisicalDirectory="C:\Drops\NorthwindTest\$(BuildNumber)\debug\_PublishedWebsites\NorthwindWeb\" />
    
    <MSBuild Projects="$(SolutionRoot)\src\DbEdition\NorthwindTest\NorthwindTest\NorthwindTest.dbproj"
                Properties="OutDir=$(DropLocation)\$(BuildNumber)\debug\;TargetDatabase=NorthwindCiTest;DefaultDataPath=C:\Program Files\Microsoft SQL Server\MSSQL10.SQL2008\MSSQL\DATA;TargetConnectionString=Data Source=WS2008V1\SQL2008,1433%3Buser=sa%3Bpwd=Pa$$w0rd;DeployToDatabase=true;"
                Targets="Deploy"  />
</Target>

The first two <UsingTask directive are needed to import a couple of custom msbuild task. The first is the XmlPokeTask, used to change part of an xml file. In this example you can verify how I change the connection string, because the Test Server can have a different setup, in this situation I use sa credentials, this is very bad for security, but this is only an example. In real scenario you must use integrated security.

The second task is a simple task that uses the technique described in this post, I basically change the directory used by the site NorthWindtest of a machine called WS200v1. The most important stuff here, is that drop location is on WS2008V1 machine, and since I know that the physical directory is c:\drops\ I can simply know the real directory from the BuildNuber. This action permits me to point the test site on the latest build.

image 

Finally since the database could be changed I need to follow the instruction of this post to deploy the data on test database. Now each time a build is triggered after the build Test machine IIS was automatically redirected to the latest version, the web.config was changed to suite the test environment, and the database is automatically upgraded. I simply did the first checkin of the project and after the build I test the site from my dev machine

image

The site was deployed, and I’m able to insert a record in the new empty database created by the first build.

Alk.

Tags:

Take control of assembly numbering during a tfs build

One of the most important stuff in a project build, is the ability to mark the assemblies with unique numbers that permits us to reproduce the build. Tfs does not have a standard way of doing this, but with a couple of MsBuild actions it is really simple to overcome this limitation. This is a good example that shows how you can extend build script to do complex task.

First of all I want to change only AssemblyFileVersion and not the AssemblyVersion, in this way all builds are compatible until someone manually changes AssemblyVersion. A standard technique I like very much is letting the programmers to manage major and minor number manually, and letting my builds generates build and revision ones. For build number I want to be able to generate a unique number each build, a sequential generator will be fine; but for revision number I want to use the changeset used to generate the build. To accomplish this we need essentially four macro steps.

In the first step I need to generate unique integer build number, most of the time sequential generator is ok, then I need also to find a way to correlate this generated number with the build label of the TFS. Step two is used to find latest changeset, then in step three we need to check-in modified files (the one used by the generator), being sure that this check-in does not trigger another build, finally we need to modify a file named ProjectVersion.cs that is used by all projects.

To modify AssemblyFileVersion for a project I love this technique: I remove AssemblyFileVersion and AssemblyVersion attributes from assemblyinfo.cs, put them in a single ProjectVersion.cs file stored in the root of the team project. Here is a typical content for the ProjectVersion.cs .

using System.Reflection;
[assembly: AssemblyVersion("1.2.0.0")]
[assembly: AssemblyFileVersion("1.2.0.0")]

Next I import this file as link in every project that belong to this team project

image

With this little trick to change AssemblyFileVersion attribute for every project I need only to change one file. Now it is time to build some custom tasks that will help us to manage the whole process, first of all the task that generates unique numbers and correlate them with build label.

public class BuildVersionNumberManagerTask : Task
{
    /// <summary>
    /// Filename that will store the versions
    /// </summary>
    [Required]
    public String VersionFileName { get; set; }

    /// <summary>
    /// label of the current build.
    /// </summary>
    [Required]
    public String BuildLabel { get; set; }

    /// <summary>
    /// new incremental version numer.
    /// </summary>
    [Output]
    public Int32 NewVersionNumber { get; set; }

    public override bool Execute()
    {
        if (!File.Exists(VersionFileName))
        {
            Log.LogError("The file {0}[{1}] with version number does not exists", VersionFileName, Path.GetFullPath(VersionFileName));
            return false;
        }
        String[] allLines = File.ReadAllLines(VersionFileName);
        String startChar = "";
        if (allLines.Length > 0 && !String.IsNullOrEmpty(allLines[allLines.Length - 1]))
        {
            startChar = "\n";
            String lastline = allLines[allLines.Length - 1];
            String lastNum = lastline.Substring(0, lastline.IndexOf("|"));
            Int32 lastNumber;
            if (!Int32.TryParse(lastNum, out lastNumber))
            {
                Log.LogError("There are errors in the version file, the last line does not contains lastnum|lastlabel valid format");
                return false;
            }
            NewVersionNumber = lastNumber + 1;
        } else
        {
            //There is no lines, or the last line is empty.
            NewVersionNumber = 1;
        }
        String newLastLine = String.Format("{0}{1}|{2}",startChar,  NewVersionNumber, BuildLabel);
        File.AppendAllText(VersionFileName, newLastLine);
        return true;
    }
}

The BuildVersionNumberManagerTask is responsible of the generation of a sequential number, as well as storing in a file the relationship between autogenerated numbers and build labels. It use a simple text file and write a line for each generated number. Here is an example of file content.

...
3|Standard build for CI_20090820.20
4|Standard build for CI_20090820.21
5|Standard build for CI_20090820.24
...

With such a technique, we can immediately find the build label associated with each auto generated number. Now if you have problem with an assembly that have 4 as the build number, I immediately find in the file that it was build by “Standard build for CI_20090820.21” Here is how I call this task into TFSBuild.proj file

<PropertyGroup>
        <tfTool>"C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\tf.exe"</tfTool>
        ...
<Target Name="BeforeCompile" Condition=" '$(IsDesktopBuild)' != 'true' ">
        <Message Text="Beginning generation of new AssemblyFileVersionAttribute" />
        <Exec Command="$(tfTool) checkout ..\sources\src\StandardBuildForCIVersion.txt"  />
        <BuildVersionNumberManagerTask
            VersionFileName="..\sources\src\StandardBuildForCIVersion.txt"
            BuildLabel="$(BuildNumber)">
            <Output TaskParameter="NewVersionNumber" PropertyName="NewVersionNumber" />
        </BuildVersionNumberManagerTask>

I override the BeforeCompile task, since I need to change version number before the compile phase. As you can see I use the Exex command to call the tf.exe tool to checkout the file StandardBuildForCIVersion.txt, the one used to store autogenerated numbers. In this way I can use a different file for each build definition or I can use same file for different builds, I have great flexibility. The checkout is needed because I want to check in modified file at the end of the process, in this way the autogenerated number can be accessed from other build agents. Then I need another custom task capable to modify the ProjectVersion.cs file.

public class AssemblyInfoVersionManagerTask : Task
{
    [Required]
    public String AssemblyInfoFileName { get; set; }

    [Required]
    public Int32 NewBuildNumber { get; set; }

    [Required]
    public Int32 NewRevisionNumber { get; set; }


    public override bool Execute()
    {
        if (!File.Exists(AssemblyInfoFileName))
        {
            Log.LogError("The file {0} does not exists", AssemblyInfoFileName);
            return false;
        }
        String filecontent = File.ReadAllText(AssemblyInfoFileName);
        Match curVersionMatch =
            Regex.Match(filecontent, "AssemblyVersion\\(\"(?<curversion>.*?)\"\\)");
        if (!curVersionMatch.Success)
        {
            Log.LogError("The content of file {0} does not contains valid Assemblyversion attribute");
            return false;
        }
        String curversion = curVersionMatch.Groups["curversion"].Value;
        String[] versionParts = curversion.Split('.');
        String newVersion = String.Format("{0}.{1}.{2}.{3}", versionParts[0], versionParts[1], NewBuildNumber, NewRevisionNumber);
        String newFileContent = Regex.Replace(
            filecontent,
            "AssemblyFileVersion\\(\"(?<curversion>.*?)\"\\)",
            String.Format(@"AssemblyFileVersion(""{0}"")", newVersion));
        FileAttributes currentFileAttributes = File.GetAttributes(AssemblyInfoFileName);
        File.SetAttributes(AssemblyInfoFileName, FileAttributes.Normal);
        File.WriteAllText(AssemblyInfoFileName, newFileContent);
        File.SetAttributes(AssemblyInfoFileName, currentFileAttributes);
        return true;
    }
}

This is another simple task, it uses a little bit of regular expression to find actual version number stored in the ProjectVersion.cs, then it creates another version number with simple composition. Major and minor number are taken from the original content, while build and revision are passed by the caller. Remember also to remove the Readonly attribute from the file, because it is usually readonly since it is under version control. Now I need the last piece, a simple TfTask that permits me to grab the changeset number associated with the current project.

public class TfTask : ToolTask
{
    public TfTask()
    {
        ToolPath = @"C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\";
    }

    [Required]
    public String Operation { get; set; }

    [Output]
    public String TfOutput { get; set; }

    public String TfsUrl { get; set; }

    public String TeamProject { get; set; }

    protected override int ExecuteTool(string pathToTool, string responseFileCommands, string commandLineCommands)
    {
        TfTaskOperation operation = (TfTaskOperation)Enum.Parse(typeof(TfTaskOperation), Operation);
        String commandline;
        switch (operation)
        {
            case TfTaskOperation.GetLatestChangeset:
                commandline = String.Format(
                    "history /s:{0} /stopafter:1 /noprompt /recursive /version:T $/{1}", TfsUrl, TeamProject);
                break;
            default:
                throw new NotSupportedException();
        }
        using (System.Diagnostics.Process process = new System.Diagnostics.Process())
        {
            process.StartInfo.FileName = Path.GetFullPath(pathToTool);
            process.StartInfo.Arguments = commandline;
            process.StartInfo.WorkingDirectory = Path.GetDirectoryName(pathToTool);
            process.StartInfo.WindowStyle = ProcessWindowStyle.Normal;
            process.StartInfo.UseShellExecute = false;
            //process.StartInfo.ErrorDialog = false;
            //process.StartInfo.CreateNoWindow = true;
            process.StartInfo.RedirectStandardOutput = true;
            process.Start();
            process.WaitForExit();
            TfOutput = process.StandardOutput.ReadToEnd();
        }
        return 0;
    }

    protected override string GenerateFullPathToTool()
    {
        throw new NotImplementedException();
    }

    protected override string ToolName
    {
        get { return "tf.exe"; }
    }
}

public enum TfTaskOperation
{
    GetLatestChangeset = 0,
}

This is a simple wrapper to the tf.exe tool, and since I invoke it with the System.Diagnostic.Process class, I’m able to intercept the output. There are a lot of possibilities on how to get latest changeset, but the simplest is to invoke tf.exe with a command line like this: tf.exe history /s:http://tfsalkampfer:8080 /stopafter:1 /noprompt /recursive /version:T $/MsBuildExtension

This This command gives a result like this one.

Changeset User          Date       Comment
--------- ------------- ---------- ---------------------------------
129       TfsService    8/21/2009  ***NO_CI***

That can be parsed with a simple regular expression, here is the remaining of the build script.

<TfTask 
    TfsUrl="$(TeamFoundationServerUrl)" 
    TeamProject="$(TeamProject)" 
    Operation="GetLatestChangeset">
    <Output    TaskParameter="TfOutput" PropertyName="historyOutput"/>
</TfTask>

<Message Text="historyOutput is $(historyOutput)" />

<RegexTask TextToMatch="$(historyOutput)" Regex="\n(?&lt;changeset&gt;\d+)" FailIfNoMatch="true" GroupName="changeset">
    <Output
         TaskParameter="Result"
         PropertyName="lastChangeset"/>
</RegexTask>

<AssemblyInfoVersionManagerTask
    AssemblyInfoFileName="..\sources\src\ProjectVersion.cs"
    NewBuildNumber="$(NewVersionNumber)"
    NewRevisionNumber="$(lastChangeset)">
</AssemblyInfoVersionManagerTask>

<Exec Command='$(tfTool) checkin /comment:"***NO_CI***" ..\sources\src\StandardBuildForCIVersion.txt' />

IT is quite simple, I first invoke the TfTask, then I use a simple RegexTask to find the changeset number from tf output. Thanks to TfTask custom task, the script remains simple, I need only to specify tfsurl, team project name and the operation I want to execute (in this situation GetLatestChangeset), and the task will store in the historyOutput property the full output of the tf.exe command.

Finally I use my AssemblyInfoVersionManager task to change the ProjectVersion.cs file and finally I do a check-in of the file that contains the new generated sequential build number with comment ***NO_CI*** to avoid going in loop with continuos integration engine.

After a build you can verify that everything is gone ok.

image

Now if you deploy this assembly into a customer computer, if you have a problem in the future, you can immediately verify that this was compiled with the build called 8, then you check the StandardBuildForCIVersion.txt

7|Standard build for CI_20090820.37
8|Standard build for CI_20090820.39
9|Standard build for CI_20090821.2

You can immediately find the label of the build, but the most important thing in my opinion is the revision number, 128 in the picture above, because it is the changeset that generates this assembly, to replicate and debug any problem, you can simply do a Get Specific Version of that changeset, and you can work with the exact source code and build tools that generates that assembly.

This example shows how simple can be extending build process with the creation of some ad-hoc tasks and a good use of the tf.exe command line tool.

A zip of all the repository can be downloaded here.

Alk.

Tags: