Is it possible to run Coded Ui Test in a Virtual Machine instead that in my primary machine?

Code UI Test are an effective way to test your UI through automation, they absolutely are not a substitution for Unit Testing, but they can be used effectively to verify the whole application behave as expected. The most annoying part of Coded UI Test is that while they are running you cannot do anything else on the computer, because mouse and UI are used to run the test. This lead to a very common question

Is it possible to run Coded Ui Test in a Virtual Machine instead that in my primary machine?

The answer is yes, and it is a really simple task to accomplish. First of all install a Test controller on a machine in your network (or a virtual machine if you want to run test inside a local VM) and configure it.

image

Figure 1: Configure user used to run the service and nothing more 

You need only to specify the account used, and you should not register it to a Team Project Collection, just leave any other option blank. Then you need to install Test Agent on the same machine or to another one, and configure that test agent to connect to the previous installed test controller.

image

Figure 2: Configure the agent to log on automatically and connect to the test controller

Now you should see that the Agent component connects to the controller and the Test Agent Status is online

image

Figure 3: The test agent is online and connected to the controller.

It is time to go back to Visual Studio, right-click on the solution name and add new item, from the list of templates choose Test Settings

image

Figure 4: Add a test settings to the solution

Even if in VS 2012 test settings are not used by default as in VS2010 they are still there. In VS 2012 you do not usually need a Test Settings file, unless you want to run tests with some non standard configuration, as in this example. Once you have added a Test Settings file you can edit it and choose to execute test remotely specifying the name of the machine that has the Test Controller installed.

image

Figure 5: Choose to run test with Remote Execution

Now you should simply save the file, and choose as the active test settings browsing for the file

image

Figure 6: Activate test settings

Now if you run again the Coded UI Test it will be run in the machine with the test agent.

image

Figure 7: The VM with test agent runs tests for me

Thanks to this configuration you can execute any Unit Test in a remote machine, not only CUIT.

Gian Maria.

Code coverage during TFS 2012 build

A common question about TFS 2012 build is: how can I ask for code coverage?

The answer is really simple, you should go to the test section of build definition and change the run settings from Default to another setting that includes code coverage and this is really everything you need to do.

image_thumb2

Figure 1: Change run settings to enable code coverage

Now launch the build and verify that you have your binaries instrumented and Code Coverage correctly computed.

image_thumb5

Usually this lead to another question: Can I decide which assembly to include in code coverage instead of having all of them instrumented?

The solution to this answer is creating a customized test run setting file, the only drawback is that you have no template inside Visual Studio and no editor (as you had in VS 2010 with TestSettings files). The good news is that you can simply add an xml file inside your solution, give to it the extension .runsettings and copy inside it this simple fragment of configuration.

<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <!-- Match assembly file paths: -->
            <ModulePaths>
              <Include>
                <ModulePath>.*fabrikamfiber.web.dll$</ModulePath>
              </Include>

            </ModulePaths>

            <!-- We recommend you do not change the following values: -->
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>

          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

You have lots of options, but in this example I’m interested only in limiting the number of assemblies that will be instrumented for Code Coverage. This can be done simply specifying one or more regular expression in the ModulePaths section. You can use Include to define regex for inclusion, or you can use exclude to specify which assemblies will be excluded.

Once you authored this file, you should use it as active test setting configuration to test if everything is good, and verify that after running all of your test locally, Code Coverage result is what you expected. Remember that once you’ve have this run settings activated, it will be used for all of your local test run.

image_thumb[2]

Once everything is ok you need to store this file in source control, check-in and modify Build configuration to use this customized file as a Run Settings File for the build and the game is done.

image_thumb[5]

Now fire the build again, this time only selected assemblies will be used for code coverage.

image_thumb[8]

MSDN Link to test run setting file: Customizing Code Coverage Analysis

Gian Maria.

Shim and InstanceBehavior fallthrough to isolate part of the SUT

I’ve dealt in a previous post with the new Shim library in Vs11 that permits you to test “difficult to test code” and I showed a really simple example on how to use Shim to isolate the call to DateTime.Now to simulate passing time in a Unit Test. Now I want to change a little bit the perspective of the test, in the test showed in previous post I simply exercise the sut calling Execute() a couple of time, simulating the time that pass between the two calls. Here is the test

[Fact]
public void Verify_do_not_execute_task_if_interval_is_not_elapsed()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        DateTime startDate = new DateTime(2012, 1, 1, 12, 00, 00);
        ShimDateTime.NowGet = () =>  startDate;
        sut.Execute();
        ShimDateTime.NowGet = () => startDate.AddMinutes(9);
        sut.Execute();
        Assert.Equal(1, callCount);
    }
}

I can also change the point of view and write a test that uses a shim to isolate the SUT, this is a less common scenario but it can be also really interesting, because it shows you how you can write simple White box Unit Tests isolating part of the SUT. The term White Box is used because this kind of Unit Test are created with a full knowledge of the internal structure of the SUT, in my situation I have a private method called CanExecute() that return true/false based on the interval of time passed from the last execution and since it is private it makes difficult for me to test the SUT.

private Boolean CanExecute() {

    return DateTime.Now.Subtract(lastExecutionTime)
        .TotalMinutes >= intervalInMinutes;
}

But I can create a Shime on the SUT and isolate calls to the CanExecute(), making it return the value I need for the test, here is an example

[Fact]
public void Verify_can_execute_is_honored()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        ShimPerformHeavyTask shimSut = new ShimPerformHeavyTask(sut);
        shimSut.InstanceBehavior = ShimBehaviors.Fallthrough;
        shimSut.CanExecute = () => false;
        sut.Execute();
        Assert.Equal(0, callCount);
    }
}

To write this test I’ve added another fake assembly on the assembly that contains the PerformHeavyTask class to create shim for the SUT. This test basically create a ShimPerformHeavyTask (a shim of my SUT) passing an existing instance to the SUT to the constructor of the shim, then I set the InstanceBehavior to ShimBehaviors.Fallthrough to indicate to the Shim Library to call original SUT method if the method was not isolated. At this point I can simply isolate the call to the CanExecute() private and non-virtual method, specifying to the shim to return the value false, then I call the Execute() method and verify that the heavy task is not executed.

This test shows how to create a shim of the SUT to isolate calls to its private methods, thanks to the Fallthrough behavior; if you forget to change the InstanceBehavior the test will fail with an exception of type ShimNotImplementedException, because the default behavior for a Shim is to throw an exception for any method that is not intercepted. Thanks to shim library you can simply isolate every part of the SUT, making easier to write Unit Test for classes written with no TDD and no Unit Testing in mind.

Gian Maria.

Using Shims in Visual Studio 11 to test untestable code.

I always strive myself to write testable code, but it is not always easy, especially if you do not follow TDD red-green-refactor mantra. Code written without Unit Testing in mind is usually not so easy to test and when is time to modify code written by other, if you want to create a safety net of Unit Tests to avoid breaking code it is usually an hard task. The main problem is hardcode dependency from static functions because you cannot test a single function or method in isolation, forcing you to use pattern like Back door manipulation that makes test difficult to read, write and maintain. Lets take this code as example

    public class PerformHeavyTask 
    {
        private DateTime lastExecutionTime = new DateTime(1900, 1, 1);

        private Int32 intervalInMinutes;

        private Action task;

        public PerformHeavyTask(Int32 intervalInMinutes, Action task) {

            this.intervalInMinutes = intervalInMinutes;
            this.task = task;
        }

        private Boolean CanExecute() {

            return DateTime.Now.Subtract(lastExecutionTime).TotalMinutes >= intervalInMinutes;
        }

        public void Execute() {
            if (CanExecute())
            {
                lastExecutionTime = DateTime.Now;
                task();
            }
        }
    }

This is a simple and stupid example class that is used to execute an action no more than once every X minutes. The main problem is dependency from the DateTime.Now static function because it create a dependency to the concept of “passing time” that is almost impossible to manage in a test. Suppose that the logic of CanExecute method is complex and depends not only from DateTime.Now or uses some strange algorithm based on time, how can you write a unit test that is capable of testing this algorithm simulating the time that pass? The only solution is being able to change at runtime the behavior of the DateTime.Now static function, a trick that is possible using various external library, but that is now natively available in Visual Studio 11.

This functionality is derived from Pex and Moles project, but now it is fully integrated in VS11, with small differences. First of all you need to right click the reference to the system assembly and choose to Add Fakes Assembly; this creates a special folder called Fakes, with a single file called system.fakes.

26-04-2012 19-40-41

Figure 1: Adding a fake assembly for the System assembly.

Now you can write this simple test that is capable of changing the value of the DateTime.Now static property at runtime

[Fact]
public void Verify_do_not_execute_task_if_interval_is_not_elapsed()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        DateTime startDate = new DateTime(2012, 1, 1, 12, 00, 00);
        ShimDateTime.NowGet = () =>  startDate;
        sut.Execute();
        ShimDateTime.NowGet = () => startDate.AddMinutes(9);
        sut.Execute();
        Assert.Equal(1, callCount);
    }
}

As you can see this code is written in xUnit because VS11 is capable of running Unit Test from external framework if there is the corresponding adapter, so I’m not restricted to use MSTest if I want to use some UnitTesting specific feature of Visual Studio. The first interesting line is the ShimsContext.Create() call, used to create a scope where we can use Shims and isolate calls to non-virtual functions. The concept of Isolation is being able to change how a static property or method behave without the need to change the original code. As you can see I create an instance of the PerformHeavyTask class, with 10 minutes interval and with a function that basically only increment a local variable to have a count of how many times the function is executed. Now to test that the function is not called a second time if not enough time is passed I need to isolate the call to DateTime.Now to return predetermined values.

The line ShimDateTime.NowGet = … permits me to intercepts the getter of the static Now property of DateTime Class and specify the function that will be used instead of standard getter. Shim library works with some convention and since I’ve created a Fake Assembly of the standard System reference, the Shim library will create for me a Shim class for every type in the Assembly prepending the world Shim to the original name. Since I need to change the behavior of the Now static property of DateTime class, I need to use the ShimDateTime class (created for me with the Add fakes assembly command) and use the NowGet property to isolate the Getter of the static property Now. To isolate the getter I simply specify the lambda function to call whenever any code calls DateTime.Now.

The test simple calls the Execute() method the first time, then change isolation function to returns a DateTime that is 9 minutes greater than the previous value, finally I call the Execute() method again and verify that the taks passed to PerformHeavyTask was executed only 1 time, because when I invoked Execute() the second time not enough time has passed and the function should not be executed.

Gian Maria.

Running NUnit and xUnit tests in TFS11 build

I’ve blogged in the past various solution to run NUnit tests during a TFS build, and now it is time to make it again for TFS11, but this time it is incredibly simple, because the new Test Runner supports multiple frameworks, so it works almost automatically.

You can read from Peter Provost blog that actually we have three plugin for UTE (Unit Test Explorer) available: Nunit, xUnit and HTML/JAvascript, they are simple .vsix file (Visual Studio Extension), that you can download, run, and voilà, your xUnit and NUnit tests are runnable from Visual Studio.

image

Figure 1: The new UTE is able to run Unit Tests from multiple framework, because it is extendible

Now if you create a build against this solution, you probably will be disappointed by the fact that the build only runs MStest ignoring tests supported by an external plugin.

image

Figure 2: Build result shows that only 6 test were run, xUnit and NUnit tests are ignored

This blog post explain the reason, if you are on 64 bit machine, the vsix installer is not able to make the Extension visible to build controller, so you need to do a little extra step. First of all locate the folder of the two extensions under the plugin for the current user

image

Figure 3: UTE plugins are located in the standard plugin folder for Visual Studio

This location may vary, in my system it installed only for current user so they are on my profile folder, if you do not find the extension there you can simply search for the dll NUnit.VisualStudio.TestAdapter.dll into your Hard Disk. Once you found xUnit and NUnit UTE plugin folder, you should copy all dll inside a source controlled folder.

image

Figure 4: All UTE plugin assemblies are now in a source controlled folder

Finally go to Team Explorer –> Build –> Actions –> Manage Build Controllers and specify that all custom assemblies are located inside that folder

image

Figure 5: Configure the test controller specifying the location of all the assemblies that contains build extension

Now you can queue another build, and this time all tests should be run.

image

Figure 6: Now all 16 tests were run

As you can see, with TFS and VS11 running unit test from various Unit Test Frameworks is really easy.

Alk.