Quickly run Unit Test With VS 2012/2013/..

With VS 2012 and newer versions we can run Unit Tests from various frameworks directly from Visual Studio IDE, thanks to the concept of Test Adapters. When you are doing Test Driven Development you usually go with Red/Green/Refactor workflow; what you need is a way to quickly run all or part of your unit tests after you modified the code. The quickest solution is using the option to Run Tests After Build but it is available only for Premium and Ultimate edition, but you can also run test with little manual intervention resorting to Keyboard Shortcut.

Simply go to TOOLS->Customize menu, then choose to customize keyboard.

image

Figure 1: Keyboard customization in Visual Studio

Once configuration window is opened, search for TestExplorer.RunAllTests command, place cursor in the “Press shortcut keys:” textbox and press a shortcut, then press “assign” button to assign to this command. In my standard configuration I like to have CTRL+SHIFT+ALT+A shortcut, because it is not assigned to any other command and it is easy to press with left hand.

image

Figure 2: Assign CTRL+SHIFT+ALT+A to TestExplorer.RunAllTests command

Now I can write code, use shortcut and VS will build solution and run test for me automatically, without leaving my hands from the keyboard. Thanks to the various test adapter and the various grouping and filtering possibility offered by Test Explorer, you can do TDD in Visual Studio without the need of third party tools.

Gian Maria.

Nunit test not found for some assemblies with Visual Studio Test Runner

I’ve a project in Visual Studio 2013 where one of the assembly containing Tests refuses to show tests in Test Explorer window. The solution has tests written both in Nunit and in MSpec, and everything is good except for that specific assembly. If you notice that Test Explorer window misses some tests, the first thing you need to check is the output windows, where you can find some output for Test Adapters.

image

Figure 1: Output for test adapters can be found in standard Output Window

In the above situation the assembly that is ignored is Intranet.Tests.dll and the reason is that it is built for Framework45 and x64 platform, while the test runner is executing against x86 platform. Everything seems ok, every project is compiled in .ANY CPU, but looking at the raw project file I can confirm that PlatformTarget is set in x64. Changing to x86 (or removing it completely) solves the problem.

image

Figure 2: Platform target changed from x64 to x86

After I changed the PlatformTarget attribute, all tests belonging to that assembly are now available Test Explorer window.

Gian Maria.

How to deal with Slow Unit Tests with Visual Studio Test Runner

The problem

 

One of the most dreadful problem of Unit Testing is slow testing. If your whole suite of tests runs in 10 minutes, it is normal for developers not to run the whole suite at each build. One of the most common question is

How can I deal with slow Unit Tests?

Here is my actual scenario: in a project I’m working in, we have some multilingual full text search done in Elastic Search and we have a battery of Unit Tests that verify that searches work as expected. Since each test deletes all documents, insert a bunch of new documents and finally commits lucene index, execution times is high compared to the rest of tests. Each test need almost 2 seconds to run on my workstation, where I have really fast SSD and plenty of RAM.

This kind of tests cannot be run in memory or with some fancy trick to make then run quickly. Actually we have about 30 tests that executes in less than one seconds, and another 13 tests that runs in about 23 seconds, this is clearly unacceptable. After few hours of work, we already reached the point where running the whole suite becomes annoying.

The solution

 

This is a real common problem and it is quite simple to fix. First of all Visual Studio Test runner actually tells you execution time for each Unit Test, so you can immediately spot slow tests. When you identify slow tests you can mark them with a specific category, I use slowtest

    [TestFixture]
    [Category("elasticsearch")]
    [Category("slowtest")]
    public class EsSearcherFixture : BaseTestFixtureWithHelpers

Since I know in advance that this test are slow I immediately mark the entire class with the attribute slowtest. If you have no idea what of your tests are slow, I suggest grouping test by Duration in Visual Studio Test Runner.

image

Figure 1: Group tests by Duration

The result is interesting, because Visual Studio consider every test that needs more than one second to be slow. I tend to agree with this distinction.

image

Figure 2: Test are now grouped by duration

This permits you to immediately spot slow tests, so you can add the category slowtest to them. If you keep your Unit Tests organized and with a good usage of categories, you can simply ask VS Test Runner to exclude slow test with filter –Traits:”slowtest”

image

Figure 3: Thanks to filtering I can now execute continuously only test that are not slow.

I suggest you to do a periodic check to verify that every developers is using the slowtest category wisely, just group by duration, filters out the slowtest and you should not have no tests that are marked slow.

image

Figure 4: Removing the slowtest category and grouping by duration should list no slow test.

The nice part is that I’m using NUnit, because Visual Studio Test Runner supports many Unit Tests Frameworks thanks to the concepts of Test Adapters.

If you keep your tests well organized you will gain maximum benefit from them :).

Gian Maria.

Shim constructors to isolate “Future Objects”

I’ve already blogged previously about the new Shim Library of Visual Studio 2012

Now it is time to explore another scenario where shim can save your life. Suppose you are working with Hardware, Es. a barcode reader and you have a class in your system wrote in this way.

public class BarcodeReader
{
    public BarcodeReader() { 
        //connect to the hardware, if the hardware is not there 
        //throw an exception
    }

    public String ReadCode() 
    {
        //dialogate with the hardware and get the Barcode
    }
}

This class manage the connection to a physical BarcodeReader in the constructor and if something went wrong throws an exception and expose a ReadCode method that ask the Physical reader to read a Barcode. This class is used with the following pattern throughout all the software.

public class MyLogic
{
    private BarcodeReader reader;

    public MyLogic() {
        reader = new BarcodeReader();
    }

    public String RetrieveIndustrialCode()
    {
        String rawCode = reader.ReadCode();
        return rawCode.Substring(2, 10);
    }
}

MyLogic is a class with some business logic based on the Barcode returned from the call to ReadCode() method, as you can see it simply instantiate an instance of BarcodeReader in the constructor and read the code in the RetrieveIndustrialCode Method used to extract the industrial code from the raw content of the barcode. In this stupid example the algorithm to extract the Industrial Code from the Raw barcode read from the reader is simply reading chars from 2 to 10, but in real scenario we could have complex string parsing code that needs to be tested, but how?? As you can see from the code you can test only when the real hardware is connected to the machine because if a physical reader is not connected the BarcodeReader will throw exception, but the most annoying part is that it is impossible to simulate a read during a Unit Test.

This is another scenario where shims comes to the rescue, just add a fakes assembly (as described in the previous samples) and you can write this test.

[Fact]
public void verify_retrieval_of_industrialCode()
{
    using (ShimsContext.Create())
    {
        ShimBarcodeReader.Constructor = @this =>  new ShimBarcodeReader(@this) {
            ReadCode = () => "IC1234567890BLABLABLA",
        };
        MyLogic sut = new MyLogic();
        var industrialCode = sut.RetrieveIndustrialCode();
        Assert.Equal("1234567890", industrialCode);
    }
}

Whenever you fake an assembly, each Shim that gets created has one or more static property called ConstructorXXX, where the XXX part is used to distinguish from the overloaded version using paramers type. In this example I have a default constructor so my Shim has a static property called Constructor. This specific property permits you to intercept the future creation of that type and even substitute it with a shim. This specific property needs a lambda where the first parameter is the Instance of the object being created followed by all other parameters of the constructor. In my sample test I pass a lambda that based on the instance created, returns a new ShimBarcodeReader configured to return the code “IC1234567890BLABLABLA” from ReadCode() method.

This technique is really powerful because permits you to Shim every future instance that will be created for a specific object and makes possible to inject Shims almost everywhere. When the test run, the MyLogic constructor creates an instance of BarcodeReader, but thanks to the Shim library I’m able to pass a shim where I’ve already set the return value of ReadCode() method.

Thanks to this capability I can write tests until I eventually found a bug, to show how the Shim Constructor changes when constructor has parameters, I’ve changed the BarcodeReader class to make the constructor accepts a Single String parameter and I want to write a test that verify that when the BarcodeReader returns an Empty String the Industrial code returned is also Empty.

[Fact]
public void verify_retrieval_of_industrialCode_too_short()
{
    using (ShimsContext.Create())
    {
        ShimBarcodeReader.ConstructorString = (@this, value) => new ShimBarcodeReader(@this)
        {
            ReadCode = () => "",
        };
        MyLogic sut = new MyLogic("COM2");
        var industrialCode = sut.RetrieveIndustrialCode();
        Assert.Equal("", industrialCode);
    }
}

To shim the creation of a ShimBarcodeReader I now need to use ConstructorString property, since the constructor has now a parameter of type String. Thanks to Visual Studio 2012 and Shim Library you can now easily test untestable legacy code with minimum effort.

Gian Maria.

Shim and InstanceBehavior fallthrough to isolate part of the SUT

I’ve dealt in a previous post with the new Shim library in Vs11 that permits you to test “difficult to test code” and I showed a really simple example on how to use Shim to isolate the call to DateTime.Now to simulate passing time in a Unit Test. Now I want to change a little bit the perspective of the test, in the test showed in previous post I simply exercise the sut calling Execute() a couple of time, simulating the time that pass between the two calls. Here is the test

[Fact]
public void Verify_do_not_execute_task_if_interval_is_not_elapsed()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        DateTime startDate = new DateTime(2012, 1, 1, 12, 00, 00);
        ShimDateTime.NowGet = () =>  startDate;
        sut.Execute();
        ShimDateTime.NowGet = () => startDate.AddMinutes(9);
        sut.Execute();
        Assert.Equal(1, callCount);
    }
}

I can also change the point of view and write a test that uses a shim to isolate the SUT, this is a less common scenario but it can be also really interesting, because it shows you how you can write simple White box Unit Tests isolating part of the SUT. The term White Box is used because this kind of Unit Test are created with a full knowledge of the internal structure of the SUT, in my situation I have a private method called CanExecute() that return true/false based on the interval of time passed from the last execution and since it is private it makes difficult for me to test the SUT.

private Boolean CanExecute() {

    return DateTime.Now.Subtract(lastExecutionTime)
        .TotalMinutes >= intervalInMinutes;
}

But I can create a Shime on the SUT and isolate calls to the CanExecute(), making it return the value I need for the test, here is an example

[Fact]
public void Verify_can_execute_is_honored()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        ShimPerformHeavyTask shimSut = new ShimPerformHeavyTask(sut);
        shimSut.InstanceBehavior = ShimBehaviors.Fallthrough;
        shimSut.CanExecute = () => false;
        sut.Execute();
        Assert.Equal(0, callCount);
    }
}

To write this test I’ve added another fake assembly on the assembly that contains the PerformHeavyTask class to create shim for the SUT. This test basically create a ShimPerformHeavyTask (a shim of my SUT) passing an existing instance to the SUT to the constructor of the shim, then I set the InstanceBehavior to ShimBehaviors.Fallthrough to indicate to the Shim Library to call original SUT method if the method was not isolated. At this point I can simply isolate the call to the CanExecute() private and non-virtual method, specifying to the shim to return the value false, then I call the Execute() method and verify that the heavy task is not executed.

This test shows how to create a shim of the SUT to isolate calls to its private methods, thanks to the Fallthrough behavior; if you forget to change the InstanceBehavior the test will fail with an exception of type ShimNotImplementedException, because the default behavior for a Shim is to throw an exception for any method that is not intercepted. Thanks to shim library you can simply isolate every part of the SUT, making easier to write Unit Test for classes written with no TDD and no Unit Testing in mind.

Gian Maria.