How to deal with Slow Unit Tests with Visual Studio Test Runner

The problem

 

One of the most dreadful problem of Unit Testing is slow testing. If your whole suite of tests runs in 10 minutes, it is normal for developers not to run the whole suite at each build. One of the most common question is

How can I deal with slow Unit Tests?

Here is my actual scenario: in a project I’m working in, we have some multilingual full text search done in Elastic Search and we have a battery of Unit Tests that verify that searches work as expected. Since each test deletes all documents, insert a bunch of new documents and finally commits lucene index, execution times is high compared to the rest of tests. Each test need almost 2 seconds to run on my workstation, where I have really fast SSD and plenty of RAM.

This kind of tests cannot be run in memory or with some fancy trick to make then run quickly. Actually we have about 30 tests that executes in less than one seconds, and another 13 tests that runs in about 23 seconds, this is clearly unacceptable. After few hours of work, we already reached the point where running the whole suite becomes annoying.

The solution

 

This is a real common problem and it is quite simple to fix. First of all Visual Studio Test runner actually tells you execution time for each Unit Test, so you can immediately spot slow tests. When you identify slow tests you can mark them with a specific category, I use slowtest

    [TestFixture]
    [Category("elasticsearch")]
    [Category("slowtest")]
    public class EsSearcherFixture : BaseTestFixtureWithHelpers

Since I know in advance that this test are slow I immediately mark the entire class with the attribute slowtest. If you have no idea what of your tests are slow, I suggest grouping test by Duration in Visual Studio Test Runner.

image

Figure 1: Group tests by Duration

The result is interesting, because Visual Studio consider every test that needs more than one second to be slow. I tend to agree with this distinction.

image

Figure 2: Test are now grouped by duration

This permits you to immediately spot slow tests, so you can add the category slowtest to them. If you keep your Unit Tests organized and with a good usage of categories, you can simply ask VS Test Runner to exclude slow test with filter –Traits:”slowtest”

image

Figure 3: Thanks to filtering I can now execute continuously only test that are not slow.

I suggest you to do a periodic check to verify that every developers is using the slowtest category wisely, just group by duration, filters out the slowtest and you should not have no tests that are marked slow.

image

Figure 4: Removing the slowtest category and grouping by duration should list no slow test.

The nice part is that I’m using NUnit, because Visual Studio Test Runner supports many Unit Tests Frameworks thanks to the concepts of Test Adapters.

If you keep your tests well organized you will gain maximum benefit from them :).

Gian Maria.

Shim constructors to isolate “Future Objects”

I’ve already blogged previously about the new Shim Library of Visual Studio 2012

Now it is time to explore another scenario where shim can save your life. Suppose you are working with Hardware, Es. a barcode reader and you have a class in your system wrote in this way.

public class BarcodeReader
{
    public BarcodeReader() { 
        //connect to the hardware, if the hardware is not there 
        //throw an exception
    }

    public String ReadCode() 
    {
        //dialogate with the hardware and get the Barcode
    }
}

This class manage the connection to a physical BarcodeReader in the constructor and if something went wrong throws an exception and expose a ReadCode method that ask the Physical reader to read a Barcode. This class is used with the following pattern throughout all the software.

public class MyLogic
{
    private BarcodeReader reader;

    public MyLogic() {
        reader = new BarcodeReader();
    }

    public String RetrieveIndustrialCode()
    {
        String rawCode = reader.ReadCode();
        return rawCode.Substring(2, 10);
    }
}

MyLogic is a class with some business logic based on the Barcode returned from the call to ReadCode() method, as you can see it simply instantiate an instance of BarcodeReader in the constructor and read the code in the RetrieveIndustrialCode Method used to extract the industrial code from the raw content of the barcode. In this stupid example the algorithm to extract the Industrial Code from the Raw barcode read from the reader is simply reading chars from 2 to 10, but in real scenario we could have complex string parsing code that needs to be tested, but how?? As you can see from the code you can test only when the real hardware is connected to the machine because if a physical reader is not connected the BarcodeReader will throw exception, but the most annoying part is that it is impossible to simulate a read during a Unit Test.

This is another scenario where shims comes to the rescue, just add a fakes assembly (as described in the previous samples) and you can write this test.

[Fact]
public void verify_retrieval_of_industrialCode()
{
    using (ShimsContext.Create())
    {
        ShimBarcodeReader.Constructor = @this =>  new ShimBarcodeReader(@this) {
            ReadCode = () => "IC1234567890BLABLABLA",
        };
        MyLogic sut = new MyLogic();
        var industrialCode = sut.RetrieveIndustrialCode();
        Assert.Equal("1234567890", industrialCode);
    }
}

Whenever you fake an assembly, each Shim that gets created has one or more static property called ConstructorXXX, where the XXX part is used to distinguish from the overloaded version using paramers type. In this example I have a default constructor so my Shim has a static property called Constructor. This specific property permits you to intercept the future creation of that type and even substitute it with a shim. This specific property needs a lambda where the first parameter is the Instance of the object being created followed by all other parameters of the constructor. In my sample test I pass a lambda that based on the instance created, returns a new ShimBarcodeReader configured to return the code “IC1234567890BLABLABLA” from ReadCode() method.

This technique is really powerful because permits you to Shim every future instance that will be created for a specific object and makes possible to inject Shims almost everywhere. When the test run, the MyLogic constructor creates an instance of BarcodeReader, but thanks to the Shim library I’m able to pass a shim where I’ve already set the return value of ReadCode() method.

Thanks to this capability I can write tests until I eventually found a bug, to show how the Shim Constructor changes when constructor has parameters, I’ve changed the BarcodeReader class to make the constructor accepts a Single String parameter and I want to write a test that verify that when the BarcodeReader returns an Empty String the Industrial code returned is also Empty.

[Fact]
public void verify_retrieval_of_industrialCode_too_short()
{
    using (ShimsContext.Create())
    {
        ShimBarcodeReader.ConstructorString = (@this, value) => new ShimBarcodeReader(@this)
        {
            ReadCode = () => "",
        };
        MyLogic sut = new MyLogic("COM2");
        var industrialCode = sut.RetrieveIndustrialCode();
        Assert.Equal("", industrialCode);
    }
}

To shim the creation of a ShimBarcodeReader I now need to use ConstructorString property, since the constructor has now a parameter of type String. Thanks to Visual Studio 2012 and Shim Library you can now easily test untestable legacy code with minimum effort.

Gian Maria.

Shim and InstanceBehavior fallthrough to isolate part of the SUT

I’ve dealt in a previous post with the new Shim library in Vs11 that permits you to test “difficult to test code” and I showed a really simple example on how to use Shim to isolate the call to DateTime.Now to simulate passing time in a Unit Test. Now I want to change a little bit the perspective of the test, in the test showed in previous post I simply exercise the sut calling Execute() a couple of time, simulating the time that pass between the two calls. Here is the test

[Fact]
public void Verify_do_not_execute_task_if_interval_is_not_elapsed()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        DateTime startDate = new DateTime(2012, 1, 1, 12, 00, 00);
        ShimDateTime.NowGet = () =>  startDate;
        sut.Execute();
        ShimDateTime.NowGet = () => startDate.AddMinutes(9);
        sut.Execute();
        Assert.Equal(1, callCount);
    }
}

I can also change the point of view and write a test that uses a shim to isolate the SUT, this is a less common scenario but it can be also really interesting, because it shows you how you can write simple White box Unit Tests isolating part of the SUT. The term White Box is used because this kind of Unit Test are created with a full knowledge of the internal structure of the SUT, in my situation I have a private method called CanExecute() that return true/false based on the interval of time passed from the last execution and since it is private it makes difficult for me to test the SUT.

private Boolean CanExecute() {

    return DateTime.Now.Subtract(lastExecutionTime)
        .TotalMinutes >= intervalInMinutes;
}

But I can create a Shime on the SUT and isolate calls to the CanExecute(), making it return the value I need for the test, here is an example

[Fact]
public void Verify_can_execute_is_honored()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        ShimPerformHeavyTask shimSut = new ShimPerformHeavyTask(sut);
        shimSut.InstanceBehavior = ShimBehaviors.Fallthrough;
        shimSut.CanExecute = () => false;
        sut.Execute();
        Assert.Equal(0, callCount);
    }
}

To write this test I’ve added another fake assembly on the assembly that contains the PerformHeavyTask class to create shim for the SUT. This test basically create a ShimPerformHeavyTask (a shim of my SUT) passing an existing instance to the SUT to the constructor of the shim, then I set the InstanceBehavior to ShimBehaviors.Fallthrough to indicate to the Shim Library to call original SUT method if the method was not isolated. At this point I can simply isolate the call to the CanExecute() private and non-virtual method, specifying to the shim to return the value false, then I call the Execute() method and verify that the heavy task is not executed.

This test shows how to create a shim of the SUT to isolate calls to its private methods, thanks to the Fallthrough behavior; if you forget to change the InstanceBehavior the test will fail with an exception of type ShimNotImplementedException, because the default behavior for a Shim is to throw an exception for any method that is not intercepted. Thanks to shim library you can simply isolate every part of the SUT, making easier to write Unit Test for classes written with no TDD and no Unit Testing in mind.

Gian Maria.

Traffic light vNext

It is a long time I did not post about simple Traffic Light experiment. I’ve ended with a super simple Domain with no Getters and no Setters, but there is still something I really do not like about that sample and it is represented by this test.

[Fact]
public void When_both_semaphore_are_red_one_become_green()
{

    //move all to yellow fixed
    using (DateTimeService.OverrideTimeGenerator(DateTime.Now.AddSeconds(1)))
    {
        sut.Tick();
    }
    _domainEvents.Clear();
    //now move again to both red
    using (DateTimeService.OverrideTimeGenerator(DateTime.Now.AddSeconds(10)))
    {
        sut.Tick();
    }
    _domainEvents.Should().Have.Count.EqualTo(2);
    _domainEvents.All(e => e.NewStatus == LightColor.Red).Should().Be.True();
    _domainEvents.Clear();
    using (DateTimeService.OverrideTimeGenerator(DateTime.Now.AddSeconds(1)))
    {
        sut.Tick();
    }
    //only one semaphore should be go to green state
    _domainEvents.Should().Have.Count.EqualTo(1);
    _domainEvents[0].NewStatus.Should().Be.EqualTo(LightColor.Green);
}

This test is quite ugly and it requires some initialization code, contained in the constructor of the test class.

public TwoTrafficLightFixture()
{
    DomainEvents.ClearAllRegistration();
    CommunicationBus.ClearAllRegistration();

    DomainEvents.Register<ChangedLightStatus>(this, e => _domainEvents.Add(e));
    sut = CrossRoadFactory.For(2.Roads()).Create();
    sut.Start();
}

The bad part about this code is that I want to test this situation: when both traffic light are in red state, only one of them can become green after a given amount of time passed. The awful part about this test is how I setup the fixture; to bring both the Traffic Light in the Red state, I need to create the CrossRoadFactory in the constructor of the test (this is because all tests share this common initialization), then I need to call Tick() several time simulating passing time and moving the system from the initial status to the status that represents the fixture (both light red).

This test is simply wrong, because if if fails you cannot tell if the failure is caused by the initialization code or the real part of the domain logic you want to test, because I’m actually exercising the SUT until it reach the status I want to test with my unit test and this can cause a failure in the Fixture. This is done because if the Traffic Light has no public properties and it is not possible to manipulate the status directly in the test.

To simplify the test I need a way to change the status of a Traffic Light, so I can simply fixture creation and the obvious solution is to use Event Sourcing. I do not want to evolve the Traffic Light to a full Event Sourcing enabled domain, but I wish to verify if having the ability to reconstruct the state of a Domain Object from domain events he raised in the past can solve my test smell. I decided to add a constructor on the TrafficLight domain class that permits to create an instance from a sequence of Domain Events.

public TrafficLight(params BaseEvent[] eventStream)
{
    ActualState = YellowBlinkingState.Instance();
    CommunicationBus.Register<MayITurnGreen>(this, MayITurnGreen);
    Load(eventStream);
}

public void Load(params BaseEvent[] eventStream)
{
    foreach (var eventInstance in eventStream)
    {
        HandleChangedLightStatus(eventInstance as ChangedLightStatus);
    }
}

This is really primitive, it is really far from being a real entity based on Event Sourcing, but the key concept is, I want to be able to reconstruct the private state of an entity simply passing a series of domain events that he raised in the past. Now the previous test can be modified to make it really simpler:

[Fact]
public void When_both_semaphore_are_red_one_become_green()
{

    TrafficLight first = new TrafficLight(new ChangedLightStatus(LightColor.Red, null));
    TrafficLight second = new TrafficLight(new ChangedLightStatus(LightColor.Red, null));
    CrossRoad theSut = new CrossRoad(
        new TrafficLightCreated(first),
        new TrafficLightCreated(second));

    using (DateTimeService.OverrideTimeGenerator(DateTime.Now.AddSeconds(1)))
    {
        theSut.Tick();
    }

    //only one semaphore should be go to green state
    _domainEvents.Should().Have.Count.EqualTo(1);
    _domainEvents[0].NewStatus.Should().Be.EqualTo(LightColor.Green);
}

Now I’m able to create two traffic light passing a ChangedLightStatus event that bring the Traffic Light to the status requested by the fixture, then I can create the CrossRoad class passing two TrafficLightCreated domain events and this is all the code I need to setup the fixture of my test. Now I can simulate that one second is passed, call Tick() function on the CrossRoad and verify that only one Domain Event is raised, because only one Traffic Light Should have changed the value of the light from Red to Green.

Even with this super simple example you can understand that Event Sourcing simplify your tests, because they give you the ability to recreate a specific state of the domain under test from a stream of Domain Events.

Gian Maria.

Using Shims in Visual Studio 11 to test untestable code.

I always strive myself to write testable code, but it is not always easy, especially if you do not follow TDD red-green-refactor mantra. Code written without Unit Testing in mind is usually not so easy to test and when is time to modify code written by other, if you want to create a safety net of Unit Tests to avoid breaking code it is usually an hard task. The main problem is hardcode dependency from static functions because you cannot test a single function or method in isolation, forcing you to use pattern like Back door manipulation that makes test difficult to read, write and maintain. Lets take this code as example

    public class PerformHeavyTask 
    {
        private DateTime lastExecutionTime = new DateTime(1900, 1, 1);

        private Int32 intervalInMinutes;

        private Action task;

        public PerformHeavyTask(Int32 intervalInMinutes, Action task) {

            this.intervalInMinutes = intervalInMinutes;
            this.task = task;
        }

        private Boolean CanExecute() {

            return DateTime.Now.Subtract(lastExecutionTime).TotalMinutes >= intervalInMinutes;
        }

        public void Execute() {
            if (CanExecute())
            {
                lastExecutionTime = DateTime.Now;
                task();
            }
        }
    }

This is a simple and stupid example class that is used to execute an action no more than once every X minutes. The main problem is dependency from the DateTime.Now static function because it create a dependency to the concept of “passing time” that is almost impossible to manage in a test. Suppose that the logic of CanExecute method is complex and depends not only from DateTime.Now or uses some strange algorithm based on time, how can you write a unit test that is capable of testing this algorithm simulating the time that pass? The only solution is being able to change at runtime the behavior of the DateTime.Now static function, a trick that is possible using various external library, but that is now natively available in Visual Studio 11.

This functionality is derived from Pex and Moles project, but now it is fully integrated in VS11, with small differences. First of all you need to right click the reference to the system assembly and choose to Add Fakes Assembly; this creates a special folder called Fakes, with a single file called system.fakes.

26-04-2012 19-40-41

Figure 1: Adding a fake assembly for the System assembly.

Now you can write this simple test that is capable of changing the value of the DateTime.Now static property at runtime

[Fact]
public void Verify_do_not_execute_task_if_interval_is_not_elapsed()
{
    using (ShimsContext.Create())
    {
        Int32 callCount = 0;
        PerformHeavyTask sut = new PerformHeavyTask(10, () => callCount++);
        DateTime startDate = new DateTime(2012, 1, 1, 12, 00, 00);
        ShimDateTime.NowGet = () =>  startDate;
        sut.Execute();
        ShimDateTime.NowGet = () => startDate.AddMinutes(9);
        sut.Execute();
        Assert.Equal(1, callCount);
    }
}

As you can see this code is written in xUnit because VS11 is capable of running Unit Test from external framework if there is the corresponding adapter, so I’m not restricted to use MSTest if I want to use some UnitTesting specific feature of Visual Studio. The first interesting line is the ShimsContext.Create() call, used to create a scope where we can use Shims and isolate calls to non-virtual functions. The concept of Isolation is being able to change how a static property or method behave without the need to change the original code. As you can see I create an instance of the PerformHeavyTask class, with 10 minutes interval and with a function that basically only increment a local variable to have a count of how many times the function is executed. Now to test that the function is not called a second time if not enough time is passed I need to isolate the call to DateTime.Now to return predetermined values.

The line ShimDateTime.NowGet = … permits me to intercepts the getter of the static Now property of DateTime Class and specify the function that will be used instead of standard getter. Shim library works with some convention and since I’ve created a Fake Assembly of the standard System reference, the Shim library will create for me a Shim class for every type in the Assembly prepending the world Shim to the original name. Since I need to change the behavior of the Now static property of DateTime class, I need to use the ShimDateTime class (created for me with the Add fakes assembly command) and use the NowGet property to isolate the Getter of the static property Now. To isolate the getter I simply specify the lambda function to call whenever any code calls DateTime.Now.

The test simple calls the Execute() method the first time, then change isolation function to returns a DateTime that is 9 minutes greater than the previous value, finally I call the Execute() method again and verify that the taks passed to PerformHeavyTask was executed only 1 time, because when I invoked Execute() the second time not enough time has passed and the function should not be executed.

Gian Maria.