How to deal with Slow Unit Tests with Visual Studio Test Runner

The problem

 

One of the most dreadful problem of Unit Testing is slow testing. If your whole suite of tests runs in 10 minutes, it is normal for developers not to run the whole suite at each build. One of the most common question is

How can I deal with slow Unit Tests?

Here is my actual scenario: in a project I’m working in, we have some multilingual full text search done in Elastic Search and we have a battery of Unit Tests that verify that searches work as expected. Since each test deletes all documents, insert a bunch of new documents and finally commits lucene index, execution times is high compared to the rest of tests. Each test need almost 2 seconds to run on my workstation, where I have really fast SSD and plenty of RAM.

This kind of tests cannot be run in memory or with some fancy trick to make then run quickly. Actually we have about 30 tests that executes in less than one seconds, and another 13 tests that runs in about 23 seconds, this is clearly unacceptable. After few hours of work, we already reached the point where running the whole suite becomes annoying.

The solution

 

This is a real common problem and it is quite simple to fix. First of all Visual Studio Test runner actually tells you execution time for each Unit Test, so you can immediately spot slow tests. When you identify slow tests you can mark them with a specific category, I use slowtest

    [TestFixture]
    [Category("elasticsearch")]
    [Category("slowtest")]
    public class EsSearcherFixture : BaseTestFixtureWithHelpers

Since I know in advance that this test are slow I immediately mark the entire class with the attribute slowtest. If you have no idea what of your tests are slow, I suggest grouping test by Duration in Visual Studio Test Runner.

image

Figure 1: Group tests by Duration

The result is interesting, because Visual Studio consider every test that needs more than one second to be slow. I tend to agree with this distinction.

image

Figure 2: Test are now grouped by duration

This permits you to immediately spot slow tests, so you can add the category slowtest to them. If you keep your Unit Tests organized and with a good usage of categories, you can simply ask VS Test Runner to exclude slow test with filter –Traits:”slowtest”

image

Figure 3: Thanks to filtering I can now execute continuously only test that are not slow.

I suggest you to do a periodic check to verify that every developers is using the slowtest category wisely, just group by duration, filters out the slowtest and you should not have no tests that are marked slow.

image

Figure 4: Removing the slowtest category and grouping by duration should list no slow test.

The nice part is that I’m using NUnit, because Visual Studio Test Runner supports many Unit Tests Frameworks thanks to the concepts of Test Adapters.

If you keep your tests well organized you will gain maximum benefit from them :).

Gian Maria.

Unit Test class that makes use of HttpWebRequest thanks to Visual Studio fakes library

Thanks to Visual Studio Fakes is it possible to isolate your unit test and testing difficult to test code. Today I need to test a class that issues some Web Request around the internet and I’m concerned about testing the SUT when the web response contains some specific code like 404 (Not Found) or something.

If you simply right-click the reference to system assembly in your project and Add a fakes assembly, you probably will be annoyed by the fact that when you try to create a ShimHttpWebRequest you are not able to do it. When some shim are not available, the reason is usually due to some limitation of fakes library, so types are skipped when shims and stub code are generated (during compilation).

Apart not being able to use a shim of HttpWebRequest, when you add a fakes assembly to Visual Studio test project, you are actually asking to the fakes library to create a stub and a shim for every class of that assembly, and this can be a problem because

  1. Your build time increase
  2. Some classes probably will not be generated due to Fakes Limitation

Solution is simple, just locate the System.Fakes files under Fakes directory of your test project, they usually contain only the name of the assembly and nothing more. This is the default and basically it means: Create a shim and a stub for every type in that assembly. You should change that file to specify only the types you are really going to use in your Unit Tests. Es.

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/" Diagnostic="true">
  <Assembly Name="System" Version="4.0.0.0"/>
  <StubGeneration>
    <Clear />
  </StubGeneration>
  <ShimGeneration>
    <Clear />
    <Add FullName="System.Net.WebRequest!"/>
    <Add FullName="System.Net.HttpWebRequest!"/>
    <Add FullName="System.Net.HttpWebResponse!"/>
  </ShimGeneration>
</Fakes>

This code is actually instructing the Fakes library on what class you are going to use. In the above example I’m telling that I’m not going to use any Stub and I want to use Shim for WebRequest, HttpWebRequest and HttpWebResponse. If you specify only the classes you are really going to isolate, build time will be shorter, and there is much more possibility that shims gets generated.

Remember also that when you create fake library for System assembly you actually will get two distinct fakes configuration, one for system, and the other for Mscorlib.fakes. I usually modify the mscorlib.fakes in such a way.

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="mscorlib" Version="4.0.0.0"/>
  <StubGeneration>
    <Clear />
  </StubGeneration>
  <ShimGeneration>
    
    
  </ShimGeneration>
</Fakes>

For some reason, I need not to clear ShimGeneration for Mscorlib entirely, because it will makes Unit Test stop with StackOverflowException in my system even if I’m not going to use any class for mscorlib.

Now I can write shim for HttpWebResponse

ShimHttpWebResponse res = new ShimHttpWebResponse();
res.StatusCodeGet = () => HttpStatusCode.NotFound;

This simple snippet created a fake response to identify a NotFound (404) web response. I can now proceed to isolate call to WebRequest to return that instance and write Unit Tests that isolates calls to WebRequest classes. A full possible test is the following.

[Test]
public void Verify_behavior_on_404()
{
    using (ShimsContext.Create())
    {
        SolrServer sut = new SolrServer("http://nonexistent.url.it/solr/blbalba");
        sut.Logger = NullLogger.Instance;


        ShimHttpWebResponse res = new ShimHttpWebResponse();
        res.StatusCodeGet = () => HttpStatusCode.NotFound;

        ShimWebException wex = new ShimWebException();
        wex.ResponseGet = () => res;

        ShimHttpWebRequest.AllInstances.GetResponse = url => { throw (WebException) wex; };
        sut.RawSearch("*:*", "id,name", 0, 100);
    }
}

With these few lines of code I’m telling that each request to all instance of HttpWebRequest should throw a specific WebException that will behave as I want. This absolutely invaluable if you need to test various response such of 403 if you are trying to simulate authentication or 500 if you need to simulate some particular error of called service. The good part of this kind of test is that the Sut was absolutely not designed to be unit tested. Here is the code in the Sut that is issuing call to Solr server

HttpWebRequest oRequest = (HttpWebRequest)WebRequest.Create(fullUrl);
if (Logger.IsDebugEnabled) Logger.Debug("Solar query " + fullUrl);
oRequest.Method = "GET";
HttpWebResponse oResponse = null;
try
{
    oResponse = (HttpWebResponse)oRequest.GetResponse();
}
catch (WebException e)

As you can see the code directly create an HttpWebRequest and issue a GET Request, but thanks to fakes library during the test the call to GetResponse are detoured to my test code.

Remember also that after Visual Studio Quarterly Update 2 fakes library is now available even in Premium edition and not only Ultimate.

Gian Maria

New Unit Test functionality in VS2012 Update 2: Test Playlist

In VS2012 we have support for basically any Unit Test Framework, because the new test runner is plugin based, and everyone can write plugin to make it compatible with the framework of choice. From the original version that was released with Visual Studio 2012 RTM, in Update 1 and Update 2 the test runner gained a lot of new feature, really useful for everyday work.

Update 2 introduced the concept of “Test Playlist”: basically a simple tool to create a subset of Unit Tests that are meant to be managed together. Since Update 1 already introduced the concept of Traits to group tests, this feature can be seen as a duplicate, but traits and playlist have different meaning. Let’s see how to create a playlist.

To create a list simply choose one or how many tests you like in test explorer, right click on one of them and choose Add to Playlist; if there is no playlist already loaded, the only option is “New Playlist”, just give the playlist a name and a location and create a new one.

image

Figure 1: Create a playlist from a test in Test Explorer

Now you can continue to add other test to the playlist if you want; once you are done you can select the playlist from the top menu of the Test Explorer.

image

Figure 2: You can now choose the playlist to use.

A playlist is just an XML file that contains the list of all the test you added to the playlist. To continue adding test to the playlist you should use the top Menu to select “All Tests”, find tests you want to add to the playlist, and simple right click and choose to “Add to Playlist”; now you should see all the testlist of the project and not only the “new Playlist” option.

Once you select a playlist, the test runner works exactly as before, but it consider only the tests included in current playlist, you can filter, group by traits, etc etc. To remove a test from a playlist, simply choose that playlist, right click on the test (or the tests) you want to remove and choose: Remove from current playlist.

As I told at the beginning of the post: the use of playlist is an alternative to use Traits to group test and run only a subset, but the main difference is that a Trait expresses a real property of the test, (ex the logical area of the software that got tested, or other attribute such Database to indicate that it needs a test database), while playlist are primarily meant to group together heterogeneous tests to simplify management, both for running test and manage them.

image

Figure 3: Group by traits.

This feature is also needed by all old MsTest users, that are used to create list of test to run with the old test runner.

Gian Maria.

Shim constructors to isolate “Future Objects”

I’ve already blogged previously about the new Shim Library of Visual Studio 2012

Now it is time to explore another scenario where shim can save your life. Suppose you are working with Hardware, Es. a barcode reader and you have a class in your system wrote in this way.

public class BarcodeReader
{
    public BarcodeReader() { 
        //connect to the hardware, if the hardware is not there 
        //throw an exception
    }

    public String ReadCode() 
    {
        //dialogate with the hardware and get the Barcode
    }
}

This class manage the connection to a physical BarcodeReader in the constructor and if something went wrong throws an exception and expose a ReadCode method that ask the Physical reader to read a Barcode. This class is used with the following pattern throughout all the software.

public class MyLogic
{
    private BarcodeReader reader;

    public MyLogic() {
        reader = new BarcodeReader();
    }

    public String RetrieveIndustrialCode()
    {
        String rawCode = reader.ReadCode();
        return rawCode.Substring(2, 10);
    }
}

MyLogic is a class with some business logic based on the Barcode returned from the call to ReadCode() method, as you can see it simply instantiate an instance of BarcodeReader in the constructor and read the code in the RetrieveIndustrialCode Method used to extract the industrial code from the raw content of the barcode. In this stupid example the algorithm to extract the Industrial Code from the Raw barcode read from the reader is simply reading chars from 2 to 10, but in real scenario we could have complex string parsing code that needs to be tested, but how?? As you can see from the code you can test only when the real hardware is connected to the machine because if a physical reader is not connected the BarcodeReader will throw exception, but the most annoying part is that it is impossible to simulate a read during a Unit Test.

This is another scenario where shims comes to the rescue, just add a fakes assembly (as described in the previous samples) and you can write this test.

[Fact]
public void verify_retrieval_of_industrialCode()
{
    using (ShimsContext.Create())
    {
        ShimBarcodeReader.Constructor = @this =>  new ShimBarcodeReader(@this) {
            ReadCode = () => "IC1234567890BLABLABLA",
        };
        MyLogic sut = new MyLogic();
        var industrialCode = sut.RetrieveIndustrialCode();
        Assert.Equal("1234567890", industrialCode);
    }
}

Whenever you fake an assembly, each Shim that gets created has one or more static property called ConstructorXXX, where the XXX part is used to distinguish from the overloaded version using paramers type. In this example I have a default constructor so my Shim has a static property called Constructor. This specific property permits you to intercept the future creation of that type and even substitute it with a shim. This specific property needs a lambda where the first parameter is the Instance of the object being created followed by all other parameters of the constructor. In my sample test I pass a lambda that based on the instance created, returns a new ShimBarcodeReader configured to return the code “IC1234567890BLABLABLA” from ReadCode() method.

This technique is really powerful because permits you to Shim every future instance that will be created for a specific object and makes possible to inject Shims almost everywhere. When the test run, the MyLogic constructor creates an instance of BarcodeReader, but thanks to the Shim library I’m able to pass a shim where I’ve already set the return value of ReadCode() method.

Thanks to this capability I can write tests until I eventually found a bug, to show how the Shim Constructor changes when constructor has parameters, I’ve changed the BarcodeReader class to make the constructor accepts a Single String parameter and I want to write a test that verify that when the BarcodeReader returns an Empty String the Industrial code returned is also Empty.

[Fact]
public void verify_retrieval_of_industrialCode_too_short()
{
    using (ShimsContext.Create())
    {
        ShimBarcodeReader.ConstructorString = (@this, value) => new ShimBarcodeReader(@this)
        {
            ReadCode = () => "",
        };
        MyLogic sut = new MyLogic("COM2");
        var industrialCode = sut.RetrieveIndustrialCode();
        Assert.Equal("", industrialCode);
    }
}

To shim the creation of a ShimBarcodeReader I now need to use ConstructorString property, since the constructor has now a parameter of type String. Thanks to Visual Studio 2012 and Shim Library you can now easily test untestable legacy code with minimum effort.

Gian Maria.

Running NUnit and xUnit tests in TFS11 build

I’ve blogged in the past various solution to run NUnit tests during a TFS build, and now it is time to make it again for TFS11, but this time it is incredibly simple, because the new Test Runner supports multiple frameworks, so it works almost automatically.

You can read from Peter Provost blog that actually we have three plugin for UTE (Unit Test Explorer) available: Nunit, xUnit and HTML/JAvascript, they are simple .vsix file (Visual Studio Extension), that you can download, run, and voilà, your xUnit and NUnit tests are runnable from Visual Studio.

image

Figure 1: The new UTE is able to run Unit Tests from multiple framework, because it is extendible

Now if you create a build against this solution, you probably will be disappointed by the fact that the build only runs MStest ignoring tests supported by an external plugin.

image

Figure 2: Build result shows that only 6 test were run, xUnit and NUnit tests are ignored

This blog post explain the reason, if you are on 64 bit machine, the vsix installer is not able to make the Extension visible to build controller, so you need to do a little extra step. First of all locate the folder of the two extensions under the plugin for the current user

image

Figure 3: UTE plugins are located in the standard plugin folder for Visual Studio

This location may vary, in my system it installed only for current user so they are on my profile folder, if you do not find the extension there you can simply search for the dll NUnit.VisualStudio.TestAdapter.dll into your Hard Disk. Once you found xUnit and NUnit UTE plugin folder, you should copy all dll inside a source controlled folder.

image

Figure 4: All UTE plugin assemblies are now in a source controlled folder

Finally go to Team Explorer –> Build –> Actions –> Manage Build Controllers and specify that all custom assemblies are located inside that folder

image

Figure 5: Configure the test controller specifying the location of all the assemblies that contains build extension

Now you can queue another build, and this time all tests should be run.

image

Figure 6: Now all 16 tests were run

As you can see, with TFS and VS11 running unit test from various Unit Test Frameworks is really easy.

Alk.