How to deal with Slow Unit Tests with Visual Studio Test Runner

The problem

 

One of the most dreadful problem of Unit Testing is slow testing. If your whole suite of tests runs in 10 minutes, it is normal for developers not to run the whole suite at each build. One of the most common question is

How can I deal with slow Unit Tests?

Here is my actual scenario: in a project I’m working in, we have some multilingual full text search done in Elastic Search and we have a battery of Unit Tests that verify that searches work as expected. Since each test deletes all documents, insert a bunch of new documents and finally commits lucene index, execution times is high compared to the rest of tests. Each test need almost 2 seconds to run on my workstation, where I have really fast SSD and plenty of RAM.

This kind of tests cannot be run in memory or with some fancy trick to make then run quickly. Actually we have about 30 tests that executes in less than one seconds, and another 13 tests that runs in about 23 seconds, this is clearly unacceptable. After few hours of work, we already reached the point where running the whole suite becomes annoying.

The solution

 

This is a real common problem and it is quite simple to fix. First of all Visual Studio Test runner actually tells you execution time for each Unit Test, so you can immediately spot slow tests. When you identify slow tests you can mark them with a specific category, I use slowtest

    [TestFixture]
    [Category("elasticsearch")]
    [Category("slowtest")]
    public class EsSearcherFixture : BaseTestFixtureWithHelpers

Since I know in advance that this test are slow I immediately mark the entire class with the attribute slowtest. If you have no idea what of your tests are slow, I suggest grouping test by Duration in Visual Studio Test Runner.

image

Figure 1: Group tests by Duration

The result is interesting, because Visual Studio consider every test that needs more than one second to be slow. I tend to agree with this distinction.

image

Figure 2: Test are now grouped by duration

This permits you to immediately spot slow tests, so you can add the category slowtest to them. If you keep your Unit Tests organized and with a good usage of categories, you can simply ask VS Test Runner to exclude slow test with filter –Traits:”slowtest”

image

Figure 3: Thanks to filtering I can now execute continuously only test that are not slow.

I suggest you to do a periodic check to verify that every developers is using the slowtest category wisely, just group by duration, filters out the slowtest and you should not have no tests that are marked slow.

image

Figure 4: Removing the slowtest category and grouping by duration should list no slow test.

The nice part is that I’m using NUnit, because Visual Studio Test Runner supports many Unit Tests Frameworks thanks to the concepts of Test Adapters.

If you keep your tests well organized you will gain maximum benefit from them :).

Gian Maria.

Programmatically use of Coded UI in Visual Studio

Coded UI Tests are a specific type of UI testing introduced with Visual Studio 2010. You can create your first Coded UI test following simple instruction from MSDN documentation. Most of the introductory examples shows you how you can use the Recorder tools to record interaction with a software (Web, WinForm, Wpf. etc) to generate what is called a UiMap. An UiMap is nothing more than a big Xml files where the recorder records the interaction with the UI and a bunch of automatic generated classes to interact with the UI.

Using a UiMap is probably not the best option for large projects, because the cost of maintaining it could become really high. This is usually not a big problem, because UiMap is used to generate code based on a set of classes belonging to Visual Studio Testing Framework that makes possible to interact with a UI from code. If maintaining a UiMap is difficult for you you can directly use these classes in your test. To show you the “hello world” equivalent of CUIT, here is the code needed to open a page in a browser and click an hyperlink.

using ( BrowserWindow browserWindow =
            BrowserWindow.Launch
            (
                new System.Uri("http://tailspintoys.azurewebsites.net/")
            ))
{
               
    HtmlHyperlink link = new HtmlHyperlink(browserWindow);
    link.SearchProperties.Add
        (
            HtmlHyperlink.PropertyNames.InnerText,
            "Model Airplanes"
        );
    Mouse.Click(link); 
}

The code is really simply, you must use the BrowserWindow.Launch static method to create an instance of BrowserWindow class pointing to a given Url. The BrowserWindow class is a wrapper defined Visual Studio Coded Ui assemby used to abstract the interaction with a web browser. The next step is locating the hyperlink you want to click, operation that can be accomplished with the HtmlHyperlink object. This object derives from the UiTestControl base class, and abstracts the concept of a control in the User Interface. The constructor of HtmlHyperlink object needs an instance of a containing control, in this example the whole browserWindows object. The need for the Container is having a root control that will be searched for the control.

To specify the exact Hyperlink control you want to interact with, you should populate SearchProperties collection, specifying the criteria you want to use. In this example I used the InnerText property, but you can use a lot of other criteria. Thanks to PropertyNames static collection of HtmlHyperlink object you can enumerate all the properties that can be used to locate the control. Inner Text is not usually the best option, using unique Id is usually a better approach, but the key concept is: You should use the criteria that is most stable in your scenario/environment. If you can ask to development team to assign unique id or unique names to each control, tests will be more robust and quicker.

Once SearchProperties collection is filled with critera, you can interact with the control, accessing properties or passing it to Mouse.Click method to simulate a click. CodedUI engine will locate the control on the page only when You will access properties or pass the control to some method that interact with it. This is really important, until you do not access properties the engine will not try to locate the control on the UI.

Remember to enclose the BrowserWindow object in a using block, this will ensure that the instance of the browser opened during the test will be always closed. This prevents multiple browser windows to remain opened after the test if some exception occurred.

Gian Maria.

Visual Studio Plugin: stop build at first error

When it is time to work with big solutions composed by many projects, it is useful to have the ability to stop the build at the very first build error. There are several reason to do this, first of all probably many of the subsequent error can be caused by the fact that a base project does not compile, or simply the build process takes a long time that it is not useful to continue the build when a project does not build, (quite often we are interested only in build the whole solution, so there is no point in waiting for all the other project to compile, because at the end everything we want to do is fixing the project that does not compile and try again to rebuild).

Implementing this logic in a plugin is super easy, first of all I create a simple class that contains all the code needed by the plugin, and in constructor I register handlers for a couple of Visual Studio Events.

Boolean _enabled = false;
private Boolean _alreadyStopped = false;

public StopBuildAtFirstError(DTE2 dte)
{
    _dte = dte;
    dte.Events.BuildEvents.OnBuildProjConfigDone += OnBuildProjConfigDone;
    dte.Events.BuildEvents.OnBuildBegin += (sender, e) => _alreadyStopped = false;
}

As you can verify, once you have a reference to a DTE2 object you can simply access Events property that contains all useful events raised by Visual Studio, grouped by categories; in my plugin I’m interested to be notified when the build starts (OnBuildBegin) and when a project finishes compiling (OmBuildProjConfigDone). I use a simply private boolean field to verify if I already stopped the build, this is needed if the build goes in parallel so I can have two concurrent project that failed building, but I need to issue only one command to stop the build. Here is the logic to stop the build if a project fails.


private void OnBuildProjConfigDone(string project, string projectConfig, string platform, string solutionConfig, bool success)
{
    if (_alreadyStopped || success || !_enabled) return;

    _alreadyStopped = true;

    _dte.ExecuteCommand("Build.Cancel");

    var pane = _dte.ToolWindows.OutputWindow.OutputWindowPanes
                                .Cast<OutputWindowPane>()
                                .SingleOrDefault(x => x.Guid.Equals(AlkampferVsix2012.Utils.Constants.BuildOutput, StringComparison.OrdinalIgnoreCase));

    if (pane != null)
    {
        Int32 lastSlashIndex = project.LastIndexOf('\\');
        String projectFileName = project.Substring(lastSlashIndex + 1, project.LastIndexOf('.') - lastSlashIndex - 1);
        var message = string.Format("INFO: Build stopped because project {0} failed to build.\n", projectFileName);
        pane.OutputString(message);
        pane.Activate();
    }
}

Super Easy isn’t it? To know if the build of the project succeeded you can simply check a parameter called success; to stop the build you can simply send a Build.Cancel command to DTE2 object. The ExecuteCommand method is really useful, because you can check every command available in VS directly from the menu Tools->Customize press the button Keyboard to add shortcut to command, and you can browse to every command that is available in the system. Once you find the command you want to raise from a plugin you can simply use the ExecuteMethod of DTE2 object passing command string as single argument and you are done.

The rest of the handler simply grab a reference to the Build OutputWindow and add a message telling that the build was stopped because the specific project failed to build. Now you can simply press F5 and debug your plugin in Visual Studio experimental Hive.

image

Figure 1: New menu option created by the plugin

This plugin will create a couple of new Menu Items under Tools menu, the first one is the “Attach to IIS” discussed previously, the second one is the “Stop Build on 1st error” that is disabled by default, you can enable simply clicking on it. Now if you build a solution and a project failed to build you will receive a message like this one.

image

Figure 2: Here is error message of the build and the subsequent info that tells you that the build was stopped

The coloration of the build output is another feature of the same plugin that helps me to immediately hilight in red all lines that contains the word error and in blue everything that starts with INFO: (informational message of the plugin itself).

Code of the plugin can be found here (file AlkampferVsix2.zip)

Alk.

Converting Visual Studio Macro to Visual Studio plugin

Visual Studio 2012 is really faster than 2010, this is due to an excellent work of the team to maximize performance and because in this release some of the older and less used part of the IDE were removed. One of this part is the Macro editor that is not anymore available in Visual Studio. This is one of the feature I missed most because I’ve used it in the past to automate some basic task, like attach to IIS automatically with a keyboard shortcut. The Macro engine had some limitation, it supported only Visual Basic, it was old code that needs time to be maintained and finally everything you can do with a Macro can be done with Visual Studio Plugin so there was no need to maintain anymore in the product.

If you used macro with VS2010 and you want to use them in VS2012 you need to create an addin, the easiest way is to go for a Visual Studio Package:

image

Figure 1: Creating a Visual Studio Package

Choosing a Project type of “Visual Studio Package” will launch a wizard that asks you some information about the type of extension you want to create, the most important part is choosing the functions that the plugin want to implement, to convert a macro the simplest solution is implement one or more Commands.

image

Figure 2: Choose to implement Menu Command.

The default structure created by the wizard implements a simple command placed under the Tools menu, it can be invoked with the menu or you can assign shortcut to launch with a combination of keys. Once the project is created converting the macro is just a matter of copying the code of the macro inside the plugin and call inside the handler of the command.

Dte = Package.GetGlobalService(typeof(EnvDTE.DTE)) as DTE2;

// Add our command handlers for menu (commands must exist in the .vsct file)
_attachToIIS = new AttachToIIS(Dte);
OleMenuCommandService mcs = GetService(typeof(IMenuCommandService)) as OleMenuCommandService;
if (null != mcs)
{
    // Create the command for the menu item.
    CommandID menuCommandID = new CommandID(GuidList.guidAlkampferVsix2012CmdSet, (int)PkgCmdIDList.attachToIIS);
    MenuCommand menuItem = new MenuCommand(_attachToIIS.MenuItemCallback, menuCommandID);
    mcs.AddCommand(menuItem);
}

This snippet of code is based on code generated by the wizard, I’ve only added line 1 to retrieve a reference to the DTE2 object needed by my AttachToIIS macro code and to keep the code clean I’ve moved all the code of the macro inside a dedicated class called AttachToIIS. This class has a method called MenuItemCallback that gots called when the user ask to execute the command (from the menu or from a shortcut).

I’ve then converted all of my old VB code to C# and pasted inside the class, then press F5 and an instance of VS2012 Experimental Hive started, permitting me to debug the plugin to verify if everything is ok.

image

Figure 3: My plugin is loaded correctly and the menu is there

The icon is the default one used by the wizard it is an aspect I do not care about because I just want to use my old macro code :), now it is time to use Tools –> Customize menu to assign a shortcut to this command.

image

Figure 4: My new command is loaded and is associated to the Tools namespace, so it is called tools.AttachToIIS

Et voilà, the game is done, the whole process of Macro Conversion took no more than 15 minutes and I’m able to use the same macro code in VS2012. This is my feedback of the process.

Advantages:

  • Creating a VS package is a matter of few clicks and you have a solution completely configure to create your plugin
  • The result is much more manageable, instead of having a bunch of macro code that you need to copy and paste inside the Macro IDE when you reinstall you have a vsix package that can be installed by everyone with a simple double click.
  • You are not forced to use Visual Basic

Disadvantages:

  • It is much simpler to experiment with macro, just write code, press F5 and you are immediately debugging the code, with a plugin when you press F5 another instance of VS starts, and it needs some seconds.

I’ll release the code the next week.

Gian Maria.

Location of your vsix files in experimental hive during plugin development

When you are developing Visual Studio plugin, your project is usually configured to launch a special instance of Visual Studio under the Visual Studio debugger (excellent example of dogfooding), this special instance is called experimental HIVE.

image

Figure 1: Configuration of a standard VSIX project to test your addin

As you can see the project is configured to start Visual Studio as external program to be debugged and it is started with the option /rootsuffix Exp to start in Experimental Hive. This permits to test plugin in a configuration of Visual Studio that is isolated from your standard instance, think as an example if you develop a plugin that makes Visual Studio instable and you cannot open Visual Studio due to this addin.

If you want to remove or look at current configuration of experimental hive you can go to folder

C:\Users\username\AppData\Local\Microsoft\VisualStudio\

You should see all configuration folder for the various editions of Visual Studio, those that ends with Exp are related to experimental hive.

image

Figure 2: Folders related to experimental hive.

You can now enter in experimental hive folder and in extensions folder you should see all of your plugin that are under development, now you can remove them if you want to restore everything to the original value.

Alk.