Quickly run Unit Test With VS 2012/2013/..

With VS 2012 and newer versions we can run Unit Tests from various frameworks directly from Visual Studio IDE, thanks to the concept of Test Adapters. When you are doing Test Driven Development you usually go with Red/Green/Refactor workflow; what you need is a way to quickly run all or part of your unit tests after you modified the code. The quickest solution is using the option to Run Tests After Build but it is available only for Premium and Ultimate edition, but you can also run test with little manual intervention resorting to Keyboard Shortcut.

Simply go to TOOLS->Customize menu, then choose to customize keyboard.

image

Figure 1: Keyboard customization in Visual Studio

Once configuration window is opened, search for TestExplorer.RunAllTests command, place cursor in the “Press shortcut keys:” textbox and press a shortcut, then press “assign” button to assign to this command. In my standard configuration I like to have CTRL+SHIFT+ALT+A shortcut, because it is not assigned to any other command and it is easy to press with left hand.

image

Figure 2: Assign CTRL+SHIFT+ALT+A to TestExplorer.RunAllTests command

Now I can write code, use shortcut and VS will build solution and run test for me automatically, without leaving my hands from the keyboard. Thanks to the various test adapter and the various grouping and filtering possibility offered by Test Explorer, you can do TDD in Visual Studio without the need of third party tools.

Gian Maria.

Nunit test not found for some assemblies with Visual Studio Test Runner

I’ve a project in Visual Studio 2013 where one of the assembly containing Tests refuses to show tests in Test Explorer window. The solution has tests written both in Nunit and in MSpec, and everything is good except for that specific assembly. If you notice that Test Explorer window misses some tests, the first thing you need to check is the output windows, where you can find some output for Test Adapters.

image

Figure 1: Output for test adapters can be found in standard Output Window

In the above situation the assembly that is ignored is Intranet.Tests.dll and the reason is that it is built for Framework45 and x64 platform, while the test runner is executing against x86 platform. Everything seems ok, every project is compiled in .ANY CPU, but looking at the raw project file I can confirm that PlatformTarget is set in x64. Changing to x86 (or removing it completely) solves the problem.

image

Figure 2: Platform target changed from x64 to x86

After I changed the PlatformTarget attribute, all tests belonging to that assembly are now available Test Explorer window.

Gian Maria.

How to deal with Slow Unit Tests with Visual Studio Test Runner

The problem

 

One of the most dreadful problem of Unit Testing is slow testing. If your whole suite of tests runs in 10 minutes, it is normal for developers not to run the whole suite at each build. One of the most common question is

How can I deal with slow Unit Tests?

Here is my actual scenario: in a project I’m working in, we have some multilingual full text search done in Elastic Search and we have a battery of Unit Tests that verify that searches work as expected. Since each test deletes all documents, insert a bunch of new documents and finally commits lucene index, execution times is high compared to the rest of tests. Each test need almost 2 seconds to run on my workstation, where I have really fast SSD and plenty of RAM.

This kind of tests cannot be run in memory or with some fancy trick to make then run quickly. Actually we have about 30 tests that executes in less than one seconds, and another 13 tests that runs in about 23 seconds, this is clearly unacceptable. After few hours of work, we already reached the point where running the whole suite becomes annoying.

The solution

 

This is a real common problem and it is quite simple to fix. First of all Visual Studio Test runner actually tells you execution time for each Unit Test, so you can immediately spot slow tests. When you identify slow tests you can mark them with a specific category, I use slowtest

    [TestFixture]
    [Category("elasticsearch")]
    [Category("slowtest")]
    public class EsSearcherFixture : BaseTestFixtureWithHelpers

Since I know in advance that this test are slow I immediately mark the entire class with the attribute slowtest. If you have no idea what of your tests are slow, I suggest grouping test by Duration in Visual Studio Test Runner.

image

Figure 1: Group tests by Duration

The result is interesting, because Visual Studio consider every test that needs more than one second to be slow. I tend to agree with this distinction.

image

Figure 2: Test are now grouped by duration

This permits you to immediately spot slow tests, so you can add the category slowtest to them. If you keep your Unit Tests organized and with a good usage of categories, you can simply ask VS Test Runner to exclude slow test with filter –Traits:”slowtest”

image

Figure 3: Thanks to filtering I can now execute continuously only test that are not slow.

I suggest you to do a periodic check to verify that every developers is using the slowtest category wisely, just group by duration, filters out the slowtest and you should not have no tests that are marked slow.

image

Figure 4: Removing the slowtest category and grouping by duration should list no slow test.

The nice part is that I’m using NUnit, because Visual Studio Test Runner supports many Unit Tests Frameworks thanks to the concepts of Test Adapters.

If you keep your tests well organized you will gain maximum benefit from them :).

Gian Maria.

Programmatically use of Coded UI in Visual Studio

Coded UI Tests are a specific type of UI testing introduced with Visual Studio 2010. You can create your first Coded UI test following simple instruction from MSDN documentation. Most of the introductory examples shows you how you can use the Recorder tools to record interaction with a software (Web, WinForm, Wpf. etc) to generate what is called a UiMap. An UiMap is nothing more than a big Xml files where the recorder records the interaction with the UI and a bunch of automatic generated classes to interact with the UI.

Using a UiMap is probably not the best option for large projects, because the cost of maintaining it could become really high. This is usually not a big problem, because UiMap is used to generate code based on a set of classes belonging to Visual Studio Testing Framework that makes possible to interact with a UI from code. If maintaining a UiMap is difficult for you you can directly use these classes in your test. To show you the “hello world” equivalent of CUIT, here is the code needed to open a page in a browser and click an hyperlink.

using ( BrowserWindow browserWindow =
            BrowserWindow.Launch
            (
                new System.Uri("http://tailspintoys.azurewebsites.net/")
            ))
{
               
    HtmlHyperlink link = new HtmlHyperlink(browserWindow);
    link.SearchProperties.Add
        (
            HtmlHyperlink.PropertyNames.InnerText,
            "Model Airplanes"
        );
    Mouse.Click(link); 
}

The code is really simply, you must use the BrowserWindow.Launch static method to create an instance of BrowserWindow class pointing to a given Url. The BrowserWindow class is a wrapper defined Visual Studio Coded Ui assemby used to abstract the interaction with a web browser. The next step is locating the hyperlink you want to click, operation that can be accomplished with the HtmlHyperlink object. This object derives from the UiTestControl base class, and abstracts the concept of a control in the User Interface. The constructor of HtmlHyperlink object needs an instance of a containing control, in this example the whole browserWindows object. The need for the Container is having a root control that will be searched for the control.

To specify the exact Hyperlink control you want to interact with, you should populate SearchProperties collection, specifying the criteria you want to use. In this example I used the InnerText property, but you can use a lot of other criteria. Thanks to PropertyNames static collection of HtmlHyperlink object you can enumerate all the properties that can be used to locate the control. Inner Text is not usually the best option, using unique Id is usually a better approach, but the key concept is: You should use the criteria that is most stable in your scenario/environment. If you can ask to development team to assign unique id or unique names to each control, tests will be more robust and quicker.

Once SearchProperties collection is filled with critera, you can interact with the control, accessing properties or passing it to Mouse.Click method to simulate a click. CodedUI engine will locate the control on the page only when You will access properties or pass the control to some method that interact with it. This is really important, until you do not access properties the engine will not try to locate the control on the UI.

Remember to enclose the BrowserWindow object in a using block, this will ensure that the instance of the browser opened during the test will be always closed. This prevents multiple browser windows to remain opened after the test if some exception occurred.

Gian Maria.

Visual Studio Plugin: stop build at first error

When it is time to work with big solutions composed by many projects, it is useful to have the ability to stop the build at the very first build error. There are several reason to do this, first of all probably many of the subsequent error can be caused by the fact that a base project does not compile, or simply the build process takes a long time that it is not useful to continue the build when a project does not build, (quite often we are interested only in build the whole solution, so there is no point in waiting for all the other project to compile, because at the end everything we want to do is fixing the project that does not compile and try again to rebuild).

Implementing this logic in a plugin is super easy, first of all I create a simple class that contains all the code needed by the plugin, and in constructor I register handlers for a couple of Visual Studio Events.

Boolean _enabled = false;
private Boolean _alreadyStopped = false;

public StopBuildAtFirstError(DTE2 dte)
{
    _dte = dte;
    dte.Events.BuildEvents.OnBuildProjConfigDone += OnBuildProjConfigDone;
    dte.Events.BuildEvents.OnBuildBegin += (sender, e) => _alreadyStopped = false;
}

As you can verify, once you have a reference to a DTE2 object you can simply access Events property that contains all useful events raised by Visual Studio, grouped by categories; in my plugin I’m interested to be notified when the build starts (OnBuildBegin) and when a project finishes compiling (OmBuildProjConfigDone). I use a simply private boolean field to verify if I already stopped the build, this is needed if the build goes in parallel so I can have two concurrent project that failed building, but I need to issue only one command to stop the build. Here is the logic to stop the build if a project fails.


private void OnBuildProjConfigDone(string project, string projectConfig, string platform, string solutionConfig, bool success)
{
    if (_alreadyStopped || success || !_enabled) return;

    _alreadyStopped = true;

    _dte.ExecuteCommand("Build.Cancel");

    var pane = _dte.ToolWindows.OutputWindow.OutputWindowPanes
                                .Cast<OutputWindowPane>()
                                .SingleOrDefault(x => x.Guid.Equals(AlkampferVsix2012.Utils.Constants.BuildOutput, StringComparison.OrdinalIgnoreCase));

    if (pane != null)
    {
        Int32 lastSlashIndex = project.LastIndexOf('\\');
        String projectFileName = project.Substring(lastSlashIndex + 1, project.LastIndexOf('.') - lastSlashIndex - 1);
        var message = string.Format("INFO: Build stopped because project {0} failed to build.\n", projectFileName);
        pane.OutputString(message);
        pane.Activate();
    }
}

Super Easy isn’t it? To know if the build of the project succeeded you can simply check a parameter called success; to stop the build you can simply send a Build.Cancel command to DTE2 object. The ExecuteCommand method is really useful, because you can check every command available in VS directly from the menu Tools->Customize press the button Keyboard to add shortcut to command, and you can browse to every command that is available in the system. Once you find the command you want to raise from a plugin you can simply use the ExecuteMethod of DTE2 object passing command string as single argument and you are done.

The rest of the handler simply grab a reference to the Build OutputWindow and add a message telling that the build was stopped because the specific project failed to build. Now you can simply press F5 and debug your plugin in Visual Studio experimental Hive.

image

Figure 1: New menu option created by the plugin

This plugin will create a couple of new Menu Items under Tools menu, the first one is the “Attach to IIS” discussed previously, the second one is the “Stop Build on 1st error” that is disabled by default, you can enable simply clicking on it. Now if you build a solution and a project failed to build you will receive a message like this one.

image

Figure 2: Here is error message of the build and the subsequent info that tells you that the build was stopped

The coloration of the build output is another feature of the same plugin that helps me to immediately hilight in red all lines that contains the word error and in blue everything that starts with INFO: (informational message of the plugin itself).

Code of the plugin can be found here (file AlkampferVsix2.zip)

Alk.