MTM and the cycle of a bug

Thanks to Microsoft Test Manager, testing teams have a complete tool to manage every part of testing, but since it is a new tool we need to understand how to use at its best, so I’ll try to explain a typical cycle of a test-bug-fix-test with MTM.

Everything starts with the definition of a Test Case in a Test suite, in this example I want to verify what happens when the user insert some specific quantity in the Chart page of TailSpin toys.


Figure 1: Definition of a test

In this test I’m requesting the tester to: add an item to the chart, change the quantity to @NewQuantity value, and finally verify that the system update the charts showing the value: @ExpectedQuantity. Thanks to parameters, testers can specify more set of data to use in the test to verify boundary values etc etc. In this specific situation the stakeholder requests that a non numeric or negative value should not update the value of the chart, while a quantity of zero should change the quantity to zero, leaving the object in the chart.

A tester now executes the test, the first time he enables the recording asking MTM to record all his actions. This is useful because the recording can be used to do a fast forward during test execution, actually reducing time needed to execute the test.

In the video I show how the tester can use existing action recording to speedup test execution, the tester find an error, creates a bug, and now it is time for developer to fix it.

The developer begins working with the bug and he need first of all to be able to reproduce the bug before starting to understand how to fix it. Thanks to the recording of test execution, he can create a new CodedUI test based on MTM recording just choosing “use Existing action recording”  to create the CUIT; Visual Studio now opens a form to select the Test Case that contains the action recording


Figure 2: Choose the Test Case that contains the action recording to import in the CodedUI test

Since the developer knows that the bug was related to a failure of the Test Case 45, he can import  related test execution recording, obtaining this CUIT.

   1: [DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://localhost:8080/tfs/defaultcollection;Tailspin Toys", "45", DataAccessMethod.Sequential), TestMethod]

   2:  public void CodedUITestMethod1()

   3:  {

   4:      // To generate code for this test, select "Generate Code for Coded UI Test" from the shortcut menu and select one of the menu items.

   5:      // For more information on generated code, see

   6:      this.UIMap.Openhttpwings9gmujits88000();

   7:      this.UIMap.ClickModelAirplanes();

   8:      this.UIMap.ClickFourthCoffeeFlyer();

   9:      this.UIMap.ClickAddtoCart();

  10:      this.UIMap.ClickonwhitespaceinwebsiteParams.UIQuantityEditText = TestContext.DataRow["NewQuantity"].ToString();

  11:      this.UIMap.Clickonwhitespaceinwebsite();

  12:      this.UIMap.ClickblueXtoremoveitemfromcart();

  13:      this.UIMap.Closebrowser();

  14:  }

The most interesting part is the DataSource attributes that makes the test connect to TFS and grab all parameters value directly from the test case definition. Since the CUIT is generated from an action recording it has no assertions, but the developer knows that, since verification steps requires the user to only check a value in the textbox, he can write a CodedUI test assertion to fully automate the test.

First of all the dev executes manually all the steps of the test, until he reach the chart page. Now he can edit the UIMap, just Right Click on the UIMap.uitest file and choose Edit Wit Coded UI Test Builder


Figure 3: Editing the UIMap

Now it is really easy to create an assertion. As you can see in Figure 4 you can simply drag and drop the Crosshair to the quantity textbox control, CUIT Builder shows all properties of the control, now find and select the Text property and press the “Add Assertion” button to create an assertion.


Figure 4: Adding an assertion thanks to Coded UI Test Builder.

Now that the assertion is added, since this is the only condition we want to assert, we can: press the “Generate Code”, give a name to the assertion and press the button “Add and Generate”, as visible in Figure 5.


Figure 5: How to add the assertion to coded UI map.

Now this assertion can be used in the CodedUI test:

   1: [DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase", "http://localhost:8080/tfs/defaultcollection;Tailspin Toys", "45", DataAccessMethod.Sequential), TestMethod]

   2: public void CodedUITestMethod1()

   3: {

   4:     // To generate code for this test, select "Generate Code for Coded UI Test" from the shortcut menu and select one of the menu items.

   5:     // For more information on generated code, see

   6:     this.UIMap.Openhttpwings9gmujits88000();

   7:     this.UIMap.ClickModelAirplanes();

   8:     this.UIMap.ClickFourthCoffeeFlyer();

   9:     this.UIMap.ClickAddtoCart();

  10:     this.UIMap.ClickonwhitespaceinwebsiteParams.UIQuantityEditText = TestContext.DataRow["NewQuantity"].ToString();

  11:     this.UIMap.Clickonwhitespaceinwebsite();

  12:     //added the assertion

  13:     this.UIMap.AssertQuantityInChartExpectedValues.UIQuantityEditText =

  14:         TestContext.DataRow["ExpectedQuantity"].ToString();

  15:     this.UIMap.AssertQuantityInChart();


  17:     this.UIMap.ClickblueXtoremoveitemfromcart();

  18:     this.UIMap.Closebrowser();

  19: }

I added lines 12-15. Lines 13-14 is where I set the expected quantity of the assertion. Each time you record an  assertion, Coded UI Test Builder tools try to parametrize generated UIMap when possible. When we created the assertion, test builder parametrizes the ExpectedValue, thanks to this feature I’m able to set expected value to the value of the parameter @ExpectedQuantity (see Figure 1) of MTM, finally line 15 executes the assertion. Now if the developer runs the test he got this output.


Figure 6: Result of CUIT execution after the assertion

As you can see in Figure 6 the test got executed three times (once for each parameter set), and the test failed on run number 1, because we expect the value of the textbox quantity in the Chart to have the value 0 but it has the value 1.

This is a great achievement, because the developer, with few clicks, was able to create an automated CUIT test that guarantee bug reproducibility. Now the dev can inspect the code, find the reason of the bug, try to fix it and verify if the fix is ok running the above CodedUI Test. Since developer usually uses local IIS to test local build, he just need to change the url used by the test ,adding this line at the head of the test

   1: this.UIMap.Openhttpwings9gmujits88000Params.UIBlankPageWindowsInteWindowUrl 

   2:     = "";


Usually this line is removed once the bug is fixed so the test will point to the original test server. Once the bug is fixed, the developer check-in the code, opens the definition of Test Case, and automate the test with the new CUIT just created. This operation is really important, because now we does not need anymore to do a manual execution of this Test Case, but we can simply run the CUIT to obtain the same result.


Figure 7: The CUIT just created is added to the Test Case as “associated automation”

After some time, test manager decides to test a new build.  He deployed the new build in all test environments, then testers looks at test impact or at the list of closed bug to see what is changed and what to test. When a tester verifies that the bug related to test case 45 is marked as resolved, he needs to verify that the bug is really fixed running test again and if everything is ok close it. When he opens MTM and looks at the the test case he see that now this test is automated.


Figure 8: Test case 45 now contains a related automation

Now he can simply right click the test and ask for automatic execution in all compatible environments.

This bug lifecycle shows how useful MTM is, because it guarantees seamless interaction between testers and developers. Test manager write a test, a tester executes it and creates a bug if a failure occurs, developer uses MTM recording to create a CUIT and add assertion to it, the bug got fixed and the test automated, so it can be automatically executed during a Lab Management build.

The main advantage of this workflow is that, thanks to action recording, the developer can replicate tester actions, thus reducing drastically the time needed to reproduce the bug. Another strength point is the ability to create and automate a CUIT, that will be added to the suite of automated test.


Testing capabilities in TFS 2010

In TFS 2010 testing capabilities are really exceptional, and I’m not referring to testing tools that actually permits to record and replay web pages, CUIT etc, but I’m referring to the possibility to do a serious management of test plans for your projects.

I’ve seen in the past team doing test plan management with Excel, and if you are managing your test plan with Excel…you have a problem. The reason is that Excel is not a tool designed to manage Test Cases, you can clearly use it to write down your test, but it is difficult to integrate it into your ALM process and really difficult to read. The main problem is that Test plans in excel are completely dethatched from all other artifacts of your process, even if you manage to put all of them into team project SharePoint site. Moreover Excel does not helps you to run test or does not help testers to write down the results, usually resulting in Excel Test Spreasheet that are not aligned with the real testing run results.

Thanks to MTM you can leverage the power of TFS for managing testing suites; as an example suppose you have a user story about User Management of your application.


Figure 1: A simple user story.

Now the testing team begin to create test suite for this User Story, and thanks to MTM they can simply press the “add requirements” button, choose that User Story and create a test suite linked to that user story.


Figure 2: a Test case called “Gestione Utenti” linked to the User Story “Gestione Utenti”

The main advantage is that the association is managed by TFS, that automatically list each test case of this test suite as belonging to the corresponding User Story. The tester team add two Test Case to the Suite, and then all developers can look at them in Visual Studio.


Figure 3: In visual studio you can see all test case that are “testing” this user story

This is the key in ALM: traceability. Now you can easily understand what are the test cases related to the user story, and technical project manager can verify that all the tasks of the User Story are enough to satisfy test cases. Sometimes this helps a lot in discovering “hidden requirements”, suppose that the technical lead see a test case that states

1) open the web page

2) login

3) close the browser

4) open the web page

5) verify you are still logged.

This test case could highlight a requirement that was not clearly stated elsewhere, and the technical project manager can create a new task to support this scenario. This makes possible for developers to look at all the tests that will be run on a certain portion of the software, so they can easily verify if everything is really ready to be sent do tester. In the previous case, developers can implement a new feature to support the test case, avoiding the usual ping pong problem

1) dev send a build to testers

2) testers reject because the test case xxx does not pass

3) developers should look somewhere the test case, modify software and send another build.

Thanks to Teast case traceability, developers can easily verify if the code is ready to be tested, and if all the feature are implemented.

Thanks to TFS web access, I can easily look at the test cases even from a browser and not only from Visual Studio or MTM, so everyone can easily check all test cases to spot error or just to understand if testing is progressing well.


Figure 4: You can check a test case even online with a simple web browser.

If you are the Test Plan Manager, you should also allocate resources to the test, and especially you should assign test cases to test team members, also you should plan when each test should be run and you could have the need to tell that a specific test case should be executed after another test case. This kind of management is perfect to do in Microsoft Project.


Figure 5: How to assign a resource to a test case, in this example I’m assigning the Administrator to the “Esempio test case chiuso” test case

Clearly you can use your GANTT chart to plan the execution.


Figure 6: planning execution of test in Microsoft Project.

Publishing the modifications to TFS results in reallocation of the testing resource


Figure 7: The test case 63 was assigned to Administrator with the “assign resources” of Microsoft Project and the changes are published back to TFS, so they are visible everywhere.

In Microsoft Project you can use Task Inspector to verify if some resource is overallocated because they have to work on other tasks. In Figure 8 we can see that project complains about the assignment of Administrator to a test case, because he is also assigned to other tasks. Clearly gantt planning for single test cases could be a little bit too granular, but this is only an example on how flexible is test case management thanks to TFS and MTM.


Figure 8: the Task Inspector in Project has a warning because the resource Administrator was assigned to too many task.


Intellitrace for application does not get collected during a test run with MTM

If you have problem collecting intellitrace during a test run with MTM for an web application, and you really checked everything but cannot find any clue, check the user you are using to run the application. I have an installation where, using the default application pool will result in intellitrace being collected, while if I use a domain user to run application pool, when I try to signal a bug from MTM, no intellitrace is in the bug’s attachment.

The reason of this problem could derive from missing User Profile. Intellitrace is using User Profile to inject environment variables per owner of the application pool, and since User Profile is an OS feature, if no User Profile is loaded by OS, intellitrace could not inject variables, and data gets no collected.

The solution to this problem is to force OS to load user profile for the user before starting the test. Suppose the user that runs the application pool is named tfslab\webAppUsr, you simply need to run this command from a command prompt before you start the test

runas  /user:tfslab\webAppUsr /profile cmd.exe

This command simply opens a command prompt with the credentials of the user, thus the OS will create the User Profile. Now keep the cmd.exe windows open during the test, run the test again, and this time intellitrace data should be collected.