Test Explorer in VS 15.6

The latest version of VS introduces some new cool feature, and it is quite big (almost 1 GB of download). If I should give you a single reason to upgrade to this new version, is the new Test Runner that now has the ability to show tests hierarchically


Figure 1: Hierarchical view in action in Test Explorer.

Since I’m a great NUnit fan, I always like the ability to see my tests following the namespace structure of my test projects, this is surely much more better than a flat list. Actually I can detatch test runner from VS IDE and keep it on my second monitory to have a quick look at my test outcome.

The full list of bugfix and improvements can be found here.

Gian Maria

Troubleshoot “service unavailable” in TFS

Yesterday I’ve started an old virtual machine with an old version of TFS and when I try to access the instance I got a “Service Unavailable” error.


Figure 1: General Service Unavailable error for TFS Web interface

This error happens most of the time if you have wrong user credentials in the worker process used by IIS to run the TFS Application. To verify this assumption, you can simply open IIS Manager console and verify the status of the worker process that is used to run IIS (Figure 2)


Figure 2: The application pool in IIS is stopped.

As you can verify from Figure 2, the IIS app pool used to run service is stopped, if I tried to start again the pool, it immediately stopped again. This is usually the symptom of  bad authentication, this means that the pool is running with wrong user credentials. You can verify this in Event Viewer log, but absolutely avoid messing with the setting of the Application Pools directly, this is a task that should be demanded to the administration console.

As a general rule, you should never manually edit the configuration used for TFS in your IIS instance, everything should be done through the correct command

To fix this error open the Admin console and verify the Service Account used to run your TFS Instance


Figure 3: Application tier configuration, you can view Service Account user.

You can run Service Account with standard NETWORK SERVICE account, but I prefer using specific domain account, because I have more control on how all TFS Services will authenticate on the other machine of the domain. I changed the password of that account a couple of month ago, but that specific VM was never updated with the new credentials.

This is something that can happens in a domain, especially if you care about security and you force every account to change password every certain number of months. In this scenario my TFS instance cannot start again because it was still configured with the old password, but you can fix it with a couple of clicks.


Figure 4: Reapplying account can save your days when password of service user account of TFS was changed.

If you look in figure 4, the solution is really simple, because the Reapply Account command gives you the ability to re-enter the new password for the account, use the test function to verify that it is correct and once you press OK, the administration console takes care of everything.


Figure 5: Result of re-applying the account.

As you can see in Figure 5, the account is used not only in the application Tier, but also for Message Queue, TFSJobAgent and so on. This is the reason why I warned you not to fix the credential in IIS manually, doing this does not fix every place where wrong authentication are used.

Everything was green now green in the console, so I immediately tried to access the instance again, to verify that indeed everything is up and running again.


Figure 6: Everything is up and running again.

Gian Maria.

VSTS / TFS: use Wildcards for continuous integration in Git

If you setup a build in VSTS / TFS against a git repository, you can choose to trigger the build when some specific branch changed. You can press plus button and a nice combobox appears to select the branch you want to monitor.


Figure 1: Adding a branch as trigger in VSTS / TFS Build

This means that if you add feature/1312_languageSelector, each time a new commit will be pushed on that branch, a new build will trigger. Actually if you use GitFlow you are interested in building each feature branch, so you do not want to add every feature branch in the list of monitored branches.

If you look closely, it turns out that the combo you use to choose branches admit wildcards. If you look at Figure 2, you can verify that, pressing the add button, a new line is added and you have a textbox that allow you to “Filter my branches”.


Figure 2: Filter branch text

Actually the name is misleading, because you can simply write feature/* and press enter, to instruct VSTS to monitor each branch that starts with feature/.

Using wildcards in trigger list allows you to trigger a build for every feature branch.

For people not used to VSTS, this feature goes often unnoticed, because they think that the textbox can be used only to filter one branch. With this technique you can trigger a build for each change in each standard branch in GitFlow process.


Figure 3: Trigger with wildcards

As you can see in Figure 3, I instructed TFS / VSTS to build develop, master and all hotfix, feature or release branches.

Gian Maria

New cool feature of VSTS to limit impact of erratic tests

I’ve blogged some time ago about running UAT testing with a mix of Build + Release in VSTS. Actually, UAT testing are often hard to write, because they can be erratic. As an example, we have a software composed by 5 services that collaborates together, CQRS and Event Sourcing, so most of the tests are based on a typical pattern: Do something then wait for something to happen.

Writing tests that interact with the UI or are based on several services interacting togheter can be difficult.

The problem is: whenever you are “wait for something to happen” you are evoking dragons. Actually we wrote a lot of extensions that helps writing tests, such as WaitForReadModel, that poll the readmodel waiting for the final result of a command, but sometimes those kind of tests are not stable because during a full run, some tests are slower or simply they fails because some other tests changed some precondition.

We spent time to make test stable, but some tests tend to have erratic failure and you cannot spent tons of time to avoid this. When they fails in VS or Nunit console, you can simply re-run again only failed tests and usually they pass without a single error. This is perfectly fine when you are running everything manually, you launch tests in a VM, then come back after a little bit, if some tests failed, just re-run failed test and verify if something was really broken.

Erratic tests are really difficult to handle in a build, because if the build was always red because of them, the team tend to ignore build output.

When a build is often failing due to false-positive, team tend to ignore that build and it starts losing value.

Thanks to a cool new feature in VSTS, we can instruct our builds to re-run automatically those kind of tests. You can read everything about it in the release notes for VSTS.


Figure 1: Option to re-run failed tests

It could seem a little feature, especially because usually your tests should not be erratic, but actually it is a HUGE feature for UAT testing, when having not erratic test is simply not feasible or a really huge work. Thanks to this feature, the build can automatically re-run failed tests, completely eliminating the problem of erratic tests, and showing as failed only tests that really failed constantly for more than a given amount of runs.

Another situation where this feature has tremendous impact is when you want to  migrate a huge set of tests from sequential execution to parallel execution. If your test were written assuming sequential execution, when you run them in parallel you will have many failures due to interacting tests. You can simply create a build, enable parallel execution and enable re-run failed test. This will simply helps you to spot interacting tests so you can start fixing them so they can run in parallel.

If you look at Figure 2, you can verify that all erratic tests are easily identified, because you can filter all tests that passed on rerun. This will leave your build green, but can spot erratic tests that should be investigated to make them more stable.


Figure 2: Erratic test can be immediately discovered from build output.

I think that this feature is really huge and can tremendously helps running UAT and Integration tests in build and releases.

If you do not see the option in figure 1, please be sure you are actually using the version number 2 of the test runner task as shown in Figure 3.


Figure 3: Be sure to use version 2.* of the test runner, or the option to re-run failed test is not present.

Gian Maria.

Increase RAM for ElasticSearch for TFS Search

If you are experiencing slow search in TFS with the new Search functionality based on ElasticSearch a typical suggestion is to give more RAM to the machine where ES is installed. Clearly you should use HQ or other tools to really pin down the root cause but most of the time the problem is not enough RAM, or slow disks. The second cause can be easily solved moving data to an SSD disk, but giving more RAM to the machine, usually gives more space for OS File cache and can solve even the problem of slow disk.

ElasticSearch greatly benefit from high amount of RAM, both for query cache and for operating system cache of Memory Mapped Files

There are really lots of resources in internet that explain perfectly how ElasticSearch uses memory and how you can fine tune memory usage, but I want to start with a quick suggestion for all people that absolutely does not know ES, because one typical mistake is giving to ES machine more RAM but leaving ES settings unaltered.

Suppose I have my dedicated search machine with 4GB of RAM, the installation script of TFS configured ES instance to use a fixed memory HEAP of half of the available RAM. This is the most typical settings that works well in most of the situations. Half the memory is dedicated to Java Heap and gives ES space for caching queries, while the other 2 GB of RAM will be used by operating system to cache File access and for standard OS processes.

Now suppose that the index is really big and the response time starts to be slow, so you decided to upgrade search machine to 16 GB of RAM. What happens to ES?


Figure 1: Statistics on ES instance with HQ plugin shows that the instance is still using 2GB of heap.

From Figure 1 you can verify that ES is still using 2 GB of memory for the Heap, leaving 14 GB free for file system cache and OS. This is clearly not the perfect situation, because a better solution is to assign half the memory to java HEAP.

ElasticSearch has a dedicated settings for JAVA Heap usage, do not forget to change that setting if you change the amount of RAM available to the machine.

The amount of HEAP memory that ES uses is ruled by the ES_HEAP_SIZE environment variable, so you can simply change the value to 8G if you want ES to use 8 GB of Heap memory.


Figure 2: Elastic search uses an environment variable to set the amount of memory devoted to HEAP

But this is not enough, if you restart ElasticSearch windows service you will notice that ES still uses 1.9 GB of memory for the HEAP. The reason is: when you install ES as service, the script that install and configure the service will take the variable from the environment variable and copy it to startup script. This means that even if you change the value, the service will use the old value.

To verify this assumption, just stop the ElasticSearch service, go to ES installation folder and manually starts ElasticSearch.bat. Once the engine started, check with HQ to verify that ES is really using use 8GB of ram (Figure 2).


Figure 3: ES now uses the correct amount of RAM.

To solve this problem, open an administrator prompt in the bin directory of ElasticSearch, (you can find the location from the service as shown in previous post) and simply uninstall and reinstall the service. First remove the service with the service remove command, then immediately reinstall again with the service install command. Now start the service and verify that ES is using correctly 8GB of RAM.

When ElasticSearch is installed as a service, settings for Memory Heap Size are written to startup script, so you need to uninstall and reinstall the service again for ES_HEAP_SIZE to be taken in consideration

If you are interested in some more information about ES and RAM usage you can start reading some official article in ES site like: https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

I finish with a simple tip, ES does not love virtual machine dynamic memory (it uses a Fixed HEAP size), thus it is better to give the machine a fixed amount of ram, and change HEAP size accordingly, instead of simply relying on Dynamic Memory. 

Gian Maria.