TFS 2018 Update 3 is out, what changes for Search

TFS 2018 Update 3 is finally out and in release notes there is a nice news for Search functionality, basic security now enforced through a custom plugin. Here is what you can read in release notes

Basic authorization is now enabled on the communication between the TFS and Search services to make it more secure. Any user installing or upgrading to Update 3 will need to provide a user name / password while configuring Search (and also during Search Service setup in case of remote Search Service).

ES is not secured by default, anyone can access port 9200 and interact with all the data without restriction. There is a commercial product made by ElasticSearch Inc to add security (Called Shield), but it is not free.

Traditionally for TFS search servers, it is usually enough to completely close port 9200 in the firewall (if the search is installed in the same machine of Application Tier) or to open the port 9200 of Search Server only for Application Tiers instances if Search services are installed on different machine, disallowing every other computer of the network to directly access Elastic Search instance.

Remember to always ensure minimum attack surfaces with a good Firewall Configuration. For ElasticSearch the port 9200 should be opened only for TFS Application Tiers.

Here is the step you need to perform when you upgrade to Update 3 to install and configure search services: first of all in your search configuration you can notice a warning sign, nothing was really marked as wrong, so you can teoretically move on with configuration.

2018-09-05_13-14-38

Figure 1: Search configuration page in TFS Upgrade wizard, notice the warning sign and User and Password fields

When you are in the review pane, the update process complain for missing password in the Search configuration (Figure 1). At this point people get a little bit puzzled because they do not know what to user as username and password.

2018-09-05_13-15-00

Figure 2: Summary of upgrade complains that you did not specified user and password in search configuration (Figure 1)

If you move on, you find that it is Impossible to prosecute with the update because the installer complains of a missing ElasticSearch plugin installed.

The error ElasticSearch does not have a plugin AlmSearchAuthPlugin installed is a clear indication that installation on Search server was outdated.

2018-09-05_13-22-37

Figure 3: During Readiness check, the upgrade wizard detect that search services installed in the search server (separate machine) missed some needed components.

The solution is really simple, you need to upgrade the Search component installation before you move on with Upgrading the AT instance. In my situation the search server was configured in a separate machine (a typical scenario to avoid ES to suck up too resource in the AT).

All you need to do is to copy search installation package (You have a direct link in search configuration page shown in Figure 1) to the machine devoted to search services and simply run the update command.

2018-09-05_13-25-32

Figure 4: With a simple PowerShell command you can upgrade the installation of ElasticSearch in the Search Server.

The –Operation update parameter is needed because I’ve already configured Search services in this server, but for Update 3 I needed also to specify a user and password to secure your ES instance. User and password could be whatever combination you want, just choose a secure and long password. After the installer finished, all search components are installed and configured correctly; now  you should reopen the Search configuration page (Figure 1) in the upgrade wizard, specify the same username and password you used during the Search Configuration and simply re-run readiness checks.

Now all the readiness checks should pass, and you can verify that your ElasticSearch instance is secured simply browsing port 9200 of your search server. Instead of being greeted with server information you will be first ask for user and password. Just type user and password chosen during Search component configuration and the server will respond.

2018-09-05_13-29-08

This is a huge step to have a more secure TFS Configuration, because without resorting to commercial plugin, ElasticSearch is at least protected with basic authentication.

Remember to always double check your TFS environment for potential security problems and always try to minimize attack surface with a good local firewall configuration.

I still strongly encourage you to configure firewall to allow for connection in port 9200 only from TFS Application Tier machines, because is always a best practice not to leave ports accessible to every computer in the organization.

Gian Maria.

Increase RAM for ElasticSearch for TFS Search

If you are experiencing slow search in TFS with the new Search functionality based on ElasticSearch a typical suggestion is to give more RAM to the machine where ES is installed. Clearly you should use HQ or other tools to really pin down the root cause but most of the time the problem is not enough RAM, or slow disks. The second cause can be easily solved moving data to an SSD disk, but giving more RAM to the machine, usually gives more space for OS File cache and can solve even the problem of slow disk.

ElasticSearch greatly benefit from high amount of RAM, both for query cache and for operating system cache of Memory Mapped Files

There are really lots of resources in internet that explain perfectly how ElasticSearch uses memory and how you can fine tune memory usage, but I want to start with a quick suggestion for all people that absolutely does not know ES, because one typical mistake is giving to ES machine more RAM but leaving ES settings unaltered.

Suppose I have my dedicated search machine with 4GB of RAM, the installation script of TFS configured ES instance to use a fixed memory HEAP of half of the available RAM. This is the most typical settings that works well in most of the situations. Half the memory is dedicated to Java Heap and gives ES space for caching queries, while the other 2 GB of RAM will be used by operating system to cache File access and for standard OS processes.

Now suppose that the index is really big and the response time starts to be slow, so you decided to upgrade search machine to 16 GB of RAM. What happens to ES?

image

Figure 1: Statistics on ES instance with HQ plugin shows that the instance is still using 2GB of heap.

From Figure 1 you can verify that ES is still using 2 GB of memory for the Heap, leaving 14 GB free for file system cache and OS. This is clearly not the perfect situation, because a better solution is to assign half the memory to java HEAP.

ElasticSearch has a dedicated settings for JAVA Heap usage, do not forget to change that setting if you change the amount of RAM available to the machine.

The amount of HEAP memory that ES uses is ruled by the ES_HEAP_SIZE environment variable, so you can simply change the value to 8G if you want ES to use 8 GB of Heap memory.

image

Figure 2: Elastic search uses an environment variable to set the amount of memory devoted to HEAP

But this is not enough, if you restart ElasticSearch windows service you will notice that ES still uses 1.9 GB of memory for the HEAP. The reason is: when you install ES as service, the script that install and configure the service will take the variable from the environment variable and copy it to startup script. This means that even if you change the value, the service will use the old value.

To verify this assumption, just stop the ElasticSearch service, go to ES installation folder and manually starts ElasticSearch.bat. Once the engine started, check with HQ to verify that ES is really using use 8GB of ram (Figure 2).

image

Figure 3: ES now uses the correct amount of RAM.

To solve this problem, open an administrator prompt in the bin directory of ElasticSearch, (you can find the location from the service as shown in previous post) and simply uninstall and reinstall the service. First remove the service with the service remove command, then immediately reinstall again with the service install command. Now start the service and verify that ES is using correctly 8GB of RAM.

When ElasticSearch is installed as a service, settings for Memory Heap Size are written to startup script, so you need to uninstall and reinstall the service again for ES_HEAP_SIZE to be taken in consideration

If you are interested in some more information about ES and RAM usage you can start reading some official article in ES site like: https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

I finish with a simple tip, ES does not love virtual machine dynamic memory (it uses a Fixed HEAP size), thus it is better to give the machine a fixed amount of ram, and change HEAP size accordingly, instead of simply relying on Dynamic Memory. 

Gian Maria.

Monitor TFS Search data Usage

In previous post I’ve explained how to move searches component in a different machine in a TFS 2018 installation, now it is time to understand how to monitor that instance.

First of all you should monitor the folder that physically contains data, in my installation is C:\tfs\ESDATA (it is a parameter you choose when you configure the Search Server with Configure-TFSSearch.ps1 -Operation install PowerShell script). Data folder should be placed on a fast disk, usually SSD are the best choice. If the SSD will ever fail, you can always restart ES with empty data folder and use scripts to force TFS to reindex everything.

Data in ElasticSearch can be wiped away without problem, like WareHouse database, its content can always be restored.

If you want to have a better idea on what is happening to your search instance you can install plugins to easy management. First step: connect to the machine where the Search Service is running and locate the folder where ElasticSearch was installed. A simple trick is checking corresponding Windows Service.

image

Figure 1: Locate ElasticSearch binaries from Windows Service

Now simply open a command prompt and change directory to the bin installation directory of ElasticSearch, from here you can install various plugin to simplify ES management, I usually starts with the HQ plugininstallable with the simple command

plugin install royrusso/elasticsearch-HQ/v2.0.3

You should check the correct version in HQ home site, version 2.0.3 is the latest version that works with ES 2.4.1, the version used by TFS Search Service. For this command to being able to run, the machine should have an internet connection and GitHub site should not be blocked.

image

Figure 1: HQ was correctly installed

Now you can just browse to http://localhost:9200/_plugin/HQ to open the web interface of HQ plugin and connect to the actual instance.

image

Figure 3: HQ plugin up and running and ready to be connected to local instance of ES.

SNAGHTML4b1aed

Figure 4: Home page of HQ, where you can find the total size used by the server (1) and you can also have some Node Diagnostics (2)

From Node Diagnostics tool you can find if some statistics are not good, in my example I have search server in a server with slow disk and query time is sub-optimal. Usually ES is supposed to answer in less than 50 milliseconds, in my installation I have an average time of 660ms.

image

Figure 5: Statistics on search.

If you move the mouse over the number, it can give you some hint on the reason why the specific number is calculated and why is not considered to be good.

image

Figure 6: Help on the various number gives you an idea on what is wrong and how you could solve this.

If you are experienced slow search, there are many factors that can impact the instance of ElasticSearch and using a good plugin for diagnose the real root cause of the problem is usually the best approach.

I suggest you to navigate in HQ interface to familiarize with the various information it can gives to you, especially clicking on the name of a single node (there is only one node in my installation), you can get some interesting data on that single node, especially on RAM usage.

image

Figure7: Ram usage of a single node.

There are lots of other plugins that can show lots of useful information about your ElasticSearch instance, the installation is similar to HQ and you can install and use them with very few clicks.

Gian Maria.

TFS 2018 search components in different server

When it is time to design topology of a TFS installation, for small team the single server is usually the best choice in term of licensing (one one Windows Server license is  needed) and simplicity of maintenance. Traditionally the only problem that can occur is: some component (especially the Analysis and Reporting services) starts to slow down the machine if the amount of data starts to become consistent.

Single machine TFS installation is probably the best choice for small teams..

The usual solution is moving Analysis Service, cube and Warehouse database on a different server. With such a configuration if the analysis service has a spike in CPU, Disk or Memory usage, the main machine with operational DB and AT continue to work with standard performance. The trick is leaving the core services in a server that is not overloaded with other tasks, this is the reason why running a build agent in TFS machine is usually a bad idea.

With TFS 2017 a new search component is introduced, based on ElasticSearch. ES is de facto the most used and most advanced Search Engine on the market, but it tends to eat CPU RAM and Disk if the workload is big. The suggestion is to start configuring search on the very same machine with AT and DB (Single server installation) and move search components if you start to notice that ES uses too many RAM or is using too much CPU when it is indexing data. Moving search components on another machine is a really simple process.

First of all you need to remove the search feature, using the remove feature wizard as shown in Figure 1 (select the first node in administration console to find the Remove Feature Link)

SNAGHTML1521f72

Figure 1: Remove feature from TFS instance

Now you should choose to remove the search feature.

image

Figure 2: Removing the Search Service functionality

After the wizard finished, you should go to the folder C:\Program Files\Microsoft Team Foundation Server 2018\Search\zip and with powershell run the command

Configure-TFSSearch.ps1 -Operation remove

to completely remove Elastic Search from your TFS instance.

Do not forget to use Configure-TFSSEarch PowerShell script to completely remove any trace of Elastic Search from the TFS machine

Now search component is not working anymore, if you issue a search you should get a message like shown in Figure 3:

image

Figure 3: Search component is not operational anymore.

At this point you open TFS administration console and start the wizard to configure the search, but this time, instead of configuring everything on the current machine, you will choose to use an existing Search Service.

image

Figure 4: Use an existing search service instead of install locally the search service

If you see in Figure 4, the instructions to install search service on another computer are really simple, you need to click the “search service package” link in the wizard to open a local folder that contains everything to setup ElasticSearch and the search service. Just copy content of that folder on another machine, install java and set the JAVA_HOME environment variable and you are ready to install Search Service.

You can find a README.txt file that explain how to configure the search service, just open a PowerShell console and then run the command

Configure-TFSSearch.ps1 -Operation install -TFSSearchInstallPath C:\ES -TFSSearchIndexPath C:\ESDATA

Please change the two folder if you want to change the location of ElasticSearch binary and ElasticSearch data. After the script ran without any error, it is time to verify that ElasticSearch is correctly installed and started. As usual, please create a DNS entry to avoid using the real name of the machine where you installed the service. In my installation I’ve configured the name tfssearch.cyberpunk.local to point to the real machine where I configured search services. Just open a browser and issue a request to http://tfssearch.cyberpunk.local:9200

image

Figure 5: Try to access local instance of ElasticSearch using DNS name

Please pay attention at firewall configuration, because ElasticSearch has no security in base installation and everyone can mess up and read data just browsing in port 9200

Now you should open your firewall to allow connection to port 9200 from every machine where an Application Tier is running. In my situation the machine with TFS installation has IP number 10.0.0.116. Remember that Elastic Search has NO Authentication (it is a module that is not free, called shield), thus it is really better to allow connections only from the TFS machine. All you need to do is creating a rule to open port 9200, but allowing connection only from the IP of the TFS machines.

image

Figure 6: Firewall configuration, the machine with the search service opens port 9200 only from the IP of the TFS machine.

Now remote desktop to TFS machine, and verify that you are indeed able to browse http://tfssearch.cyberpunk.local:9200, this confirm that configuration of the firewall allows TFS to contact the search service. Then try to access the very same address from another computer in the network and verify that it CANNOT access ElasticSearch instance. This guarantees that no one in the network can access Elastic Search directly and mess up with its data.

Now you can proceed with the Configuration Wizard in TFS instance, specifying the address of your new Search Server

image

Figure 7: Configure new search service in TFS.

Proceed and finish the wizard. At the end TFS machine will re-index all the data in the new search server, just wait some minutes and you will be able to use again search, but now all Search Components are on a dedicated server.

Gian Maria.

Case sensitivity in lucene search

We ended last post with a good knowledge of how to do complex searches in Lucene.NET indexes but a question remains unresolved, is Lucene.NET search case insensitive or case insensitive?. Suppose you search for

+mime + format

Here is the first result returned from the above query.

image

Figure 1: Searching for +mime +format returns a document that contains MIME in uppercase.

As you can see this documents satisfy the query with the word MIME, thus suggesting that the query is Case Insensitive (you searched for mime but MIME satisfied the search), but this is not true. If you look at Lucene documentation you can find that all searches are always case sensitive, and there is no way to do case insensitive search. This fact usually puzzled the user because searches seems to be Case insensitive, so where is the trick?. The answer is: StandardAnalyzer transforms in lowercase all the tokens before storing them into the index and QueryParser lowercase all the terms during query parsing.

image

Figure 2: The query resulted from the parsing of +MIME +FORMAT

As you can see in Figure 2 I’ve showed the result of parsing the query +MIME +FORMAT; the result correctly has the default field content before each search term, but it also present every term in lowercase. If you look closely at the constructor of QueryParser you can verify that it needs a reference to the analyzer used to create the index. During initialization QueryParser ask to that analyzer if he manipulate tokens before storing them to the index to apply the very same manipulation to search terms during query parsing. If you manually create a query to issue a search for term MIME (all uppercase) with this simple line of code you can be surprised by the result.

Query query = new TermQuery(new Term("content", "MIME"));

The index does not return a single result even if the previous query showed in Figure 1 that the word MIME is present in the original text. The reason is: StandardAnalyzer converted every term in lowercase so the index contains term mime not MIME and the above query has no result. This example shows how important is to initialize the QueryParser with the very exact Analyzer class type or you can incur in really bad surprise.

To build a Case Sensitive index you can simply create your own analyzer, inheriting from StandardAnalyzer and overriding the TokenStream method

    public class CaseSensitiveStandardAnalyzer : StandardAnalyzer
    {
        public CaseSensitiveStandardAnalyzer(Lucene.Net.Util.Version version) : base(version) { }

        public override Lucene.Net.Analysis.TokenStream TokenStream(string fieldName, System.IO.TextReader reader)
        {
            StandardTokenizer tokenizer = new StandardTokenizer(reader);
            Lucene.Net.Analysis.TokenStream filterStream = (Lucene.Net.Analysis.TokenStream)new StandardFilter((Lucene.Net.Analysis.TokenStream)tokenizer);
            //Lucene.Net.Analysis.TokenStream tokenStream = (Lucene.Net.Analysis.TokenStream)new LowerCaseFilter(filterStream);
            Lucene.Net.Analysis.TokenStream stream = (Lucene.Net.Analysis.TokenStream)new StopFilter(true, filterStream, StopAnalyzer.ENGLISH_STOP_WORDS_SET, true);
            return stream;
        }
    }

The code inside TokenStream() method is similar to the one contained in StandardAnalyzer and shows clearly its default configuration. The TokenStream is a specific stream that permits to “tokenize” the content to split words from the original content. For a StandardAnalyzer the base tokenizer is of type StandardTokenizer  a grammar enabled tokenizer, capable to recognize specific patterns of english language. In the second line the the StandardTokenizer is used as base stream for a StandardFilter that is used to handle standard punctuation like aphostrophe or dots. As you can see each new Stream is created passing the previous one as a source to build a chain of filters that gets applied to tokens one after another. The third line was commented in my analyzer because it is the one that adds the LowerCaseFilter, used to transform each token to lowercase and finally the StopFilter remove some noise words that should not be indexed because are known to be noise in the English language. If you comment the line that adds the LowerCaseFilter, all the tokens will be stored inside the index with original casing obtaining a Case Sensitive index. If you need to support case sensitive and case insensitive search in the same application, you need to build two different indexes in two distinct folders and use the right one during searches.

With this Analyzer I created another index of the first five million post of the dump of StackOverflow, then I issued the query +Mime +Format, obtaining this result:

Insert search string:+Mime +Format

N°4 found in  9 ms. Press a key to look at top results

As you can see, searching for +Mime +Format returns only 4 resultsw, because now the search was case sensitive so words like MIME does not satisfy the query. The cool part is that QueryParser correctly identifies that our new analyzer does not apply the LowerCaseFilter so it does not lowercase terms during query parsing.

This example shows how you can  manage search casing with lucene and gave you some golden rule:

Rule #1: all lucene searches are case sensitive, if you build query directly with TermQuery class you need to be aware of casing used by the analyzer.

Rule #2: standard analyzer applies a lowercase filter to tokens to make search case insensitive.

Rule #3: if you need a Case Sensitive search ,you need to create an index with an Analyzer that does not apply lowercase to tokens, such as the above CaseSensitiveStandardAnalyzer

Rule #4: always pass to QueryParser constructor the very same type of Analyzer used to create the index, because QueryParser needs it to determine if search terms should be lowercased to build the query.

Happy searching 🙂

Gian Maria.