Hilight matched text inside documents indexed with Solr plus Tika

I’ve already dealt on how to index documents with Solr and Tika and in this article I’ll explain how you can not only search for documents that match your query, but returns even some text extract that shows where the document match the query. To achieve this, you should store the full content of the document inside your index, usually I create a couple of fields, one called Content that will contain the content of the file, and with a copyfield directive (  <copyField source=”content” dest=”text”/> ) automatically copy that value inside the catch all field called text.

   <field name="content" type="text_general" indexed="false" stored="true" multiValued="false"/>
   <field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>

Text field is multivalued and not stored, it is only indexed to permit search inside various field of the document. Content field store the extracted text from Tikaand it is useful both for highlighting and to troubleshoot extraction problems, because it contains the exact text extracted by Tika.

Now suppose you want to search for the term Branch and want also to highlight the part of the text where you find that term, you can simply issue a query that ask for highlighting, it is really simple.

http://localhost:8080/TestInstance/tikacrawl/select?q=text%3Abranch&fl=id%2Ctitle%2Cauthor&wt=xml&hl=true&hl.snippets=20&hl.fl=content&hl.usePhraseHighlighter=true

This simple query  ask for document with text that contains word branch, I want to extract (fl=) only title and author fields, want xml format and with hl=true I’m asking for snippet of matching text, hl.snippets=20 instruct solr to search for a maximum of 20 snippet and hl.usePhraseHighlighter=true use a specific highlighter that try to extract a single phrase from the text. The most important parameter is the hl.fl=content that specify the field of the document that contains the text used for highlight.  In the results, after all matching documents there is a new section that contains all the highlights for each document

image

Figure 1: Hilight for the TFS Branching Guid – Scenarios 2.0.pdf file

The name of the element match the id of the document (in my configuration full path of the file), and there a list of highlights that follows. But the true power of Solr comes out if you start to use languages specific fields.

   <field name="content" type="text_en" indexed="false" stored="true" multiValued="false"/>
   <field name="text" type="text_en" indexed="true" stored="false" multiValued="true"/>

I’ve just changed in schema.xml the type of Content and Text from general_text to text_en and this simple modification enables a more specific tokenizer, capable of doing full-text searches. Suppose you want to know all documents that deal with branching strategies, here is a possible query

http://localhost:8080/TestInstance/tikacrawl/select?q=text%3A%22branch+strategy%22~3&fl=id%2Ctitle&wt=xml&hl=true&hl.snippets=5&hl.fl=content&hl.usePhraseHighlighter=true&hl.fragsize=300

The key is in the search query text:”branch strategy”~3 that states I’m interested in documents containing both branch and strategy terms and with a relative distance of no more than three words. Since text was indexed with text_en field type I got full-text search, and I have a confirmation looking at the highlights.

image

Figure 2: Highlights for a proximity query with full text, as you can see word branching matches even if I searched for branch

And voilà!! You have full-text searching inside file content with minimal amount of work and a simple REST interface for querying the index

Gian Maria.

Import folder of documents with Apache Solr 4.0 and tika

In a previous article I showed how simple is to import data from a Sql database into Solr with a Data Import Handler, in this article I’ll use a similar technique to import documents stored inside a folder.

This feature is exposed by the integration with Tika, an open source document analyzer capable of extracting text by various formats of files. Thanks to this library solr is capable of crawling an entire directory, indexing every document inside it with really minimal configuration. Apache Tika is a standalone project, you can find all supported formats here and you can use directly from your java (or .NET code) but thanks to Solr Integration setting everything up is a real breeze.

First of all you need to copy all required jars from solr distribuzion inside lib subdirectory of your core, I strongly suggest you to grab all the files inside contrib\extraction\lib subdirectory of solr distribution and copy all of them inside your core, in this way you can use every Data Import Handler you want without incurring in errors because a library is not available.

To import all files you can simply configure an import handler as I described in the previous article, here is the full configuration

<dataConfig>  
	<dataSource type="BinFileDataSource" />
		<document>
			<entity name="files" dataSource="null" rootEntity="false"
			processor="FileListEntityProcessor" 
			baseDir="c:/temp/docs" fileName=".*\.(doc)|(pdf)|(docx)"
			onError="skip"
			recursive="true">
				<field column="fileAbsolutePath" name="id" />
				<field column="fileSize" name="size" />
				<field column="fileLastModified" name="lastModified" />
				
				<entity 
					name="documentImport" 
					processor="TikaEntityProcessor"
					url="${files.fileAbsolutePath}" 
					format="text">
					<field column="file" name="fileName"/>
					<field column="Author" name="author" meta="true"/>
					<field column="title" name="title" meta="true"/>
					<field column="text" name="text"/>

				</entity>
		</entity>
		</document> 
</dataConfig>  

 

This is a really simple import configuration but there are some key point you should be aware of.

All of my schema.xml have a unique field called id, that serves me as a purpose of identifying the document, and is the key used by solr to understand if a document should be inserted or updated. This importer uses a BinFiledataSource that simply crawl inside a directory looking for files, and extracting standard values as Name of the file, last modify date and so on, but it does not know the text inside the file. This dataSource has a document entity called files, and has a rootEntity=”false” because this is not the real root entity that will be used to index the document. Other attributes simply states the folder to crawl, document extension to index and so on.

In that entity you find columns related to file attributes and I’ve decided to include in the document three of them

  1. fileAbsolutePath that will be used as unique index of the document
  2. fileSize that contains the size of the file in bytes
  3. fileLastModified that contains the date of last modification of the document

After these three fields there is another entity, based on TikaEntityProcessor, that will extract both text and metadata from the document. This entity is the real entity that will be indexed and it is the one that use Tika library to extract all information from documents, not only file related one. It basically extract all the text plus all file metadata attribute if presents. Here is the list of the field I want to store inside my index

  1. file: contains files name (opposed to the id field that contains the full path of the file)
  2. Author and Title: both are metadata of the document, and they are extracted by Tika if present in the files
  3. text: that contains the text of the document.

Clearly you should define all of these field accordingly inside your schema.xml file.

   <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" /> 
   <field name="fileName" type="string" indexed="true" stored="true" />
   <field name="author" type="string" indexed="true" stored="true" />
   <field name="title" type="string" indexed="true" stored="true" />
   
   <field name="size" type="plong" indexed="true" stored="true" />
   <field name="lastModified" type="pdate" indexed="true" stored="true" />
   
   <field name="text" type="text_general" indexed="true" stored="false" multiValued="true"/>

This is all you need to do, really!!. You can now toss some documents inside specified folder, then go to solr console and try to execute the import.

image

Figure 1: Importing documents from Solr Web UI

If you have error during the import process, please refer to solr logs to understand what went wrong. The most obvious problem are: missing jar file from the lib directory, document missing some mandatory fields specified in schema.xml (such as missing id).

Now you can query and look at what is contained inside your index.

image

figure 2: A standard catch all query to verify what is inside your index

One of the cool feature of tika is extracting metadata and text from your files, as an example you can search for text that contains the word “rebase” with the query text:rebase

image

Figure 3: Return of a search inside the text of the document

As you can see, it founds progit.en.pdf book, and if you look at the properties you can find that this book does not contains nor author or title metadata, because they are missing in the original document. Since those fields are not mandatory, if they are not present in the file, nothing bad happens, but you are still able to search inside original text of the pdf file.

Gian Maria

Loading data from Sql Server to Solr with a Data Import Handler

Apache Solr is an exceptional engine for Enterprise Search based on Lucene and usually the first question I got is: how can I integrate Solr with an existing Sql Server data storage to power up searches.

Solr is used not as a primary data store because it is a Search Platform whose primary purpose is giving the ability to do complex searches with blazing performance. This means that you usually have your data in a primary data store, Es. Sql Server and you need to move data to a Solr server to power up your searches.

Quite often you do not need real-time update, you can simply update your solr server nightly, or at given interval (update each X hours). In this scenario there is not the need to setup a mechanism that immediately replicate modification of Primary data store to Solr and you can rely on DIH: Data Import Handler. Suppose you have your data inside a Sql Server inside a table called tag; the first operation you need to do is locating all jar libraries that are needed to use DIH with Solr.

image

Figure 1: Copy all needed library inside the lib folder of core home directory

I use Solr with multi core configuration, my solr test instance is installed in c:\solr\TestInstance and inside that root directory I have multiple cores used to run different schema and configurations. Inside each core you should create a lib directory to store all jar files are needed for that specific core. To run DIH you need to copy apache-solr-dataimporthandler and apache-solr-dataimporthandler-extras taken from solr main distribution (zipped file you download from Apache Solar site). I’ve added also sqljdbc4.jar that contains classes needed to connect to a SqlServer database from java jdbc.

First step is creating the Data Import configuration file, to specify how you want to import data inside your Solr.

dataConfig>  
	<dataSource type="JdbcDataSource" 
			driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" 
			url="jdbc:sqlserver://10.0.0.102;databaseName=Dictionary;" 
			user="sa" 
			password="xxxxxxxx" 
			batchSize="5" /> 
			
    <document name="TestDocument">  
        <entity name="TestEntity" query="SELECT * FROM tag">  
			<field column="Id" name="id" />
            <field column="Term" name="term" />  
            <field column="Name" name="name" />  
        </entity>  
    </document>  
</dataConfig> 

I usually call this file data-import.xml but you can use whatever name you want. DataSource node is used to specifies class of the driver to use, connection string with username and password and other option, Es batchSize to specify how many row at a time should be fetched from the data store. Once you have defined a DataSource you should at least specify one Document containing one entity, where each entity is composed at least by a query and a series of fields. The query is the simple SQL code that will be issued to the Data Store and fields are used to specify mapping from column of the query and field in schema.xml of that core.

This is the minimal configuration that is used to load data from Sql Server to Solr, after you created this file you need to specify the handler inside the solrconfig.xml file.

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">     

	<lst name="defaults">    
		<str name="config">data.import.xml</str>  
	</lst>    
</requestHandler>  

Now everything is in place, you can go to the Dataimport section of the core you configured and you can execute the import handler.

image

Figure 2: Import data with standard solr web interface.

You can now check that after the import solr contains data you expect.

image

Figure 3: Simple *:* query to verify that data was really imported from the DIH

Thanks to few xml lines of code you are able to import data from Sql Server to Solr without the need to write a single line of code.

Gian Maria.