Previous posts on the NoSql and Raven Series

• NoSql and a life without Schema
• NoSql and a life without schema continued

Other posts by Mauro on RavenDb Subject.

• RavenDb: Start your engines
• RavenDb: First Contact

In previous articles I showed how simple is to store objects inside a NO SQL database because the data storage has no schema and does not require you to specify the format of your data. I decided to use RavenDb as a NoSql storage to show some basic concepts of NoSql and I showed also how simple is to add a new property to a document, because RavenDb takes care of everything, just save and load objects and the new property is just there.

Now I want to deal with a different kind of problem: what happen when you rename a property of a document that has already some instances saved in database?

This is the typical situation where having No Schema does not solves the problem automatically. Suppose you change the Player document renaming Description property to Background, if you load an old document from the database, since the Description property does not exists anymore, it is ignored and when the object is saved again, it will be deleted. Clearly this is unacceptable because it lead to data loss, so there is some need of manual intervention to handle the scenario of property rename.

There are multiple approaches to solve this problem, but the simplest one is verifying if your NoSql storage supports the concepts of Bulk-Updating documents, a feature that is really well supported by RavenDb.

RavenDb natively support the concept of  Document Patch, a technique used to update a single document without the need to load the entire object or replacing its entire content. Basically a Patch is a dedicated command that transform a stored document directly in the store; along the many different type of patches supported by RavenDb you can use the rename patch specifically designed to change the name of a property. The code is really simple, just reference the assembly Raven.Abstractions and write this code.

   1: store.DatabaseCommands.Patch("Players/1",

   2:       new[] { new PatchRequest() {

   3:         Type = PatchCommandType.Rename,

   4:         Name = "Description",

   5:         Value = new RavenJValue("Background"),

   6:     }

   7: ;

The syntax is really simple, the IDocumentStore has a property called DatabaseCommands used to access all commands supported by RavenDb engine; with this property you can call the Patch() method specifying the Id of the object you want to patch and an array of PatchRequest (you can ask to apply a sequence of patches, in this example I only want to rename a property). To request a Patch operation you need to specify the type of Patch you want to execute, in this example PatchCommandType.Rename, then you need to fill the appropriate properties of PatchRequest object, required by the type of the path you are issuing. To rename a property you need to specify the Name of the property in the Name field and the new name of the property in the Value parameter, passed as RavenJValue element.

This technique is useful but it does not solve our original problem, because it patches a single entity, while our need is to rename the property Description of all saved documents of type Player, so we need to use a different type of operation called: set-based-operation. A set-based-operation is an operation that operates on multiple documents at once, in its most basic form it request an index to identify the list of documents to modify and a list of Patch operation. The whole concept of indexes is a really fundamental concept in RavenDb and in all NoSql database, but for this specific need just think to index as a way to create a query that identify documents in database. To define an index by code you must create a specific class for the index, but I can easily create an index with RavenStudio. In Figure 1 I created an index to identify all the documents of type Players.

Figure 1: Define an index in RavenStudio

To define a simple index you just need to give it a name and write the text of the index. The index starts with from doc in docs, this part is used to query all documents of the database, then I specify a restriction with the keyword where followed by the condition.

In this example the condition is doc[”@metadata”][“Raven-Entity-Name”] == “Players”, and uses RavenDb Metadata, a series of internal properties that RavenDb attach to each saved document. One of this metadata is called Raven-Entity-Name and for .NET object is the name of the class followed by an s to pluralize the name. In my example, since the .NET class saved is called “Player” the Raven-Entity-Name is equal to “Players”. The last part of the Index Map is the select clause, where I simply select property Name of the document, but it is not important for our example, because I’m only interested in creating an index to identify all documents that corresponds to Player entity.

You can execute the index from RavenDb Studio just to verify that it works as intended and it selects all the entities of desired type. When the index is ok you are able to issue a Set-Based-Patch operation:

   1: store.DatabaseCommands.UpdateByIndex("PlayersToBeUpdated",

   2:       new IndexQuery(),

   3:       new [] {new PatchRequest() {

   4:           Type = PatchCommandType.Rename,

   5:           Name = "Description",

   6:           Value = new RavenJValue("Background"),

   7:       }

   8: });

As you can see the syntax is really similar to the previous example, the main difference is that you should not use the Patch()  database command but UpdateByIndex() that requires:

• the name of the index to use
• an index query (to specify index parameters, if any)
• the list of Patches request.

The above command basically means:

Apply the list of Patches to all object identified by the index named “PlayersToBeUpdated”

This simple call updates all Player objects saved in database, renaming the property, as you can verify from RavenDb Studio.

Figure 2: Description property was renamed to Background

As you can see, even if NoSql databases does not require schema, if you refactor your documents changing the name of existing properties, you need to update all saved data to reflect your changes.

In the next series of posts I’ll deal with RavenDb HTTP API to show how to rename a property directly with HTTP requests, without the need to use C# code. This is useful if you want easily to generate a bat script that updates documents using only HTTP Requests.

Gian Maria

If you issue a DBCC CHECKDB on a big database to verify for consistency errors, it will take a long time to complete, but the Management Studio windows usually does not give you any hint about how long does it take, or a percentage progress. Luckily enough sql server has a Dynamic Management View that can solve your problem.

This is the SQL code to visualize progress of the operation

   1: SELECT  session_id ,

   2:         request_id ,

   3:         percent_complete ,

   4:         estimated_completion_time ,

   5:         DATEADD(ms,estimated_completion_time,GETDATE()) AS EstimatedEndTime, 

   6:         start_time ,

   7:         status ,

   8:         command 

   9: FROM sys.dm_exec_requests

  10: WHERE database_id = 16

In my example I filtered the results only for the database and used the Id of the database that you can obtain with the DB_ID function.

An example of what you got with this query is represented in the following picture.

As you can see you can easily visualize percentage of completion, estimated end time and the command that is running.

Gian Maria.

In the first part I showed how simple is to store object inside a NoSql database like RavenDb, today I want to point out how cool is having no schema when it is time to add properties to your documents.

Suppose that your Player entity changed and you add a new property called Description and you already saved some Players on RavenDb. The question is: what happens when you try to save a new Player now that the class has another property? What happens when you load an old entity, that was saved when that property did not exist?

Since we have no schema we have absolutely no problem, just save another object to Raven and you can find that it got saved without even a warning.

Figure 1: New object is saved without any problem even if it is different from the old one

If you double click a Document (Raven call each object saved to the db a Document) you can simply look at its content and the new property called Description got saved without any problem.

Figure 2: The object contains the new Description property

Probably you had already recognized JSON format to store the object, but the question is, what happened to the old object saved before you added the Description property ? The answer is: it is still there, clearly without the Description property.

Figure 3: The old object does not contain the Description property, but it can coexist with the new one without any problem

Now if you load all object from the database, you can find that for this instance Description property is simply null, because it did not exists in the database, but no error or warning occurred. If you simply load all Player entities and finally issue a SaveChanges, as shown in the following snippet:

   1: using (var session = store.OpenSession())

   2: {

   3:     foreach (var player in session.Query<Player>().ToList())

   4:     {

   5:         Console.WriteLine(player.Name + " " + player.RegistrationDate + " " + player.Description);

   6:     }

   7:     session.SaveChanges();

   8: }

you will end with an automatic update of object and now if you look in the database you can see in figure xxx now the player entity has a Description property equal to null

Figure 4: If you ask to SaveChanges, the old player object with id:1 is updated in Raven, now the Description property is present.

The same happens for removed properties, they simply are ignored during the load process, and removed during the save.

This really trivial example shows how simple is to deal to add or remove a property from documents when your data storage is a NO SQL one and you have no need to define a schema.

Gian Maria.

NoSql is not a replacemente for SQL databases, but it is a valid alternative for a lot of situations where standard SQL is not the best approach to store your data. Since we were taught that whenever you need to store data on a “data store” and you need to query that data for retrieval, SQL is the best solution, you have only to decide what Sql Engine to use and the game is done.

In 2012 this sentence has proven wrong, what I mean is that is not possible to assume anymore that SQL is the “only way to go” to store data; but you should be aware that other alternative exists and it is called NO SQL. Under this term we have various storage engine that are not based on SQL and in .NET we have an exceptional product called RavenDB (you can find a really good introduction to RavenDb in Mauro’s Blog).

The first big difference with standard Sql is being Schemaless. One of the most annoying restriction of Sql Server is the need to specify exactly the format of the data you want to store inside your storage. This is needed for a lot of good reason, but there are situation when you really does not care about it, especially if your software is heavily based on OOP concepts. Suppose you have this object

   1: class Player

   2: {

   3:     public String Name { get; set; }

   4:  

   5:     public DateTime RegistrationDate { get; set; }

   6:  

   7:     public Int32 Age { get; set; }

   8: }

For a moment do not care about the fact that this object is not well encapsulated (it has public getter and setter) but focus only on the need to “store” this object somewhere. If you use a standard Sql storage, first of all you need to create a table, then define columns, decide maximum length for the Name column and finally decide an ORM to use or build a dedicate data layer and finally you can save the object.

If you work with raven, this is the only code you need

   1: var store = new DocumentStore { Url = "http://localhost:8080" }; 

   2: store.Initialize();

   3: using (var session = store.OpenSession())

   4: {

   5:     var player = new Player

   6:     {

   7:         Age = 30,

   8:         RegistrationDate = DateTime.Now,

   9:         Name = "Alkampfer",

  10:     };

  11:     session.Store(player);

  12:     session.SaveChanges();

  13: }

I simply created a DocumentStore based on a local server, opened a session and saved an object, I did not defined anything on the server, I did not need to have an ORM, the server simply takes the object and save it, period!

I liked very much this approach because

I needed to save an object to a data storage and everything I need is just a two function all, Store to tell the storage the object I want to save and SaveChanges that actually do the save.

What I got with this simple snippet of code? Just browse with a standard browser to the address of the server and you should see the content of the database.

Figure 1: Content of the database after insertion of a simple object

From Figure 1 you can see content of the raven database, it contains a player and the little 1 beside the object is the Id that Raven uses internally to uniquely identify that object. The other object called Sys Doc Hilo/players takes care of id generation for Players object with an Hilo algorithm.

That’s all folks, no need to define schema, no need to have special Id property or any other requirement to make the object compatible with the store, just call Store method on whatever .NET object and your object is inside the database, Period!.

This is only a scratch of the many functionalities of RavenDb , more to come in my blog and in Mauro’s one.

Gian Maria.

I’ve a stored procedure with a query that runs on a quite big database, it was slow (more than one minute to run) and was optimized using a temp table. The result is that execution time dropped to ~2 secs, and since this was acceptable the optimization stopped.

After a couple of months, the query become really slow again, it got executed in ~30 secs and I started to investigate why.

A quick tour with SSMS and STATISTICS I/O did not reveal some news, but the execution plan have a strange sort operation that takes 90% of the time of the overall query and this is really strange, but I did not find any reason why it should slow the query so much. To have a better picture of what is happening I decided to fire the Activity Monitor to check if the query stops for any lock in table, but I found that the task that is executing the query goes into heavy parallelism (you see a lot of row in the activity monitor with the same id), but each subtask is waiting a lot with a  CXPACKET wait type and everything seems stuck. CXPACKET wait time happens when the execution of the query is parallelized so I decided to disable parallelism of the query with OPTION (MAXDOP 1) to verify if the situation change. The result is that the query now executed in ~2 secs, like it did two months before, so I decided to leave it with parallelism disabled and it start to run just fine.

The conclusion is,: when database is big, and query are complex, it is not so simple to understand why a query is slow, sometimes you need to tweak how SQL Server issue the query with Query Hint and you should use all the tools you have to understand what is really happening .

Gian Maria.

In a production Sql Server database we had some issue with the hardware, the result is that one very big database started to gave us errors on DBCC CHECKDB, the error is the following one.

Msg 8929, Level 16, State 1, Line 1
Object ID xxxxxx, index ID 1, partition ID xxxxxx, alloc unit ID xxxxx (type In-row data): Errors found in off-row data with ID xxxxxxx owned by data record identified by RID = (3:34252:4)

This happens in one of the most central and important table of the database, that contains millions of record, and only 20 rows gave us error, so I’d like to verify witch rows had the error, because in that table there are lots of records in logical status “deleted” and the corresponding physical row can be removed from the database without much pain and reinserted with the very same data to avoid dataloss.

So my question is… how can I identify the row given the RID (Row Identifier?) First of all from Object ID you can easily find the name of the table with select Object_Name(<Object_ID>) but what about the RID. After a little search I find the sys.fn_physLocFormatter function and I created this query

   1: SELECT sys.fn_PhysLocFormatter (%%physloc%%) AS RID,  

   2:         *

   3: into debug.RIDTABLE 

   4: FROM TableName

I selected into a table in Debug schema the RID of the Row as well as all the other columns of the table with error, now I take the output of the DBCC CHECKDB, selected all the RID of rows with errors and I could execute the query

   1: select * from debug.RIDTABLE

   2: where RID IN ('(3:34252:4)', '(x:xxxxx:x)'..... 

To extract from the RIDTAble all the row with errors, then I can simply delete them from the original table, and insert them again from the RIDTABLE. Checking the data inside RIDTABLE it seems that the data is correct, except for a NVARCHAR(MAX) field that contains long string of text.

After the process DBCC CHECKDB did not give any other error, so I was quite happy to solve this problem without the need to put the database in single-user-mode, but the most important aspect is that I was able to identify the rows with error, to understand the impact of errors in the software. It turned out that only a couple of lines contained really important data, the others are in Deleted status, so we did not lost anything so important.

Gian Maria.

I’ve more to say about the Traffic Light example I explained in a previous post, if you look at that post, you can notice that the Domain is composed by properties that have only getter and not setters. This needs is explained in this post, where I explain why you should protect the status of your entities from external direct manipulation.

If I wish to use my simple Traffic-Light example in a class to teach OOP principle, it is not good enough, because it breaks the concept of Encapsulation. As stated in Wikipedia

encapsulation means that the internal representation of an object is generally hidden from view outside of the object’s definition. Typically, only the object’s own methods can directly inspect or manipulate its fields

Having no setters prevent status from external modification, but the getters permits to the outside world to view inside the object. If you think that getters are not an antipattern and that you can always use them in your domain classes without problem, probably you are still thinking in procedural code, or at least not fully Object Oriented.

If you permit to the outside world to read one property of an object, you risk that the outside world will use that value to implement logic that should be contained in the object, so you are breaking encapsulation.

Whenever you ask yourself “I need to search in code all part of the software that access this property because I’m going to modify it”, you are implicitly looking for logic that is actually using that property but it is outside the object. The result is that you have piece of Business Logic that

• Read properties of several objects
• implement logic based on these properties.

Then all these objects are bounded together by that external logic and you loose encapsulation and you are not fully OOP. Based on this considerations I took my traffic light sample and decide to try to remove all getters and thanks to DOMAIN EVENTS it was really simple, more than I tought. Here is my new domain.

Even if this is a trivial domain, I appreciate a lot that I ended having a couple of DOMAIN EVENTS composed only by readonly properties (events are in the past and are immutable) and a couple of Domain Classes that contains only methods. Removing all properties actually does not required me to modify any test, and actually I found that each time I found some code that use a getter, that code looked really better after the access to the getter is removed.

If you really can structure your core domain with object that contains only methods, you have a really modular domain where each class is really an autonomous piece of software that does some business in the domain without the need of a direct relation with other entities except communicating with Domain Events.

I know that this is a super simple trivial example (a couple of traffic light is not even close to a real business case), but the result of this stupid and little experiment convinced me that it is the right road.

Gian Maria.