Renaming a property in RavenDb with HTTP API

Previous posts on the NoSql and Raven Series

Other posts by Mauro on RavenDb Subject.

In third post of the series, I dealt with renaming of properties in an entity that has already some instances saved in RavenDB. The solution proposed was based on PATCH commands issued by C# code based on an Index created with RavenStudio, but now that I showed you how to issue HTTP API request, it is time to understand how to issue set-based PATCH commands to modify multiple documents with simple HTTP call.

First of all I want to refresh the concept that a Set-Based operation operates on indexes, thus if you want to PATCH multiple documents at once, you should create an index that identify all the documents you want to PATCH. We already saw how to create an index that identify all Documents of a certain type with C#, but if you want to create an index with HTTP api you can simple issue this Curl command

   1: curl -X PUT http://localhost:8080/indexes/PlayersToBeUpdated 

   2:     -d "{ Map: 'from doc in docs where doc[\"@metadata\"][\"Raven-Entity-Name\"] == \"Players\" \r\nselect new { doc.Name}'}

the –X PUT is used to specify a PUT request, used to store something in database, the address identifies the name of the index and should respect the mask:, –d option identify the body of the request, and it is absolutely equal to the one showed in previous post from RavenDbStudio. The advantage of creating the index with HTTP API is the ability to include it in a simple Batch or script command, without the need to pass from the UI of RavenDbStudio.

Now you can issue the PATCH command to operate on all entities that satisfy the index.

   1: curl -X PATCH http://localhost:8080/bulk_docs/PlayersToBeUpdated 

   2:     -d "[{Type: 'Rename', Name: 'Description', Value: 'Background'}]"

The HTTP API command to PATCH documents based on an index is really simple, just issue a PATCH call specifying the index to use and the payload should contain the details of PATCH operation, as described in the documentation. If the index was created only to issue the PATCH command, you can now safely delete it with this call

   1: curl -X DELETE http://localhost:8080/indexes/PlayersToBeUpdated

Thanks to three command line HTTP API calls you are able to rename a property for all documents of a certain type in a RavenDb from a simple batch script, that you can include for example in your software setup procedure to update all documents in case of a Property Rename.

Gian Maria.

First touch of RavenDb HTTP API

Previous posts on the NoSql and Raven Series

Other posts by Mauro on RavenDb Subject.

RavenDb is not accessible only from .NET Code, but it fully support an API based on HTTP that basically permits to interact with database engine with simple HTTP requests, thus you can access a RavenDb instance from every technology that is capable of issuing Web Requests.

To experiment with HTTP API you can download Curl, a command line utility to create HTTP requests, just download it and unzip in a folder accessible from command line, or modify PATH environment variable to include the folder where you unzipped Curl, and you are done.

Once you configured curl the easiest command you can issue with HTTP API to RavenDb is a simple GET to retrieve a document from its id.

c:\> curl Http://localhost:8080/docs/players/1


Figure 1: Simple GET request to retrieve a document by Id

You can also store documents with a PUT request, to accomplish this scenario you need to issue a slightly different request because this time you need to specify document content in JSON format in the body of the request.

   1: curl -X PUT http://localhost:8080/docs/players/2 -d "{Name:'Mauro', RegistrationDate:'2012-04-06', Age:36}"

As you can see it is really similar to previous request, except that I specified the –X PUT argument to ask Curl to issue a PUT request and not a standard GET, and I specified the address of the object, as shown in Figure 2, this time I want to insert a players object with Id 2, finally –d option permits to specify the Body of the request and for PUT operation you can simply specify a document in JSON Format.

The answer of the server should be something like {“Key”:”Players/2”, “ETag”:…., but if the answer is completely blank and there is no new object in RavenDb studio, probably the command encountered some error. To better understand what happened under the hood you need to specify –v switch to curl command to set maximum verbosity and have a full dump of both request and responsxe. If you issue the command again you should see something like this


   2: * About to connect() to localhost port 8080 (#0) 

   3: * Trying 

   4: * connected 

   5: * Connected to localhost ( port 8080 (#0) 

   6: >;; PUT /indexes/byEmail HTTP/1.1 

   7: >;; User-Agent: curl/7.24.0 (i386-pc-win32) libcurl/7.24.0 OpenSSL/0.9.8t zlib/1.2 

   8: .5 

   9: >;; Host: localhost:8080 

  10: >;; Accept: */* 

  11: >;; Content-Length: 78 

  12: >;; Content-Type: application/x-www-form-urlencoded

  13: >;; 

  14: * upload completely sent off: 78 out of 78 bytes 

  15: <;; HTTP/1.1 401 Unauthorized 

  16: <;; Content-Length: 0 

  17: <;; Server: Microsoft-HTTPAPI/2.0 

  18: <;; WWW-Authenticate: Negotiate 

  19: <;; WWW-Authenticate: NTLM 

  20: <;; Date: Mon, 06 Feb 2012 08:26:53 GMT 

  21: <;; 

  22: * Connection #0 to host localhost left intact 

  23: * Closing connection #0

Now you got the full request/response from curl, in this example in line 15 I see that the answer was a 401 HTTP Code, basically the user is not authorized to modify a document with HTTP API. THs usually happens because the standard configuration of raven server command line, found in the standard Raven.Server.Exe.Config file, allows only GET operations from HTTP API. If you just want to experiment and does not care about security for now, you can easily change the appSettings key Raven/AnonymousAccess to the value All, as shown in the following snippet.

   1: <appSettings>

   2:   <add key="Raven/Port" value="*"/>

   3:   <add key="Raven/DataDir" value="~\Data"/>

   4:   <add key="Raven/AnonymousAccess" value="All"/>

   5: </appSettings>

If the PUT operation was successful you can look at the RavenDb studio to verify if the document was really inserted. Everything should be ok, but the new document appears different from the old documents inserted by C# code, as shown in Figure 2. As you can see the RavenDb studio does not report the name of the class ( it is called Doc) and the color is different.


Figure 2: Document inserted with HTTP API is different from the one inserted with C# code.

This happens because the document you inserted with the previous call has no metadata, if you want to insert a document that can be loaded from C# and represents a valid Player entity you should specify metadata during PUT operation, using HTTP Headers.

curl -X PUT http://localhost:8080/docs/players/2 

  -d "{Name:'Mauro', RegistrationDate:'2012-04-06T09:20:25.6657067', Age:36}" 

  -H "Raven-Entity-Name:Players" 

  -H "Raven-Clr-Type:RavenDbAbc.Entities.Player, RavenDbAbc"

The request was divided in four lines to have a better format for the blog, but the above snippet is a single command line request. The only difference is the presence of two HTTP header (issued with curl switch –H) called “Raven-Entity-Name” and “Raven-Clr-Type” used to specify metadata of the document. After you issued the PUT HTTP Request, you can try to load object from C# code and everything should go smoothly.

Clearly this is not the preferred way to insert documents corresponding to C# objects inside RavenDb, but it is a good sample that shows how to accomplish basic operations against an instance of RavenDb using only HTTP Requests.

Gian Maria.

Rename a property in RavenDb

Previous posts on the NoSql and Raven Series

Other posts by Mauro on RavenDb Subject.

In previous articles I showed how simple is to store objects inside a NO SQL database because the data storage has no schema and does not require you to specify the format of your data. I decided to use RavenDb as a NoSql storage to show some basic concepts of NoSql and I showed also how simple is to add a new property to a document, because RavenDb takes care of everything, just save and load objects and the new property is just there.

Now I want to deal with a different kind of problem: what happen when you rename a property of a document that has already some instances saved in database?

This is the typical situation where having No Schema does not solves the problem automatically. Suppose you change the Player document renaming Description property to Background, if you load an old document from the database, since the Description property does not exists anymore, it is ignored and when the object is saved again, it will be deleted. Clearly this is unacceptable because it lead to data loss, so there is some need of manual intervention to handle the scenario of property rename.

There are multiple approaches to solve this problem, but the simplest one is verifying if your NoSql storage supports the concepts of Bulk-Updating documents, a feature that is really well supported by RavenDb.

RavenDb natively support the concept of  Document Patch, a technique used to update a single document without the need to load the entire object or replacing its entire content. Basically a Patch is a dedicated command that transform a stored document directly in the store; along the many different type of patches supported by RavenDb you can use the rename patch specifically designed to change the name of a property. The code is really simple, just reference the assembly Raven.Abstractions and write this code.

   1: store.DatabaseCommands.Patch("Players/1",

   2:       new[] { new PatchRequest() {

   3:         Type = PatchCommandType.Rename,

   4:         Name = "Description",

   5:         Value = new RavenJValue("Background"),

   6:     }

   7: ;

The syntax is really simple, the IDocumentStore has a property called DatabaseCommands used to access all commands supported by RavenDb engine; with this property you can call the Patch() method specifying the Id of the object you want to patch and an array of PatchRequest (you can ask to apply a sequence of patches, in this example I only want to rename a property). To request a Patch operation you need to specify the type of Patch you want to execute, in this example PatchCommandType.Rename, then you need to fill the appropriate properties of PatchRequest object, required by the type of the path you are issuing. To rename a property you need to specify the Name of the property in the Name field and the new name of the property in the Value parameter, passed as RavenJValue element.

This technique is useful but it does not solve our original problem, because it patches a single entity, while our need is to rename the property Description of all saved documents of type Player, so we need to use a different type of operation called: set-based-operation. A set-based-operation is an operation that operates on multiple documents at once, in its most basic form it request an index to identify the list of documents to modify and a list of Patch operation. The whole concept of indexes is a really fundamental concept in RavenDb and in all NoSql database, but for this specific need just think to index as a way to create a query that identify documents in database. To define an index by code you must create a specific class for the index, but I can easily create an index with RavenStudio. In Figure 1 I created an index to identify all the documents of type Players.


Figure 1: Define an index in RavenStudio

To define a simple index you just need to give it a name and write the text of the index. The index starts with from doc in docs, this part is used to query all documents of the database, then I specify a restriction with the keyword where followed by the condition.

In this example the condition is doc[”@metadata”][“Raven-Entity-Name”] == “Players”, and uses RavenDb Metadata, a series of internal properties that RavenDb attach to each saved document. One of this metadata is called Raven-Entity-Name and for .NET object is the name of the class followed by an s to pluralize the name. In my example, since the .NET class saved is called “Player” the Raven-Entity-Name is equal to “Players”. The last part of the Index Map is the select clause, where I simply select property Name of the document, but it is not important for our example, because I’m only interested in creating an index to identify all documents that corresponds to Player entity.

You can execute the index from RavenDb Studio just to verify that it works as intended and it selects all the entities of desired type. When the index is ok you are able to issue a Set-Based-Patch operation:

   1: store.DatabaseCommands.UpdateByIndex("PlayersToBeUpdated",

   2:       new IndexQuery(),

   3:       new [] {new PatchRequest() {

   4:           Type = PatchCommandType.Rename,

   5:           Name = "Description",

   6:           Value = new RavenJValue("Background"),

   7:       }

   8: });

As you can see the syntax is really similar to the previous example, the main difference is that you should not use the Patch()  database command but UpdateByIndex() that requires:

  • the name of the index to use
  • an index query (to specify index parameters, if any)
  • the list of Patches request.

The above command basically means:

Apply the list of Patches to all object identified by the index named “PlayersToBeUpdated”

This simple call updates all Player objects saved in database, renaming the property, as you can verify from RavenDb Studio.


Figure 2: Description property was renamed to Background

As you can see, even if NoSql databases does not require schema, if you refactor your documents changing the name of existing properties, you need to update all saved data to reflect your changes.

In the next series of posts I’ll deal with RavenDb HTTP API to show how to rename a property directly with HTTP requests, without the need to use C# code. This is useful if you want easily to generate a bat script that updates documents using only HTTP Requests.

Gian Maria

NoSql and a life without Schema continued

In the first part I showed how simple is to store object inside a NoSql database like RavenDb, today I want to point out how cool is having no schema when it is time to add properties to your documents.

Suppose that your Player entity changed and you add a new property called Description and you already saved some Players on RavenDb. The question is: what happens when you try to save a new Player now that the class has another property? What happens when you load an old entity, that was saved when that property did not exist?

Since we have no schema we have absolutely no problem, just save another object to Raven and you can find that it got saved without even a warning.


Figure 1: New object is saved without any problem even if it is different from the old one

If you double click a Document (Raven call each object saved to the db a Document) you can simply look at its content and the new property called Description got saved without any problem.


Figure 2: The object contains the new Description property

Probably you had already recognized JSON format to store the object, but the question is, what happened to the old object saved before you added the Description property ? The answer is: it is still there, clearly without the Description property.


Figure 3: The old object does not contain the Description property, but it can coexist with the new one without any problem

Now if you load all object from the database, you can find that for this instance Description property is simply null, because it did not exists in the database, but no error or warning occurred. If you simply load all Player entities and finally issue a SaveChanges, as shown in the following snippet:

   1: using (var session = store.OpenSession())

   2: {

   3:     foreach (var player in session.Query<Player>().ToList())

   4:     {

   5:         Console.WriteLine(player.Name + " " + player.RegistrationDate + " " + player.Description);

   6:     }

   7:     session.SaveChanges();

   8: }

you will end with an automatic update of object and now if you look in the database you can see in figure xxx now the player entity has a Description property equal to null


Figure 4: If you ask to SaveChanges, the old player object with id:1 is updated in Raven, now the Description property is present.

The same happens for removed properties, they simply are ignored during the load process, and removed during the save.

This really trivial example shows how simple is to deal to add or remove a property from documents when your data storage is a NO SQL one and you have no need to define a schema.

Gian Maria.

NoSql and a life without Schema

NoSql is not a replacemente for SQL databases, but it is a valid alternative for a lot of situations where standard SQL is not the best approach to store your data. Since we were taught that whenever you need to store data on a “data store” and you need to query that data for retrieval, SQL is the best solution, you have only to decide what Sql Engine to use and the game is done.

In 2012 this sentence has proven wrong, what I mean is that is not possible to assume anymore that SQL is the “only way to go” to store data; but you should be aware that other alternative exists and it is called NO SQL. Under this term we have various storage engine that are not based on SQL and in .NET we have an exceptional product called RavenDB (you can find a really good introduction to RavenDb in Mauro’s Blog).

The first big difference with standard Sql is being Schemaless. One of the most annoying restriction of Sql Server is the need to specify exactly the format of the data you want to store inside your storage. This is needed for a lot of good reason, but there are situation when you really does not care about it, especially if your software is heavily based on OOP concepts. Suppose you have this object

   1: class Player

   2: {

   3:     public String Name { get; set; }


   5:     public DateTime RegistrationDate { get; set; }


   7:     public Int32 Age { get; set; }

   8: }

For a moment do not care about the fact that this object is not well encapsulated (it has public getter and setter) but focus only on the need to “store” this object somewhere. If you use a standard Sql storage, first of all you need to create a table, then define columns, decide maximum length for the Name column and finally decide an ORM to use or build a dedicate data layer and finally you can save the object.

If you work with raven, this is the only code you need

   1: var store = new DocumentStore { Url = "http://localhost:8080" }; 

   2: store.Initialize();

   3: using (var session = store.OpenSession())

   4: {

   5:     var player = new Player

   6:     {

   7:         Age = 30,

   8:         RegistrationDate = DateTime.Now,

   9:         Name = "Alkampfer",

  10:     };

  11:     session.Store(player);

  12:     session.SaveChanges();

  13: }

I simply created a DocumentStore based on a local server, opened a session and saved an object, I did not defined anything on the server, I did not need to have an ORM, the server simply takes the object and save it, period!

I liked very much this approach because

I needed to save an object to a data storage and everything I need is just a two function all, Store to tell the storage the object I want to save and SaveChanges that actually do the save.

What I got with this simple snippet of code? Just browse with a standard browser to the address of the server and you should see the content of the database.


Figure 1: Content of the database after insertion of a simple object

From Figure 1 you can see content of the raven database, it contains a player and the little 1 beside the object is the Id that Raven uses internally to uniquely identify that object. The other object called Sys Doc Hilo/players takes care of id generation for Players object with an Hilo algorithm.

That’s all folks, no need to define schema, no need to have special Id property or any other requirement to make the object compatible with the store, just call Store method on whatever .NET object and your object is inside the database, Period!.

This is only a scratch of the many functionalities of RavenDb :), more to come in my blog and in Mauro’s one.

Gian Maria.