Mixing native query and LINQ in Mongo Query

Lets look at the following query issued to a standard MongoCollection<T> instance object:

return _bufferCollection.Find(
        GetNextBlockQuery(lastTick, lastRevisionId))
    .OrderBy(d => d.LastUpdated)
    .ThenBy(d => d.RevisionIdNumeric);

The method GetNextBlockQuery simply return a Query<T> query object expressed with C# mongo query syntax. In this query the result of Find() method is simply sorted using standard LINQ syntax.

Do you spot where the problem is?

Find() method returns an object of type MongoCursor<T> that implements IEnumerable<T> but not IQueryable<T>.

If you query MongoCollection with LINQ using the AsQueryable() extension method, there is no problem using OrderBy() or ThenBy() LINQ extension methods. In this situation the implementation of IQueryable inside Mongo C# driver will translate everything to standard mongo query syntax, then it executes translated query to the server and returns objects to the caller.

In previous example instead, the OrderBy() LINQ operator is invoked against a MongoCursor and ordering will be done in memory. The problem is: OrderBy method will operate against IEnumerable object and iterates all the objects to return them in correct order.

If you use LINQ operators against standard MongoCursor, it will operates in memory, hurting performances.

This will hurt performances of the application: each time the query is executed, the entire resultset is loaded into memory and then sorted. To avoid this problem, you need not to mix native Mongo C# query with LINQ operators. The correct query is the following one:

 return _bufferCollection.Find(
      GetNextBlockQuery(lastTick, lastRevisionId))
           .Ascending(d => d.LastUpdated, d => d.RevisionIdNumeric));

This new version uses SetSortOrder() method of Mongo C# Query, so it will be sorted directly from Mongo server and objects will be loaded in memory during standard for-each enumeration. The above problem is really bad if you want to limit number of returned objects. If you use a Take(50) method to obtain only 50 objects, actually you are loading the entire collection into memory, then returning the first 50 elements. This is really different from asking mongo to return only 50 elements directly in the query.

One of the greatest problem is that if you limit number of record with LINQ operator Take()  on the first query, yoy are doing Client Side pagination, with significant performance loss.

As general rule, avoid mixing LINQ and Mongo query classes to issue query to your Mongo server, and prefer native Query syntax over LINQ because it will offer you the whole capabilities of Mongo. LINQ query at the contrary will expose only a subset of possible queries, and beware that Select operator operates in memory, instead of limiting the number of returned field directly from the server.

Gian Maria.

Upgrading ReplicaSet with MMS

My RS of three mongo instances is running mongo 2.6.6 and now I want to upgrade to the latest version. Thanks to mms I can simply start the upgrade process directly from a web page, without needing to have an access to my real servers. The real good stuff is that the upgrade process is completely managed by mms for me, and the upgrade is done without stopping my Replica Set.

Since this is a test/dev environment running in my home server, my bandwidth is not so high and it is not unfrequent that one of the node finished downloading latest mongo version before the others. The nice stuff is that mms takes care of everything, and starts upgrade my secondary instances.


Figure 1: One of the server is upgrading, while the others are still running old version.

The super-nice feature is that one of the server upgraded to the new version, the others still are running the old version, and the Replica-Set version is mixed because I have node with different versions. I can connect to it with robomongo and workd as usual.  At a certain moment the primary node is upgraded, now we are in a state where we have only secondary nodes, in this moment users cannot write to Replica Set:


Figure 2: Status is 2, indicating that that a member in secondary is replicating, now all node are SECONDARY

As soon as the primary node starts the new mongo process with the new version, replica set is fully operative again. The downtime is really small and you upgraded everything with few clicks, letting mms agents take care of everything.

Gian Maria.

Upgrade your agents and instances with Mms

Another big advantages of using MMS to manage your mongo deployments is the ability to automatic upgrade agents and instances. For agents you can simply open the deployment area of a group and select edit mode. If some agent is outdated, the interface will immediately warn you with an alert on the top of the page as you can see in Figure 1.


Figure 1: Mms is telling you that agents are out of date

If you choose to upgrade agents pressing the update link at the top of the pages, you should see some unpublished changes appears in the Ui.


Figure 2: Upgrading agents is a simple modification of the deploy 

Remember that each modification you do through mms will be downloaded by all the agents and should be reviewed and confirmed for deploy. Pressing Review & Deploy will show pending operations:


Figure 3: Pending modification to deploy

When you press confirm and deploy, mms will confirm all modifications and the next time that the agents will poll the site, the new deploy instructions will be downloaded and applied from the various automation agents.

Upgrading automation and monitoring agents is not the only upgrade you can do from the web interface, the real killer feature is  upgrading the version of mongo that your instances are running. In Figure 4 you can see that mongo mms shows my current version as red, because is not the latest one available.


Figure 4: Mms interfaces shows mongo version in red because is not the latest one available.

If you come back to edit mode you can edit any monitored instance, in this example I want to edit configuration for the entire replica set, upgrading mongo to the latest version.


Figure 5: In edit mode you can change configuration of your mongo deployment

Now you can simply edit the configuration and choose whatever version you like to install. You can install newer version, but you can also install older version if you want to rollback your environment to previous version.


Figure 6: Editing of configuration of your replica set.


Figure 7: From edit panel you can change version to whatever version you like.

Once you have done your changes and pressed Apply button, you should review your modification and confirm as for agent upgrade.


Figure 8: Changes of your replica set ready to be deployed, version will be upgraded from 2.6.2 to 2.6.5

As usual mms interface starts showing what is happening to your machines, in Figure 9 you can verify that the three machines of the replica set are downloading the new version of mongo.


Figure 9: Environment is upgrading to new version


Figure 10: Not all the machine have same speed, rs1-red is already in goal state, while the other are still installing new version


Figure 11: Replica set was upgraded to latest version, primary has changed

After a little bit, all of your machines are upgraded to the latest version. Thanks to Mongo MMS you can manage upgrade or downgrade of your instances with few mouse clicks, because automation agents will take care of all operations needed to upgrade your servers.

Gian Maria.

Manage authentication with Mongo MMS

In an old post I’ve shown how to easily deploy mongo thanks to MMS online services. Mms is not only useful to deploy mongo instances, but it is exceptional also for monitoring and configuring. One of the most interesting features is managing users easily from mms web interface. Suppose you had installed mongo on an Azure Virtual Machine and you do not want everyone being able to access your instance. A possible solution is enabling authentication.

Thanks to mms you can simply go to Deployment / Authentication & Users and enable authentication. With the green Add User button you can add how many user you want and you can give them permission using roles.


Figure 1: Manage mongodb users through mms web interface.

Once everything is configured, mms will deploy changes to all of your instances that belong to the same group, so every database shares the same sets of users. Wait a little bit and then try to authenticate to your instance, ex with robomongo.


Figure 2: Test auth from Robomongo

Et voilà, with few clicks you have now authentication enabled on your mongo databases of that group, it could not be easier than this :).

Gian Maria.

Deploy mongo easily with mms

One of the reason why Mongo is gaining a lot of momentum in the industry is the easy of use. Just download the package, start mongod and you are donw. When you are in production, you cannot surely simply install your mongo as a service, start it and then forget it without any kind of monitoring, because this will surely lead to problems. When is time to create more complex topologies, like sharding or replica set you can commit some mistakes and you have some tedious work to do. I strongly suggest you to have a look at Mongo Mms, because the online version offers a lot of functionalities for free, and every developer can use it to simplify even deployment of dev servers.

The new version introduced some interesting capabilities to automatically deploy and manage your mongo instances with few clicks of your mouse. Suppose you want to create a classic replica set of 3 machines using 3 VM with Linux Ubuntu. After you register with MMS you can simply proceed to install the MMS automation agent on your machine with few steps


Figure 1: Instruction to install automation agent on Ubuntu

Once you installed the automation agent on all of your machines, you can simply start creating your mongo deployment using MMS web interface. Agents poll server periodically for instructions, this implies that mms only requires your machines to be able to reach mms site to poll instructions and to send monitoring data. All of your machines can be in a private NAT network. If you go in the Administration section of MMS web interface you can check status of your agents.


Figure 2: List of all of your active agents

In this picture you can check not only automation agents, but also monitoring and backup ones. The automation one is the most important, because is the responsible for installing other agents and configuring your mongo instance. Now you can move to the deployment tab, and if some of your agents are out of date you can simply upgrade with a simple click.


Figure 3: List of all of your deployment servers and in the top area of the page you can check if some of the agents are old

In the upper right part of the deployment area, there is a button called Add that can be used to create a new deployment of Mongo, in this example I want to create a new Replica Set.


Figure 4: Create a replica set from the web interface.

Now it is just a matter of configuring all the options. The most important one is the Elegible Server RegExp where you should specify a regex that will select all the servesr that will take part of the replica set. In this example I used three distinct names, and I separated with a pipe to create a regex that choose all the servers, you can use prefix for convenience.


Figure 5: Options to create a replica set.

You can configure a lot of options and once you are done you can simply press Apply. This will create a script to create your new deployment, and you can review it before agents starts applying it to destination machines.


Figure 6: Pressing apply creates the script but you still have time to review before starting deploy


Figure 7: Reviewing the script tells you every operation that will be triggered in your environment

Pressing Confirm & Deploy activates the script, and the web interface starts showing you the status of the various machines that will take part of the deploy.


Figure 8: Deploy starts and the web interface starts waiting for it to finish.

 The status is pushed from the automation client to the mms server and you can see the status of the single machines.


Figure 9: One of the machine is in goal state, the other two still needs time

In Figure 9 you can see that the machine avalanche reached Goal State, this means that the automation agent finished applying modification to the system. The other two machines are in WaitRsInit, this means that mongo is started and they are starting the replica set. Once everything is ok, all the machines are in the Goal State and the version of mongo installed in shown in Version column.

Now you can switch to View mode, where you can look at your deployment.


Figure 10: View mode shows the status of your servers, as well as witch instance is primary and which instance is secondary

You can change every aspect of your deployment after it was provisioned, as an example I decided to change priority of one of my server after the replica set was up and running because avalanche VM is stored on an SSD and writing speed is higher than the other two VM.


Figure 11: Changing options after replica set is created.

You can also change the installed version of mongo, and the automation agents will take care of all operations needed to update or downgrade your instance of mongo.

If you use Amazon AWS, Mongo MMS will be also able to automatically provision your virtual machines in the cloud, automatically installing automation agents on it, so you can really deploy everything, from virtual machines to mongo with few clicks.

I must admit that I’m really impressed from deployment capabilities of Mongo MMS especially because you can manage up to 8 servers for free.

Gian Maria.