Store a function to delete all db in MongoDb

Deleting everything in a test MongoDb is a common operation for test and dev machine and it is a relative simple operation that I described here. After a little while I got really tired every time to search my gist or into my hd the little script that deletes everything, thus I decided to store it inside the admin db.

The solution is really really simple, just connect to the admin database and register a server side function to delete all databases.

db.system.js.save(
   {
     _id: "DeleteAllDb",
     value : function(db) { 
         var dbs = db.getMongo().getDBNames()
        for(var i in dbs){
            db = db.getMongo().getDB( dbs[i] );
            if (db.getName() !== 'admin' && db.getName() !== 'local') 
            {
                print( "dropping db " + db.getName() );  
                db.dropDatabase();
            }
        }
      }
   }
)

Once the function is saved, you should see it from a GUI tool like Robo 3T.

DeleteAllDb function stored inside the admin database.

Figure 1: DeleteAllDb function stored inside the admin database.

Now you can simply load all functions from the Shell and execute the new DeleteAllDbFunction.

db.loadServerScripts();
DeleteAllDb(db);

And now you can avoid looking around from the script, just invoke DeleteAllDb(db) function from the shell and you will delete all db, except Admin and Local.

Gian Maria.

Delete all mongo db except admin and local

When you develop with mongo, sometimes you simply need to delete all database quickly because you need to start from scratch. Using shell is really simple, because this gist  does exactly what you need.

var dbs = db.getMongo().getDBNames()
for(var i in dbs){
    db = db.getMongo().getDB( dbs[i] );
    if (db.getName() !== 'admin' && db.getName() !== 'local') 
    {
        print( "dropping db " + db.getName() );  
        db.dropDatabase();
    }
}

This snippet is a little bit smarter than deleting everything, because it keeps safe Local and Admin database, and this will preserve users of the database. I always suggest people to develop with authentication enabled, it is a good way to avoid forgetting to secure your production mongo installation and exposing your data at risk.

If you simply delete ALL databases or delete data directory, you will lose users of the database, as well as any information that is stored in admin or local db.

The script is super simple, you can use the getDBNames() function to grab a list of all database names, then you can iterate for each database doing what you want. In this example I simply check the name of the db, and if the db name is different from local or admin, I simply drop it.

The very same script can be used to delete all collection named logs from all databases, or you can simply decide to delete all database with a name that starts with a specific string.

var dbs = db.getMongo().getDBNames()
for(var i in dbs){
    db = db.getMongo().getDB( dbs[i] );
    if (db.getName().startsWith('blah')) 
    {
        print( "dropping db " + db.getName() );  
        db.dropDatabase();
    }
}

The script is almost the same.

Gian Maria.

Bulk insert in MongoDb with C# driver

There are situation where you need to save a lot of documents inside a collection in MongoDb. My scenario is a migration of documents from a collection to another database, with in-memory manipulation of the documents.

The most common error in these situation is to read the documents from the original collection, then execute a function that modify the document in-memory, and finally issuing an insert in destination collection. This is wrong because you have a roundrip against MongoDb for each document you are saving.

Whenever you are calling Insert or Save function, you are paying the penality of a call to MongoDb process, network latency, etc, whenever possible you should reduce the number of calls to database engine.

In such a scenario MongoDb driver has a function called InsertBatch that allows you to insert document in batches and the fun part is that it simply accepts an IEnumerable. As an example, I have a function that manipulate a BsonDocument stored in a variable called Action, I have source and dest database where I need to copy documents with manipulation and this is the code that does everything.

 var sourceQueue = source.GetCollection(queue);
var destQueue = dest.GetCollection(queue);

if (sourceQueue.Count() == 0) return;

//migrate counterCollection
Console.WriteLine("Migrating Queue " + queue);
var allElement = sourceQueue
	.FindAll()
	.AsEnumerable()
	.Select(document => {
		Action(document);
		return document;
	});
destQueue.InsertBatch(allElement);

The name of the collection is contained in queue variable (actually I’m transforming a software that manage jobs), and as you can verify I can simply enumerate all source documents with FindAll (this code uses old 1.10 driver), for each object I’m calling the Action function that manipulate the document, and finally I can simply use the InsertBatch to insert documents in batches.

This function runs really faster than saving each document with a separate call, event if the MongoDb instance runs on the very same machine, so you do not pay the network latency.

If you use latest version of the drivers, you have the InsertMany method that offers even more options and basically does the very same operation than InsertBatch.

Gian Maria.

Long numbers are truncated in MongoDb shell

Let’s try this simple code in a mongo shell:

db.TestCollection.insert({"_id" : 1, "Value" : NumberLong(636002954392732556) })
db.TestCollection.find()

What you expect is that mongo inserted one record and then that record is returned. Actually a record is inserted, but the return value can surprise you. Here is the output I got from RoboMongo

{
    "_id" : 1.0,
    "Value" : NumberLong(636002954392732544)
}

Property “Value” has not the number you inserted, the number seems to be rounded and some precision is lost, even if it is a NumberLong and 636002954392732556 is a perfectly valid Int64 number. This behavior surprised me, because I’m expecting rounding to happen only with double, not with an Int64.

Actually a double precision floating point number, that uses 64 bit for representation, is not capable of having the same precision of an Int64 number, because part of those 64 bits are used to store exponent. If you try to represent a big number like 636002954392732556 in Double Floating Point precision some rounding is going to happen. If you are not convinced, try this online converter, to convert 636002954392732556, here is the result.

In this image there is a screenshot of the online converter, that exactly demonstrate that the rounding happens due to conversion to floating point number

Figure 1: Floating point number rounding

This confirm that my problem was indeed caused by rounding because the number is somewhat converted to Floating Point format, even if I used NumberLong bson extension to specify that I want a long and not a Floating Point type.

The reason behind this is subtle. Lets try another example, just type NumberLong(636002954392732556) in a mongo shell (I used RoboMongo), and verify the result.

calling NumberLong(636002954392732556) function returns a rounded number,

Figure 2: NumberLong gots rounded directly from the shell.

This unveils the error, the number is returned surrounded with quotes, and this suggests that quotes are the problem. In javascript, every number is a double, and if you write NumberLong(636002954392732556) javascript translate this to a call to NumberLong function passing the number 636002954392732556 as argument. Since every number in javascript is a double, the number 636002954392732556 gots rounded before it is passed to NumberLong Function.

If you surround number with quotes, you are passing a string to NumberLong, in this scenario rounding does not occours and NumberLong function is perfectly capable to conver the string to number.

In mongo shell, always use quotes when you create numbers with  NumberLong

Actually this error only happens with really big numbers, but you need to be aware of this if you are creating script that uses NumberLong.

Gian Maria.

Grant right to use $eval on Mongodb 3.2

One of the side effect of enabling authorization on MongDb is that, even if you create a user with “root” right, this account is not able to execute the $eval command. The simpthom is, when you try to execute $eval you got this error

mongodb Command '$eval' failed: not authorized on jarvis-framework-saga-test to execute command

This happens because $eval is somewhat deprecated, and it should not be used. Since it is a dangerous command, a user should have access to all action on all resources, and you need to create a role that has anyAction on anyResource.

If you really need to use $eval, you should create a role, just connect to the admin database and create a new role with the command.

db.createRole( 
	{ 
		role: "executeEval", 
		privileges: [ { 
			resource: { anyResource: true }, 
			actions: [ "anyAction" ] } ], 
		roles: []
 } ) 

Now that you have this new role, just add to all the users that need to use $eval, as an example, if you have a single admin user in admin database, just run this against the admin db.

db.grantRolesToUser("admin", [ { role: "executeFunctions", db: "admin" } ])

And now the admin user can execute $eval against all databases.

Gian Maria.