Keep your Mongo log database small deleting old logs

As I showed in an old post, Mongo Db is perfect to store logs, but as everyone knows, log databases tend to become really big, especially if the verbosity level is high, so I usually schedule a process that deletes all log older than a certain date to free space in log databases. To cleanup record in a Mongo Db I decided to create a little Powershell script to delete all entry in a collection that are older than a certain value. The whole scritp is reported here.

write-host "Cleanup Mongo Db" 
$now = Get-Date 
$limit = $now.AddDays(-20) 
write-host "Date: " + $limit.Year 
write-host "Date: " + $limit.Month 
write-host "Date: " + $limit.Day

$si = new-object System.Diagnostics.ProcessStartInfo 
$si.FileName = "Mongo.exe" 
$si.WorkingDirectory = "c:\Mongo\Bin" 
$si.Arguments = "localhost:27017/log4Net --eval ""db.mylogtable.remove({timestamp: {`$lt:new Date(" + 
$limit.Year + ", " + ($limit.Month - 1).ToString() + ", " + $limit.Day + ")}})"""

$p = [diagnostics.process]::Start($si) 

The scritp is really simple first of all I get the actual date with the Get-Date powershell function, then I subtract 20 days to create the date limit and grab the year, month and day part of it The cleanup process is a simple call to Mongo.exe command line tool that accepts the address of the database as first argument in the form of address:port/databasename followed by –eval to directly execute a command. The real delete command has this form: db.NameOfTheCollection.Remove(Condition), in this situation the condition is something like {timestamp: {$lt:new Date(year, month, day)}}. Pay attention to the thick sign (`) before the $lt, because it is the escape character used in powershell and it is needed to insert a dollar sign inside a string.

All you need to do is schedule this script to run once a day to keep in your Mongo database only the last 20 days of logs.

Gian Maria

Using Mongo Database to store Log4Net logs

One of the most simple and useful way to introduce a Documents Database like Mongo in your organization is to use as Log Storage. If you use log4Net you can download a Mongo appender capable of storing logs inside mongo with few lines of code and you are ready to go.

The original appender is really good but I’ve done a little modification to make it capable of storing complex item in extended properties, just locate the function LogginEventToBSON and modify the section that stores Composite Properties with following code

if (compositeProperties != null && compositeProperties.Count > 0)
{
    var properties = new BsonDocument();
    foreach (DictionaryEntry entry in compositeProperties)
    {
        BsonValue value;
        if (!BsonTypeMapper.TryMapToBsonValue(entry.Value, out value))
        {
            properties[entry.Key.ToString()] = entry.Value.ToBsonDocument();
        }
        else
        {
            properties[entry.Key.ToString()] = value;
        }
    }
    toReturn["customproperties"] = properties;
}

The only modification I did to the original code is trying to convert the value stored in Composite properties to a BsonValue and if the conversion does not succeeds I convert the entire object to a BsonDocument and store the entire object to Mongo. The original code stores extended properties with ToString(), this has two disadvantages, the first is that you are actually losing the type of the data and you are not able to store complex objects. With this modification if you add an Int32 as extended property, it will be stored as numeric value inside the database.

Using Mongo as Log Storage gives you lots of advantages, first of all Document Databases are really fast in inserting stuff, then they are Not Schema Bound so you do not need to define a schema to contain your logs. This feature in conjunction with the ability of Log4Net to store extended properties in the context permits you to write code like this one.

log4net.GlobalContext.Properties["CurrentItem"] = item;

With this simple line of code each subsequent log will have in Mongo Database an associated object of type item, until you will not remove the object from the GlobalContext.

Complex objects are stored inside the log

Figure 1: Your log contains all data from extended properties

As you can see from Figure 1 the log has now a complex object associated with it. This capability is awesome because you simply need to store complex objects inside GlobalContext and they will be stored inside the log as customproperties without needing additional lines of code.

Clearly you can now use all the advanced query capability of Mongo to view/filter/mapreduce logs using these additional properties.

Gian Maria.