Using Mongo Database to store Log4Net logs

One of the most simple and useful way to introduce a Documents Database like Mongo in your organization is to use as Log Storage. If you use log4Net you can download a Mongo appender capable of storing logs inside mongo with few lines of code and you are ready to go.

The original appender is really good but I’ve done a little modification to make it capable of storing complex item in extended properties, just locate the function LogginEventToBSON and modify the section that stores Composite Properties with following code

if (compositeProperties != null && compositeProperties.Count > 0)
{
    var properties = new BsonDocument();
    foreach (DictionaryEntry entry in compositeProperties)
    {
        BsonValue value;
        if (!BsonTypeMapper.TryMapToBsonValue(entry.Value, out value))
        {
            properties[entry.Key.ToString()] = entry.Value.ToBsonDocument();
        }
        else
        {
            properties[entry.Key.ToString()] = value;
        }
    }
    toReturn["customproperties"] = properties;
}

The only modification I did to the original code is trying to convert the value stored in Composite properties to a BsonValue and if the conversion does not succeeds I convert the entire object to a BsonDocument and store the entire object to Mongo. The original code stores extended properties with ToString(), this has two disadvantages, the first is that you are actually losing the type of the data and you are not able to store complex objects. With this modification if you add an Int32 as extended property, it will be stored as numeric value inside the database.

Using Mongo as Log Storage gives you lots of advantages, first of all Document Databases are really fast in inserting stuff, then they are Not Schema Bound so you do not need to define a schema to contain your logs. This feature in conjunction with the ability of Log4Net to store extended properties in the context permits you to write code like this one.

log4net.GlobalContext.Properties["CurrentItem"] = item;

With this simple line of code each subsequent log will have in Mongo Database an associated object of type item, until you will not remove the object from the GlobalContext.

Complex objects are stored inside the log

Figure 1: Your log contains all data from extended properties

As you can see from Figure 1 the log has now a complex object associated with it. This capability is awesome because you simply need to store complex objects inside GlobalContext and they will be stored inside the log as customproperties without needing additional lines of code.

Clearly you can now use all the advanced query capability of Mongo to view/filter/mapreduce logs using these additional properties.

Gian Maria.

Published by

Ricci Gian Maria

.Net programmer, User group and community enthusiast, programmer - aspiring architect - and guitar player :). Visual Studio ALM MVP

8 thoughts on “Using Mongo Database to store Log4Net logs”

  1. In the ELSE statement I should suggest to use the original
    properties[entry.Key.ToString()] = entry.Value.ToString();
    instead of
    properties[entry.Key.ToString()] = value;
    I suspect that you’re actually storing a null value if the entry isn’t mapped to a bsondocument.
    I’m riggh?

  2. Yes you are right, but for simple types as Int32, DateTime, etc this will store the right value in the db, avoiding to store everything as simple string. When I store a document I have stuff like this

    “AddDate” : new Date(“Thu, 29 Mar 2012 08:22:01 GMT +02:00”),
    “UpdateCount” : 2,
    “EditedByAdmin” : false,

    Where each property is stored as the right value. If you use entry.Value.ToString() everything is stored as string, and for stuff like dates it is annoying.

    Alk.

  3. I was considering on using mongodb+log4net but, I found some issues following this approach in an application using log4net

    1)
    – If entry.Key.ToString() is something like “prop.abcd” it causes an error because strings with ‘.’ cannot be used in Bson key names (this is an important limitation for me)

    2)
    – if loggingEvent.LocationInformation.FileName is null, then this code will fail (not here but in the appender):
    if (loggingEvent.LocationInformation != null)
    {
    toReturn[“fileName”] = loggingEvent.LocationInformation.FileName;
    (…)
    collection.Insert(doc); -> fails here

    Any suggestions?

  4. For the issue 1, if you really need to have dot in your keys, you can simple replace with other char, like _ or |, after all this is a log, you can always consider that when you will read the logs.

    For point 2, just do an extra check, if loggingevent.locationinformation.FileName is null just avoid saving that info in the BSon Document, it should work.

    Actually I did not incurr in these error, even if it a short time for me using mongo as log4net appender, and I did find other issues that I’ll blog shortly.

  5. That was my first approach to both situations, but that is something I don’t like, because we are changing the original contents and has extra parsing to do while reading.

    The null validation test is a good workaround, but I think that maybe this could be done a better way, because it will require to check also the other values, creating a big list of “if …== null”

    What I’m thinking is to create a Bson mapping class that actually ignores the null, and that would serialize into the Bson document and maps the properties somehow different

    public class BsonLocationInformation {
    (…)
    [BsonIgnoreIfNull]
    public string FileName{ get; set; }
    (…)
    }
    public class BsonLoggingEvent {
    (…)
    public BsonLocationInformation locationInformation { get; set; }
    (…)
    //something else to deal with composite properties in here…
    }

    but I’m lazy and it sounds like a lot of work… :)
    anyway, I will probably try this whenever I have some time to look into it again

    thank you!

  6. Usually you can use the coalesce operator, something like

    toReturn[“fileName”] = loggingEvent.LocationInformation.FileName ?? String.Empty;
    toReturn[“method”] = loggingEvent.LocationInformation.MethodName ?? String.Empty;
    toReturn[“lineNumber”] = loggingEvent.LocationInformation.LineNumber ?? String.Empty;
    toReturn[“className”] = loggingEvent.LocationInformation.ClassName ?? String.Empty;

    It can solve the null problem, for the point problem, I do not know if it is possible to insert a plugin or something in the driver that care trasparently for you dots in property names.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.