Store a function to delete all db in MongoDb

Deleting everything in a test MongoDb is a common operation for test and dev machine and it is a relative simple operation that I described here. After a little while I got really tired every time to search my gist or into my hd the little script that deletes everything, thus I decided to store it inside the admin db.

The solution is really really simple, just connect to the admin database and register a server side function to delete all databases.

db.system.js.save(
   {
     _id: "DeleteAllDb",
     value : function(db) { 
         var dbs = db.getMongo().getDBNames()
        for(var i in dbs){
            db = db.getMongo().getDB( dbs[i] );
            if (db.getName() !== 'admin' && db.getName() !== 'local') 
            {
                print( "dropping db " + db.getName() );  
                db.dropDatabase();
            }
        }
      }
   }
)

Once the function is saved, you should see it from a GUI tool like Robo 3T.

DeleteAllDb function stored inside the admin database.

Figure 1: DeleteAllDb function stored inside the admin database.

Now you can simply load all functions from the Shell and execute the new DeleteAllDbFunction.

db.loadServerScripts();
DeleteAllDb(db);

And now you can avoid looking around from the script, just invoke DeleteAllDb(db) function from the shell and you will delete all db, except Admin and Local.

Gian Maria.

Delete all mongo db except admin and local

When you develop with mongo, sometimes you simply need to delete all database quickly because you need to start from scratch. Using shell is really simple, because this gist  does exactly what you need.

var dbs = db.getMongo().getDBNames()
for(var i in dbs){
    db = db.getMongo().getDB( dbs[i] );
    if (db.getName() !== 'admin' && db.getName() !== 'local') 
    {
        print( "dropping db " + db.getName() );  
        db.dropDatabase();
    }
}

This snippet is a little bit smarter than deleting everything, because it keeps safe Local and Admin database, and this will preserve users of the database. I always suggest people to develop with authentication enabled, it is a good way to avoid forgetting to secure your production mongo installation and exposing your data at risk.

If you simply delete ALL databases or delete data directory, you will lose users of the database, as well as any information that is stored in admin or local db.

The script is super simple, you can use the getDBNames() function to grab a list of all database names, then you can iterate for each database doing what you want. In this example I simply check the name of the db, and if the db name is different from local or admin, I simply drop it.

The very same script can be used to delete all collection named logs from all databases, or you can simply decide to delete all database with a name that starts with a specific string.

var dbs = db.getMongo().getDBNames()
for(var i in dbs){
    db = db.getMongo().getDB( dbs[i] );
    if (db.getName().startsWith('blah')) 
    {
        print( "dropping db " + db.getName() );  
        db.dropDatabase();
    }
}

The script is almost the same.

Gian Maria.

Optimize your local git repository from time to time

Git is an exceptional piece of software and I really cannot think living without it. Now with superfast SSD, you can use git without performance problem even for large repositories (maybe you converted an old Subversion or TFVC).

When your repository has thousands of object and especially if you adopt flows of work where you rebase often, probably your repository has large number of unnecessary object that can be deleted safely. Git runs for you in the background a special command called

git gc

It is a Garbage Collection command, that will cleanup your local repository, nothing changes in the remote repository. Sometimes you can also run manually a deeper cleaning of your repository with the command

 

git gc --aggressive

You can run from time to time (documentation say you can run every few hundreds commits), it will take longer that a normal git gc, but it will help keeping your local git repository fast even if you work with really large codebase.

Gian Maria.

Use conditional task in VSTS / TFS Build

When you start using Continuous Integration extensively, some builds become complex and sometimes a simple sequence of task is not what you want. Previous build system, based on XAML and Workflow Foundation allows you to specify really complex path of execution, but the side effect is that builds become really complex, they are painful to edit and also, you need to use Visual Studio to edit. Final drawback is that writing and maintaining custom build activities was not so easy, because you need to keep the compiled version in line with build engine version.

XAML build engine allows for complex workflow, but it was far more complex to manage.

The new build system is really far superior to old XAML build, but having a simple sequence of tasks can be too simplistic for complex scenario. The solution is really really simple, just use conditional execution.

In the control options section of the task configuration you can specify that the task should be run only if a custom condition is met.

Figure 1: Conditional execution condition in action

In the Control Option section of any task configuration, you have various options to decide when the task should run.

List of options that are described in the text of the post.

Figure 2: Options available to decide when the task is run

First four options are straightforward to understand, you can decide to run the task only if all previous task succeeded, or when a previous task failed, but also you can decide to run the task even if the build was cancelled.

The last option, Custom condition is where the real power shows up, with that option you can specify a custom condition, like the one that is shown in Figure 1. The whole syntax is shown in this MSDN Page and it is quite powerful. In Figure 1 I’ve used a simple expression that runs the task only if a variable called SkipTests is not equal to true (ne operator means Not Equal). Thanks to this, I can decide to set this variable to true at queue time, but I can also change a variable from previous task.

Even if custom condition does not gives you the full power of a workflow, you can still decide that some task runs only on certain conditions and it is simple to simulate flowchart if condition or more complex flow. As an example you can decide to run some tasks if not all tests are green and other tasks if some of the tests are failing.

Have a look at the documentation MSDN Page to verify all the conditions that you can use.

Gian Maria

Running powershell before Get Sources in VSTS / TFS Build

I have a VSTS build where I need to run a PowerShell scripts at the very beginning of the build, before the agent starts downloading the sources, and it seems that this cannot be done with a simple task, because there is no way to place a task before the Get Sources default task of VSTS Build.

Luckily enough there is a way to solve this problem, because each task can define a specific script that should be run when the build starts and before each task runs, here is how. First of all create a new task with the usual TFX-Cli interface:

tfx build tasks create

I’ve already explained how to create a build task in a previous post, but it is time to add some more context, because the build system is slightly changed and you need to do some more steps to be able to run the task. First of all you can incurr in the error

##[error]File not found: ‘C:\vso\_work\_tasks\LibreOfficeKiller_9073fee0-4de5-11e7-925d-cb284842266e\0.1.0\ps_modules\VstsTaskSdk\VstsTaskSdk.psd1’

This error is telling you that VstsTasksSdk is missing from the installation folder. New scripts uses a base library to interact with the build system, you can find all the information here, it is just a matter of cloning the VSTS Task Lib in a folder of your disk, then locate  the PowerShell folder, that contains a folder called VstsTaskSdk, that should be copied in your task folder, under a folder ps_modules.

VSTS / TFS Build system is evolving rapidly, new versions have nice new capabilities, but this requires you to always read latest documentation.

After you copied the VstsTAskSdk.psd1 folder in the right place you should be able to run a build with your task. Now there is the fun part, executing some script before the GetSouce phase of the build is run.

In this scenario I have a PowerShell scripts that kills every instance of LibreOffice that is running on the system. My build runs some integration tests that involves LibreOffice, but if, for some reason, a previous execution failed to kill properly running instances of LibreOffice, source folder in the agent cannot be clean, because soffice.exe process has open handle on source folder, thus, the GetSources task fails. The sympthom of this problem is the build failing with this error in the Get Sources:  The process cannot access the file ‘\\?\C:\vso\_work\8\s\src\Jarvis.DocumentStore.Tests\bin\Debug’ because it is being used by another process.

With latest version of the tasks, I can use the prejobexecution and postjobexecution to ask the system to run scripts at the beginning and the end of the build.

Section of task.json file showing the use of prejobexecution.

Figure 1: Relevant section of the script that uses prejobexecution to execute a script before any task runs.

Now in Figure 1 you can verify that I’m invoking the script KillLibreoffice.ps1 not only in the execution node, but also with a special node called prejobexecution and postjobexecution. These two nodes allows you to specify a script to be called before the very first task of the build is executed. To verify that everything works, here is a super simple and stupid build that uses this task.

Simple buidl that uses a libreofficekiller task then build the solution

Figure 2: Task LibreOfficeKiller used in a sample build

From the output of the build you can verify that the script that kills LibreOffice is run not only before the Get Source phase, but also after all tasks are completed.

Output of the build shows that kill libre office script is run three times, before any task, during task execution and finally after all tasks were run

Figure 3: Build output of the build with KillLibreOffice task.

From Figure 3 you can verify that the LibreOfficeKiller is run three times, the first time (1) before the Get Sources phase, then during normal task execution (2) and finally after the last task executed (3). Now my build can run just fine, because I’m able to kill any instance of LibreOffice.exe before the GetSources tasks runs.

If you are interested in the whole task, here is the code.

Happy building :).

Gian Maria.