Run Selenium Test in build vNext

Previous Articles:

To run a Selenium test in a build vNext there are some modification to do apply to previous example. Let’s see how simple is running our Selenium Tests in a VSTS Build vNext.

The first modification requires adding a reference to PhantomJS, an Headless browser based on webkit that is capable of browsing a site and run javascript without a UI. Since we are interested in running test in a build server, this is a requirement if the agent runs as a service and has not access to a UI. To use PhantomJS just reference PhantomJS package on nunit and modify your base class to support this browser.

switch (browser.ToLower())
{
    case "chrome":
        DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
        WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
        break;
    case "firefox":
        WebDriver = new FirefoxDriver();
        break;
    case "phantomjs":
        WebDriver = new PhantomJSDriver();
        break;

Using Phantomjs is just a matter of creating a PhantomJSDriver with Selenium WebDriver test and the game is done. Now add “phantomjs” to test config file and you should be able to run the test.

image

Figure 1: Selenium test that uses Phantomjs headless browser to run test.

Now it is time to change the ValueSourceAttribute, to allow overriding the list of browser with an Environment Variable. While the Json Configuration file to configure tests is a really simple and useful solution for developers, running tests on their machines, when I need to run tests on a bulid server I want to be able to specify the list of browser with a build Variable.

In build vNext, each variable you add to the build will be copied in an environment variable with the same name of the variable, converted in uppercase and with dot char substituted with underscore. If I use variable Selenium.BrowsersToTest the environment variable is called: SELENIUM_BROWSERSTOTEST

Here is the new code of the ValueSourceAttribute that use that environment variable to find list of browsers to use.

public class BrowserList : ValueSourceAttribute
{
    private static IEnumerable Browsers;

    public BrowserList()
        : base(typeof(BrowserList), "Browsers")
    {
        Browsers = GetBrowserFromConfig();
    }

    private static IEnumerable GetBrowserFromConfig()
    {
        var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
        if (!String.IsNullOrEmpty(envVar))
        {
            return envVar.Split('|', ',', ';', ':');
        }
        else
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject)JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }
}

The only change is that the attribute searches an envorinment variable called SELENIUM_BROWSERSTOTEST to grab list of browser. If the variable is not present, it use json configuration file as showed in previous article. Now we can choose browser list directly from build definition.

image

Figure 2: Specifying browser list with Build Variables.

Variables can be specified a Queue Time, this allow the user to change browserslist event when queueing a new build. Here is the result of a run

image

Figure 3: Outcome of the test using browsers list from build variable

Previous build outcome of Figure 3 is obtained running the test with an agent that does not run as a service, because it should be able to launch browser and access UI. If I queue the same build with Hosted Agent, or with an Agent that is running as a service, here is the result.

image

Figure 4: Failure running selenium tests

The problem with queued agent is that he has no firefox installed, but even with firefox installed, it would not be able to run the test because hosted agent runs as a service and has no access to UI. To solve this problem we can modify our ValueSourceAttribute

  public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            
        }

        static BrowserList()
        {
            Browsers = GetBrowserFromConfig();
            if (!Environment.UserInteractive)
            {
                Browsers = Browsers
                    .Where(b => b.ToLower() == "phantomjs")
                    .ToArray();
            }
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var envVar = Environment.GetEnvironmentVariable("SELENIUM_BROWSERSTOTEST");
            if (!String.IsNullOrEmpty(envVar))
            {
                return envVar.Split('|', ',', ';', ':');
            }
            else
            {
                var settings = File.ReadAllText("testParameters.json");
                var config = (JObject)JsonConvert.DeserializeObject(settings);
                var seleniumSettings = config["Selenium"];
                var browsers = (JArray)seleniumSettings["BrowsersToTest"];

                return browsers
                    .Select(b => b.ToString())
                    .ToArray();
            }
        }
    }

 

A couple of modification are worth of notice, the first one is that initialization of browser list is done in static constructor, and it will be executed once for each test run.

Then, if the test is not executing in a UserInteractive session, the attribute remove all browsers except phantomjs, the only ones that is guaranteed to run without having access to a UI. With this simple trick we avoid to run tests that will fail because they could not run. To understand which browser can run with agent running as a service you can simply try to run all of them and verify which ones returns error. Actually some browser can run in headless mode (without UI) so they can be used even if the agent has no access to the UI, so use this technique to remove all browsers that does not supports this mode.

To verify that everything works, I changed configuration of my Visual Studio Agent to run as a service instead that running interactively and queued a new build. Here is the result of the Tests.

image

Figure 5: Now only phantomjs test is run because agent is running as a service

Gian Maria.

Parametrize NUnit Selenium Test to run with different browsers

Parametrizing NUnit Tests is a new feature introduced with version 2.5 and this feature can be really useful in a variety of scenarios, but when it is time to use Selenium this is a killer feature.

I’m not going to cover Selenium WebDriver component, but basically it allows to write tests that can drive a Browser to execute test against your web application. In this scenario a killer feature is being able to specify the list of the browsers to use in a way that is completely indipendent from your test.

Thanks to the ValueSourceAttribute obtaining this result is really simple. First of all I create a base class for the test that creates different Selenium Web Drivers based on a string specified as argument.

 public class SeleniumTestFixtureBase
    {
        protected IWebDriver WebDriver;

        public void Initialize(String browser)
        {
            switch (browser.ToLower())
            {
                case "chrome":
                    DirectoryInfo chromeDriverLocation = new DirectoryInfo(@".");
                    WebDriver = new ChromeDriver(chromeDriverLocation.FullName);
                    break;
                case "firefox":
                    WebDriver = new FirefoxDriver();
                    break;
                default:
                    throw new NotSupportedException("Browser " + browser + " not supported");
            }
        }
    }

This is really trivial, and for this example I’m going to support only chrome and firefox. Then I create a json configuration file where I specify every parameter needed by Unit Tests.

Using an external json file to specify parameters for your Unit Test makes trivial passing those parameters for every runner (VS, TeamCity, GUI, command line runner).

This is a simple configuration file that contains only the list of the browsers I want to use. Remember to ask Visual Studio to copy this file in output folder when it changes.

{
  "Selenium": {
    "BrowsersToTest": [ "Firefox", "Chrome", "IE"]
  }
}

I’ve choosen Json because it is easy to write and easy to parse. Now it is time to create my attribute based on ValueSourceAttribute; it will read the above configuration file and provide the list of the browsers I want to use.

public class BrowserList : ValueSourceAttribute
    {
        private static IEnumerable Browsers;

        public BrowserList()
            : base(typeof(BrowserList), "Browsers")
        {
            Browsers = GetBrowserFromConfig();
        }

        private static IEnumerable GetBrowserFromConfig()
        {
            var settings = File.ReadAllText("testParameters.json");
            var config = (JObject) JsonConvert.DeserializeObject(settings);
            var seleniumSettings = config["Selenium"];
            var browsers = (JArray)seleniumSettings["BrowsersToTest"];

            return browsers
                .Select(b => b.ToString())
                .ToArray();
        }
    }

Again, this is a simple and trivial class that simply looks for a testParameters.json in the test run directory and search for the Selenium.BrowserToTest array, that containing the list of the browser to use as test.

Thanks to ValueSourceAttribute we can drive and parametrize the test with a simple external file.

Thanks to those two classes, I can simply specify for each test the Browsers List I want to use.

    [TestFixture]
    public class ParametrizedSeleniumTest : SeleniumTestFixtureBase
    {
        [Test]
        public void Test_Browsers([BrowserList] String browser)
        {
            base.Initialize(browser);
            WebDriver.Navigate().GoToUrl("http://www.microsoft.com");

            WebDriver.Close();
        }
    }

Using BrowserList attribute I’m asking Nunit to create an instance of that attribute to get the list value to be bind to that specific parameter. For each value a different test is created. Now if I build my project I can verify from VS Test Runner that indeed I have three different test to run.

image

 Figure 1: My test running with different versions of browsers

If I run the tests, as I’m expecting, the IE based test fails because I’ve not configured my test to use IE Driver.

image

Figure 2: Test output with Visual Studio Test Runner.

Now remove the IE from the list of Browsers to use in the testParameters.json and rebuild the solution. The IE version of the test now is disappeared from Test Runner.

image

Figure 3: Changing test configuration parameter and rebuilding will update test list

This technique does not depend on anyhthing, and can be used to run test with different configuration if you need (dev machine, different build machines, etc).

Thanks to this technique you can specify whitch browser to use with a simple configuration file and this is a killer feature if you are planning to run test during the build. You can simply modify your testParameters.json file before running the test in your Build to choose which browser to use, or disable completely selenium testing specifying an empty array.

Another option is keeping the list of browsers to use as a comma separated string stored in an Environment Variable, something like

SELENIUM_BROWSERS=Ie,chrome,firefox

With such a technique you can run the test and let the environment specify whitch browser are available to use for testing in that environment.

Gian Maria.

Nunit test not found for some assemblies with Visual Studio Test Runner

I’ve a project in Visual Studio 2013 where one of the assembly containing Tests refuses to show tests in Test Explorer window. The solution has tests written both in Nunit and in MSpec, and everything is good except for that specific assembly. If you notice that Test Explorer window misses some tests, the first thing you need to check is the output windows, where you can find some output for Test Adapters.

image

Figure 1: Output for test adapters can be found in standard Output Window

In the above situation the assembly that is ignored is Intranet.Tests.dll and the reason is that it is built for Framework45 and x64 platform, while the test runner is executing against x86 platform. Everything seems ok, every project is compiled in .ANY CPU, but looking at the raw project file I can confirm that PlatformTarget is set in x64. Changing to x86 (or removing it completely) solves the problem.

image

Figure 2: Platform target changed from x64 to x86

After I changed the PlatformTarget attribute, all tests belonging to that assembly are now available Test Explorer window.

Gian Maria.

Running NUnit and xUnit tests in TFS11 build

I’ve blogged in the past various solution to run NUnit tests during a TFS build, and now it is time to make it again for TFS11, but this time it is incredibly simple, because the new Test Runner supports multiple frameworks, so it works almost automatically.

You can read from Peter Provost blog that actually we have three plugin for UTE (Unit Test Explorer) available: Nunit, xUnit and HTML/JAvascript, they are simple .vsix file (Visual Studio Extension), that you can download, run, and voilà, your xUnit and NUnit tests are runnable from Visual Studio.

image

Figure 1: The new UTE is able to run Unit Tests from multiple framework, because it is extendible

Now if you create a build against this solution, you probably will be disappointed by the fact that the build only runs MStest ignoring tests supported by an external plugin.

image

Figure 2: Build result shows that only 6 test were run, xUnit and NUnit tests are ignored

This blog post explain the reason, if you are on 64 bit machine, the vsix installer is not able to make the Extension visible to build controller, so you need to do a little extra step. First of all locate the folder of the two extensions under the plugin for the current user

image

Figure 3: UTE plugins are located in the standard plugin folder for Visual Studio

This location may vary, in my system it installed only for current user so they are on my profile folder, if you do not find the extension there you can simply search for the dll NUnit.VisualStudio.TestAdapter.dll into your Hard Disk. Once you found xUnit and NUnit UTE plugin folder, you should copy all dll inside a source controlled folder.

image

Figure 4: All UTE plugin assemblies are now in a source controlled folder

Finally go to Team Explorer –> Build –> Actions –> Manage Build Controllers and specify that all custom assemblies are located inside that folder

image

Figure 5: Configure the test controller specifying the location of all the assemblies that contains build extension

Now you can queue another build, and this time all tests should be run.

image

Figure 6: Now all 16 tests were run

As you can see, with TFS and VS11 running unit test from various Unit Test Frameworks is really easy.

Alk.

Unit test NHibernate query to verify N+1

When you work with ORM like nhibernate, having a tool like nhprof is the key of success. But even with NHProfiler you could not prevent people of doing wrong stuff because they do not use it :). I have a simple scenario, a developer changed a method on the server lazily fetching a property of a large resultset. The effect is that the service call, that usually responded in milliseconds starts to respond in 10 seconds.

The reason is really simple, the function loaded about 200 entities from the database and if you are sure you want to access for all 200 entities a property, you should do eager fetching because issuing ~200 queries to the database is not usually a good idea. Then after some time the function started to do timeout, so I inspected again the code and did not find anything strange, but NHprof reveals me that the query was actually fetching another related entity, from a table with millions of row causing timeout. This is due to a modification to a mapping, someone disabled proxy for that class, so NH decided to do a join with the table with millions of row, slowing the method again .

After the problem was fixed, I created some helpers that permits me to write a test that will alert me immediately in the future if such a problem is comeback again.

   1: [Test]

   2: public void verify_xxxx()

   3: {

   4:     var sut = new cccccService();

   5:     sut.DoSomething(1);

   6:     this.NhibernateQueryCount().Should().Be.EqualTo(1);

   7:     String query = this.NhibernateGetQueries().First();

   8:     query.Should().Not.Contain("relatedlin1_.Id");

   9: }

This test is not really a UnitTest, it is more an integration one, but it is able to verify that calling a function on a service class only one query is issued to the database and the query should not eager fetch data from the other table (relatedlin…). This is much more an integration test than a unit test, but it is quite interesting because it permits me to write assertion on number and text of SQL generated by NHibernate, a feature that is really interesting especially if you know that production database is quite big. To achieve this is really simple.

I wrote this simple test helper based on my infrastructure.

   1: public class InterceptNhQueriesHelper : ITestHelper

   2:  {

   3:      public const string nhQueries = "DFCDE96A-ADF7-4C46-A55B-219381364B7F";

   4:      private Dictionary<String, Level> _oldLevel = new Dictionary<string, Level>();

   5:  

   6:      #region ITestHelper Members

   7:  

   8:      public void FixtureSetUp(BaseTestFixture fixture)

   9:      {

  10:          ILogger loggerToForceInitializationOfLoggingSystem = IoC.Resolve<ILogger>();

  11:          var repository = LogManager.GetAllRepositories();

  12:          Log4NetLogEventSourceAppender interceptorAppender = new Log4NetLogEventSourceAppender();

  13:          foreach (var loggerRepository in repository)

  14:          {

  15:              Hierarchy hierarchy = (Hierarchy)loggerRepository;

  16:              if (hierarchy.GetAppenders()

  17:                      .Count(appender => appender.GetType() == typeof(Log4NetLogEventSourceAppender)) == 0)

  18:              {

  19:                  hierarchy.Root.AddAppender(interceptorAppender);

  20:                  hierarchy.RaiseConfigurationChanged(EventArgs.Empty);

  21:              }

  22:              _oldLevel[hierarchy.Name] = hierarchy.Root.Level;

  23:              hierarchy.Root.Level = Level.Debug;

  24:  

  25:              var loggers = loggerRepository.GetCurrentLoggers();

  26:              foreach (var logger in loggers)

  27:              {

  28:                  Logger realLogger = logger as Logger;

  29:                  if (realLogger.Name.IndexOf("NHIBERNATE", StringComparison.InvariantCultureIgnoreCase) >= 0)

  30:                  {

  31:                      _oldLevel[realLogger.Name] = realLogger.Level;

  32:                      realLogger.Level = Level.Debug;

  33:                      if (!realLogger.Appenders.OfType<IAppender>().Any(appender => appender.GetType() == typeof(Log4NetLogEventSourceAppender)))

  34:                      {

  35:                          //non ho appender intercettori

  36:                          realLogger.AddAppender(interceptorAppender);

  37:                      }

  38:                  }

  39:              }

  40:  

  41:              hierarchy.RaiseConfigurationChanged(EventArgs.Empty);

  42:          }

  43:  

  44:          Log4NetLogEventSourceAppender.OnLog += Log4NetLogEventSourceAppender_OnLog;

  45:          loggerToForceInitializationOfLoggingSystem.Debug("TESTLOG DEBUG");

  46:          loggerToForceInitializationOfLoggingSystem.Info("TESTLOG Info");

  47:          loggerToForceInitializationOfLoggingSystem.Error("TESTLOG Error");

  48:      }

  49:  

  50:      private BaseTestFixture currentFixture;

  51:      private List<String> SqlInstructions = new List<string>();

  52:  

  53:      void Log4NetLogEventSourceAppender_OnLog(object sender, OnLog4NetLogEventArgs e)

  54:      {

  55:          if (e.LoggingEvent.LoggerName.Equals("nhibernate.sql", StringComparison.OrdinalIgnoreCase))

  56:          {

  57:              SqlInstructions.Add(e.LoggingEvent.MessageObject as string);

  58:          }

  59:      }

  60:  

  61:      public void SetUp(BaseTestFixture fixture)

  62:      {

  63:          currentFixture = fixture;

  64:          fixture.SetIntoTestContext(nhQueries, SqlInstructions);

  65:          SqlInstructions.Clear();

  66:      }

  67:  

  68:      public void TearDown(BaseTestFixture fixture)

  69:      {

  70:          currentFixture = null;

  71:      }

  72:  

  73:      public void FixtureTearDown(BaseTestFixture fixture)

  74:      {

  75:          var repository = LogManager.GetAllRepositories();

  76:          foreach (var loggerRepository in repository)

  77:          {

  78:              Hierarchy hierarchy = (Hierarchy)loggerRepository;

  79:              hierarchy.Root.Level = _oldLevel[hierarchy.Name];

  80:  

  81:              var loggers = loggerRepository.GetCurrentLoggers();

  82:              foreach (var logger in loggers)

  83:              {

  84:                  Logger realLogger = logger as Logger;

  85:                  if (realLogger.Name.IndexOf("NHIBERNATE", StringComparison.InvariantCultureIgnoreCase) >= 0 &&

  86:                      _oldLevel.ContainsKey(realLogger.Name))

  87:                  {

  88:  

  89:                      realLogger.Level = _oldLevel[realLogger.Name];

  90:                  }

  91:              }

  92:              hierarchy.RaiseConfigurationChanged(EventArgs.Empty);

  93:          }

  94:      }

  95:  

  96:      public int Priority

  97:      {

  98:          get { return 1; }

  99:      }

 100:  

 101:      #endregion

 102:  }

this helpers seems complex, but basically it simply add during FixtureSetup an appender to log4net, this appender basically raises an event whenever it receives a log.

   1: public class Log4NetLogEventSourceAppender : AppenderSkeleton

   2:    {

   3:        private Object _syncRoot;

   4:  

   5:        public Log4NetLogEventSourceAppender()

   6:        {

   7:            _syncRoot = new object();

   8:        }

   9:  

  10:        /// <summary>

  11:        /// Occurs when [on log].

  12:        /// </summary>

  13:        public static event EventHandler<OnLog4NetLogEventArgs> OnLog;

  14:  

  15:        protected override void Append(LoggingEvent loggingEvent)

  16:        {

  17:            EventHandler<OnLog4NetLogEventArgs> temp = OnLog;

  18:            if (temp != null)

  19:            {

  20:                lock (_syncRoot)

  21:                {

  22:                    temp(null, new OnLog4NetLogEventArgs(loggingEvent));

  23:                }

  24:            }

  25:        }

  26:  

  27:    }

  28:  

  29:    public class OnLog4NetLogEventArgs : EventArgs

  30:    {

  31:        public LoggingEvent LoggingEvent { get; private set; }

  32:  

  33:        public OnLog4NetLogEventArgs(LoggingEvent loggingEvent)

  34:        {

  35:            LoggingEvent = loggingEvent;

  36:        }

  37:    }

This appender permits the helper to intercept all logs and since Nhibernate raise a log with name nhibernate.sql with the SQL Code whenever it raises a query to the database, the helper can filter for those kind of messages and store each query inside a standard String collection.

   1: public static class InterceptNhQueriesHelperMethods

   2: {

   3:     public static Int32 NhibernateQueryCount(this BaseTestFixtureWithHelper fixture)

   4:     {

   5:         return fixture.GetFromTestContext<List<String>>(InterceptNhQueriesHelper.nhQueries).Count;

   6:     }

   7:  

   8:     public static List<String> NhibernateGetQueries(this BaseTestFixtureWithHelper fixture)

   9:     {

  10:         return fixture.GetFromTestContext<List<String>>(InterceptNhQueriesHelper.nhQueries);

  11:     }

  12: }

Now I can simply decorate my Test Fixture with a  specific attribute and let the magic happens.

   1: [InterceptNhQueries]

   2:    public class XxxFixture : Test.Utilities.BaseTestFixtureWithHelper

Inside a test I can use NhibernateQueryCount() to know the number of the queries issued by NH during the test and NhibernateGetQueries() to grab the whole list and assert on how NH interacted with the database during a test.

Gian Maria.