Gallio : Why? When? How?

In a time were TDD and Continuous Integration is becoming common place Gallio is a great tool to have in the tool belt. I have been a fan of the related MbUnit for several years but only in the last 6 months have I really seen the light in the separation of the Gallio project and why it is such a good thing.

Lets back the truck up a bit and shed some light on what exactly (in my mind) Gallio really is then we can talk about why you would want to use it.

Gallio is basically an interop facility that act act as a generic test runner. Sure it can be much more than that but at the end of the day 99.99% of people that would be using it  will be using it as a  means to execute tests. Gallio actually is a project that has broken away from the MbUnit project to provide an neutral test runner for other test frameworks.

So what the hell is a test runner? Firstly we would need to look at how we would normally run a (unit) test. Firstly we would typically choose a test framework to write tests in; the common API's that fall in to this category are NUnit, MbUnit, XUnit.Net and MSTest. These allow us to write classes and methods with attributes that describe what and how we wish to test the system under test (SUT). Writing these test does not run the tests, we still need something to kick the process off. This is where our test runners come in to play. TestDriven.Net, ReSharper, Visual Studio test window and the various separate GUI that come with the frameworks (e.g. Nunit GUI Runner and Icarus for MbUnit) allow us to select what tests we wish to execute. Unfortunately there is some degree of coupling present here, i.e. the Visual studio test runner may or may not run your given test framework, NUnit GUI surely doesn't run XUnit tests. There is also the very large issue of being able to run these from the command line or a script; this is pretty important for continuous integration. This is where Gallio fits.

Gallio provide a neutral system that provides a "neutral system for .NET that provides a common object model, runtime services and tools (such as test runners) that may be leveraged by any number of test frameworks." What this means is Gallio can sneak in between your chosen test runner and the test API, providing an abstraction between the two. When I first understood this is was underwhelmed... who cares? Well apparently I do!

You see at my current place of work we, like many .Net teams, use MSTest as our test framework. Being the good kids we are we were keen to get CI up and running and with out TFS properly installed (at the time) we decide to use TeamCity as  our build server. Its a great tool and I have no regrets in using it. Unfortunately trying to get MSTest test to run from a script is a little fiddly and requires an install of a version of visual studio that has MSTest on it on the machine that want to run the test script. Obviously we want or build server to run the test for the solution too, so now we had to install Visual Studio onto our build server... this is not cool.

1. It takes up a lot of space, we had to fight to get a VM created for us to have a build server and  installing VS took up most of the space we were given
2. We had just used up one of our licences of Visual Studio. VS is not cheap. Sure, I work for a huge company that haemorrhages cash, but wasting money is still wasting money.

Enter Gallio. With a minor adjustment* of our build script I can now use Gallio to run my MSTest tests from my MSBuild script. This is pretty cool. What this means is now I have a test framework agnostic build script. If we converted all of our tests to MbUnit I would not have to change my build scripts; MbUnit is supported by Gallio so I am covered. This also means I have nice reports generated for me without crazy MSTest stuff spewing all over my hard drive. The reports are very clean, configurable and human readable. I can show my department manager (who may or may not be technical) the test reports for all of our projects and he can see what state they are in. Having a clean readable report seriously helps in promoting our good work, something an MSBuild log file or nasty MSTest XML would not do so well.

OK so who should be interested in Gallio?

People who "do" CI: Having a free test runner on the build server may be saving you cash and is a big benefit, I would say however having a neutral runner means easier maintenance and is the biggest win here. The build scripts will all use the same syntax. Gallio works with the above mention test frameworks but also integrates with MSBuild, NAnt, NCover, PowerShell, CC.Net and TeamCity.

People who use (or potentially may use) more than one test framework: Having Gallio in the mix means running NUnit from visual studio is very simple. Pick your poison; TD.Net, ReSharper and VS can all now run that or any other Gallio supported framework.

People who want good consistent Test Reports: This is certainly my opinion, but I really like the Gallio reports. They are clear, easy to navigate and if you are using multiple frameworks you can now have a consistent format to display your reports.

Something to get you started - an MSBuild template for using the Gallio.MSbuildTasks assembly:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <!-- This is needed by MSBuild to locate the Gallio task -->
    <UsingTask AssemblyFile="[Path-to-assembly]\Gallio.MSBuildTasks.dll" TaskName="Gallio" />
    <!-- Specify the tests assemblies -->
    <ItemGroup>
        <TestAssemblies Include="[Path-to-test-assembly1]/TestAssembly1.dll" />
        <TestAssemblies Include="[Path-to-test-assembly2]/TestAssembly2.dll" />
    </ItemGroup>
    <Target Name="RunTests">
        <Gallio 
            Assemblies="@(TestAssemblies)">
            IgnoreFailures="true" 
            ReportDirectory="%(ReportDirectory)"
            ReportTypes="html"
            ShowReports="true"> 
            <!-- This tells MSBuild to store the output value of the task's ExitCode property
                 into the project's ExitCode property -->
            <Output TaskParameter="ExitCode" PropertyName="ExitCode"/>
        </Gallio>
        <Error Text="Tests execution failed" Condition="'$(ExitCode)' != 0" />
    </Target>
</Project>

Hopefully this helps shed some light on the Gallio project and how it may fit into your build and test process.

*The minor adjustment is actually cleaning up the script which is also a good thing. It is much clearer what is happening. The MSTest hacks involve small amounts of wand waving.