Friday, May 23rd, 2008

Parameterized Tests With NUnit 2.5

It has been possible to write parameterized tests for NUnit for some time, using Andreas Schlapsi’s RowTest Extension, which recognizes the RowTest and Row attributes familiar to MbUnit users.

With the NUnit 2.5 alpha releases, NUnit continues to support that extension – and even bundles a copy of it. But NUnit 2.5 also introduces its own set of attributes for data-based testing, beginning with the TestCaseAttribute, which I will describe here.

Here’s a simple use of this new attribute…

[TestCase(12, 3, 4)]
[TestCase(12, 2, 6)]
[TestCase(12, 4, 3)]
[TestCase(12, 0, 0, ExpectedException = typeof(System.DivideByZeroException),
      TestName = "DivisionByZeroThrowsExceptionType")]
[TestCase(12, 0, 0, ExpectedExceptionName = "System.DivideByZeroException",
      TestName = "DivisionByZeroThrowsNamedException")]
public void IntegerDivisionWithResultPassedToTest(int n, int d, int q)
{
      Assert.AreEqual(q, n / d);
}

This creates five different test cases, displayed and reported separately. The two last tests show alternate ways to specify an expected exceptionl. Note the use of the TestName property, to specify a meaningful name for each of these cases.

Since this test consists of a single Assert.AreEqual statement, we can simplify it further by modifying the test method to return a result and indicate the expected result with the result property…

[TestCase(12, 3, Result = 4)]
[TestCase(12, 2, Result = 6)]
[TestCase(12, 4, Result = 3)]
[TestCase(12, 0, ExpectedException = typeof(System.DivideByZeroException),
      TestName = "DivisionByZeroThrowsExceptionType")]
[TestCase(12, 0, ExpectedExceptionName = "System.DivideByZeroException",
      TestName = "DivisionByZeroThrowsException")]
public int IntegerDivisionWithResultCheckedByNUnit(int n, int d)
{
      return n / d;
}

For more information on this feature, see the NUnit documentation.

Thursday, May 8th, 2008

NUnit 2.5 Alpha 2 Released

With a European trip about to start, I decided to release a second Alpha so that the new stuff would get some visibility. I won’t be doing another release till late June, so please give this one a try.

As compared to 2.4, NUnit 2.5 has quite a lot:

  • Data-driven tests using [TestCase] and [DataSource]
  • Additional asserts and constraints, including inline exception tests.
  • Parallel and distributed testing using pNUnit.
  • Bundled addins, now including RowTest and CSUnitAddin.

For more info, see the release notes.

In addition, watch the NUnit mailing lists or this blog for info about other extensions as they are released for NUnit 2.5.

Thursday, May 8th, 2008

A Simple NUnit Usage Recipe

Scott White has a blog about NUnit Best Practices. The approach may require adjustment on more complex projects, but it’s a very simple recipe for those starting out with NUnit.

Sunday, May 4th, 2008

Common.Addins.Build : Addin Building Made Simple

Let me start right out by saying: I know some of you won’t find this to be simpler – but it is! If you can’t bring yourself to install NAnt or work from the command line, then this isn’t for you. But if you can get past the initial hump – or if you’re already past it – then this is for you.

The hardest part about building – as opposed to writing – addins is getting the references right. Addin solutions in Visual Studio generally contain from three to five projects: the addin assembly, the assembly referenced from user tests, a unit test assembly, possibly a separate test fixture assembly for use by the tests and one or more samples. All of those have to reference the nunit assemblies they will be used with when they are installed. Getting it right is tedious at best, requiring multiple visits to the Add Reference Dialog. And from time to time, Visual Studio will decide to use a different assembly from the one you expected.

I finally got fed up with all this and decided to write a NAnt script for one of my addins – CSUnitAddin actually. After it was done, I looked closely and realized that most of it was boilerplate – the same code with a few property changes would build other addins. So I started to factor out the common code into an include file, with the result that my CSUnitAddin.build file now looks like this:

<?xml version="1.0"?>
<project name="CSUnitAddin" default="build" basedir=".">

  <!-- Include the common build file -->
  <include buildfile="../common.addin.build"/>    <!-- 1 -->>

  <!-- Define non-default names for sample project -->
  <property name="sample.dll" value="money.dll"/>
  <property name="sample.dir" value="money"/>
  
  <!-- Define non-default names for fixture project -->
  <property name="fixture.dll" value="CSUnitTestFixtures.dll"/>
  <property name="fixture.dir" value="CSUnitTestFixtures"/>

  <!-- This addin is for the csunit framework -->
  <property name="target.framework.dll" value="csunit.dll"/>

  <!-- Define alt library with proper version of csunit         -->
  <!-- Must be here since lib.dir is defined in the common file -->
  <property name="csunit.version" value="2.0.4"/>
  <property name="alt.lib.dir" value="${lib.dir}/csUnit-${csunit.version}"/>

</project>

This example actually has to do more than most, since it uses a “foreign” testing framework. I’ll walk through the steps for you:

  1. First, the script includes the addin.common.build sript, which will do all the work.
  2. Next, it defines non-standard names for the sample assembly and it’s directory. I could have eliminated this by calling them both “sample” but I wanted a non-generic name here.
  3. I do the same thing for the “fixture” assembly – an assembly that contains csunit-based tests, which are loaded and run by my unit test assembly. In this case, I might have stuck with the default (CSUnitAddinFixture) and saved a few lines of xml.
  4. I set the target.framework.dll to csUnit so that my sample and fixture assemblies will be built with a reference to that framework rather than the default nunit framework.
  5. This particular addin has multiple library subdirectories for different versions of csUnit, so I set the alt.lib.dir property to indicate which one I want to use.

So, let’s say that I want to build and test this addin with NUnit 2.5, using both the .NET 1.1 and 2.0 builds. I can do this with two commands:

nant release net-1.1 nunit-2.5 clean build test
nant release net-2.0 nunit-2.5 clean build test

The script will locate my NUnit 2.5 installation, build the addin and it’s associated assemblies (four in all), install it and run the tests using that same installation. NUnit 2.5 has separate net-1.1 and net-2.0 directories, which the script knows about, so the two builds will be installed separately.

This is how I’m building addins now. I use Visual Studio to do editing and syntax checking, leaving the actual builds to the script. As a result, I no longer need to deal with mismatched references or with the tedium of using the gui to switch between NUnit versions.

The Common.Addin.Build script is open source, licensed under the Academic Free Licence 3.0. You can get it here.

Friday, May 2nd, 2008

Using Addins with NUnit-Console

A recent bug pointed out that addins are not recognized when running tests under the console runner. This is due to a missing entry in the nunit-console.exe.config file, which you can easily fix yourself. Follow these steps to have your addins recognized when using the console runner:

  1. Open the nunit-console.exe.config file in any convenient editor – notepad is fine.
  2. Find the <runtime> element
  3. Insert the code below within the <runtime> element. You can copy it from nunit.exe.config if you prefer
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="addins"/>
</assemblyBinding>

From this point on, your addins should work equally well in both the Gui and Console runners.

Friday, May 2nd, 2008

NUnit 2.4.7, 2.5, 3.0: What’s With That?

A few folks are confused by the various release numbers being announced or discussed all at one time, so I thought I’d clarify:

NUnit 2.4.7 is the latest production release of NUnit. It’s the one we recommend most people use for your tests. Some fairly critical performance bugs have been fixed in the last few releases, so you should update even if you’re only one or two digits back. See what you’re missing !

NUnit 3.0 is the planned – but not yet released – next generation NUnit. We call it the NUnit Extended Testing Platform, to distinguish it from the current NUnit Framework. It will provide a superset of the functionality of the current framework and is generally described here. I’ll be posting further info on NUnit 3.0 as it progresses.

NUnit 2.5 is release that wasn’t originally planned. The 2.4 series was supposed to be followed by 3.0. However, a number of people asked for a quicker release that included features provided by other test frameworks, which are currently missing from NUnit. So 2.5 is now in alpha, with extensions to support data-driven tests, easier exception testing and a number of other goodies. It’s also being bundled with pNUnit, an extension that supports distributed parallel tests. See the release notes for details. Watch for the second Alpha release sometime next week.

Hopefully, this will clear things up till we generate some more numbers.

Wednesday, December 12th, 2007

Time for a New NUnit

As I’ve worked on NUnit 2.4 and its follow-ups, I have begun to feel that it’s time for a much more significant update to NUnit, a rethinking of what it’s all about and how it should work. Of course, at the same time, I don’t want to lose all the work that has gone into NUnit up to now. Puzzling over this, and helped by a group of dedicated NUnit users and contributors, I think I now have a direction to go.

The new NUnit 3.0 Extended Testing Platform is at the same time a simplification of NUnit and an order of magnitude larger and more complex.

How can that be, you ask? Well, it’s simpler because it’s being divided into pieces and users won’t need to understand everything, just the parts they use. Similarly, contributors will be able to focus on a small part of NUnit, adding their own touches, without rebuilding the whole thing.

On the other hand, it’s more complex because it is moving from being a framework for writing unit tests to a testing platform, one that can work with many frameworks and many runners. The new platform is centered around a testing engine, with an API that can be used by any program needing to load and run tests. You’ll be able to write your own runner, in any suitable lanuage, and get test execution functionality from the NUnit test engine. Want to write it in Ruby? No problem. Using Silverlight or Mono’s Moonlight? Again, you’re free to do so.

At the other end of things, you’ll be able to use our new plugin architecture to enable NUnit to run whatever tests you like. Plugins will be available for common alternative frameworks and to add special functionality like parallel execution to the standard NUnit tests.

I’ll post more about what’s coming in the near future. Meanwhile, for a more detailed overview, read the NUnit 3.0 Vision document.

You can expect early (alpha ) releases in the first part of next year.

Saturday, November 3rd, 2007

NUnit Stable Releases

While NUnit 2.4 was in development, I continued to maintain the NUnit 2.2 series as a stable, bug-fix-only branch. NUnit 2.2.10 was the last in that series.

After NUnit 2.4 came out, I decided to experiment with using a single code line. I would make bug fixes and – at the same time – carefully add new features. Each of the releases since then (2.4.1, 2.4.2, 2.4.3 and 2.4.4) has fixed a number of things, introduced some new features and broken one or two things.

A single code line can work, but it requires discipline. Part of that discipline involves doing many of the same things manually that would be handled automatically by a well-thought-out branching structure. The outcome of my experiment is that the approach didn’t work well for me, so I’m going back to the use of release series branches.

If you use the source code from CVS, the Release-2-4-Branch will get you the latest fixes in the 2.4 series. I’ll be tagging the next releases – 2.4.5, etc. – on that branch. The CVS head will be used for any new development, but I’m not sure how much of that there will be. My best guess is that there will be a 2.5 feature release and that will be it for NUnit 2.x.

NUnit 3.0 is now in the planning stages and I’ll be blogging about it in the coming weeks.

Saturday, November 3rd, 2007

Blogging Again

I haven’t done much blogging for a while due – in part – to the enormous amount of Spam my blog was attracting. But I just updated the software and installed a spam filter. I plan to start posting again, and re-open the blog to comments. Hopefully, the spam won’t overwhelm me this time.

Since I last wrote, NUnit 2.4.3 and 2.4.4 have come out as well as NUnitLite 0.1 and 0.2. I’ll blog separately about the status of each of those.

Tuesday, August 7th, 2007

NUnit Extras: Test Result Summarizer

NUnit is currently built under five different runtimes on Windows – plus two on Linux. Each build is tested under the same runtime on which it was built and on a number of the other runtimes. That’s a lot of TestResult files.

I use nunit-summary.exe to quickly view what happened in a particular run. For example,

nunit-summary TestResult-mono-1.0.xml -noheader -brief

gives me a simple, one-line summary of what happened when I ran my tests under mono-1.0 – the naming convention is built into my build script. The following command gives me a set of files TestResult-net-1.0.txt, TestResult-net-1.1.txt, etc. each containing the summary line as well as any failed or not run tests for the corresponding xml file.

nunit-summary TestResult*.xml -out=*.txt

See the help for the command for other things you can do, including using your own xsl to transform the output.

The nunit-summary program is available on the GitHub.