+972 52-548-6969
Unit testing frameworks extensibility
I just run into a road block, and I am posting before lunch, hopefully I'll get some ideas about how to solve this.
I have a directory full of source files, which contains tests. What I want to do is to write something like this:
public class MyTest { [ExecuteAllTestsIn("tests/validators")] public void Validators() { } }
The typical scenario for each source file is something like:
public void ExecuteSingleTest(string path) { var test = CompileAndGetInstance(path);
ExecuteTestInUnitTestingFramework(test); }
A source file can contain more than a single test, and I want all those individual tests to be reported as part of the overall test run.
The problem is that I took a look at the extensibility mechanisms for both NUnit and MbUnit, and they are so big. It is not that I can't see how I would solve the issue, but it would require a lot of work to do so.
I know how I can do this with xUnit, where it would be a truly trivial affair.
Am I missing something? I never really had to deal with the extensibility mechanisms in NUnit or MbUnit before.
Comments
I think code generation for that particular scenario works best specially if you are interested in tool integration (think running a particular test case from VS using ReSharper).
The idea is to have a simple boo script somewhere that's executed as part of the build process to generate a partial class definition for the fixture with all the test methods.The meat of test execution can go to another partial definition or to a base type.
OR, code the test assembly in boo and write a ExecuteAllTestsIn attribute. ;)
I have actually worked on this a week ago for orchestrating JSUnit tests withiin NUnit
I used a custom NUnit extension for this (http://www.taumuon.co.uk/rakija/index.html) which can dynamically generate fixtures, and then run parameterised tests from the fixtures. I wrote some code to scan an HTML file for all tests and then create a fixture for each one.
If you want more info let me know.
Actually an alternative one is IterativeTest, if you just want one fixture with all your tests in it...
http://nunit.org/index.php?p=iterativeTest
Oh, also I have no idea about MbUnit, and this doesn't work in Resharper (you need to use the NUnit GUI or Console to run the tests) as I don't think it supports NUnit extensions
Time for everyone to post their how to extend unit testing framework articles :)
Here is mine:
http://www.simple-talk.com/dotnet/.net-tools/testing-times-ahead-extending-nunit/
Look at the "Test Case Builders" section as this can build you a series of tests (one for each directory) using NUnit. But you are correct, NUnit is very heavy when it comes to extensions.
XUnit on the other hand is much easier!
Take a look at my post:
http://blog.benhall.me.uk/2008/01/introduction-to-xunitnet-extensions.html
Use a PropertyData attribute.
[Theory]
[PropertyData("MyTestData")]
Then have that a property which will yield you all the directories to run the unit test in. You would add this attribute to ExecuteSingleTest
public static IEnumerable<object[]> MyTestData
{
get
{
}
}
That's just out the box with xUnit - very quick and easy for scenarios like this I think :)
MbUnit v2 sits somewhere in the middle, you don't have to compile your attribute into a separate assembly but there is still a lot of lifting to do.
Hope this helps.
Ben
That seems to be the way most frameworks are going - an attribute which points to a property which returns the parameters to the tests.
Nice to see that it's OOTB with XUnit but I would add that XUnit is the newest framework so is probably going to try and include these sorts of things in there. The IterativeTest extension comes built into the later versions of NUnit and enables the same thing.
Good luck Ayende
This isn't too hard in MbUnit v3. You define a new data source attribute as a subclass of DataAttribute then implement its PopulateDataSource method.
The next major release (MbUnit v3 Alpha 4) will have a "Factory" mechanism for defining simple data sources as IEnumerables within your test fixture.
Simple things should be simple...
Comment preview