2013-09-16
Consider a long test that you can break into logical subsections. You could make a flat output with the standard series of TAP `ok` messages, but `Test::More` gives a more sophisticated alternative:
use Test::More tests => 3;
pass("First test");
subtest 'An example subtest' => sub { plan tests => 2;
pass("This is a subtest");
pass("So is this");
};
pass("Third test");
This works into the TAP:
1..3 ok 1 - First test 1..2 ok 1 - This is a subtest ok 2 - So is this ok 2 - An example subtest ok 3 - Third test
Notice that the subtest has it's own test count ("`1..2`").
Now, what if we decide that we could speed up this test by `fork()`ing off and running in parallel, or by using some kind of event interface where we can't predict what order the callbacks are going to run. This would make the TAP output of each process or event get mixed up with the others.
TAP is actually a defined protocol[1] with lots of features that should see more use. One of the proposals for a new version of the protocol is Test Groups[2], which looks like this:
1..3 ok 1 1..2 2 a block 1..3 2.1 another block ok 2.1.1 ok 2.1.2 ok 2.1.3 ok 2.1 # end of another block ok 2.2 ok 2 # end of a block 1..3 3 a third block ok 3.1 ok 3.2 not ok 3 # end of a third block, planned for 3 but only ran 2 tests
Which isn't the prettiest solution, I'll admit, but it does solve the interleaving problem. The parser can take the test numbers in any order and put them back together.
The other proposal along these lines is Test Blocks[3], which are similar to what `Test::More::subtest()` does:
TAP version 14 1..4 begin 1 Object creation 1..2 ok 1 Object created OK ok 2 Object isa Flunge::Twizzler end 1 Object creation ok 2 Clone OK begin 3 Methods 1..4 ok 1 has twizzle method ok 2 has burnish method ok 3 has spangle method not ok 4 has frob method end 3 Methods ok 4 Resources released
But this doesn't solve the interleaving problem.
I had mentioned these in a Lighting Talk at YAPC::NA 2012[4], and immediately got myself warnocked. So this is me standing up and asking to get this moving again :)