Underappreciated Perl Code -- Test::More::subtest()

2013-09-16

Consider a long test that you can break into logical subsections. You could make a flat output with the standard series of TAP ok messages, but Test::More gives a more sophisticated alternative:

use Test::More tests => 3;
 
pass("First test");
 
subtest 'An example subtest' => sub {
    plan tests => 2;
 
    pass("This is a subtest");
    pass("So is this");
};
 
pass("Third test");

This works into the TAP:

1..3
ok 1 - First test
    1..2
    ok 1 - This is a subtest
    ok 2 - So is this
ok 2 - An example subtest
ok 3 - Third test

Notice that the subtest has it’s own test count (“1..2“).

Now, what if we decide that we could speed up this test by fork()ing off and running in parallel, or by using some kind of event interface where we can’t predict what order the callbacks are going to run. This would make the TAP output of each process or event get mixed up with the others.

TAP is actually a defined protocol with lots of features that should see more use. One of the proposals for a new version of the protocol is Test Groups, which looks like this:

1..3
ok 1
1..2 2 a block
1..3 2.1 another block
ok 2.1.1
ok 2.1.2
ok 2.1.3
ok 2.1 # end of another block
ok 2.2
ok 2 # end of a block
1..3 3 a third block
ok 3.1
ok 3.2
not ok 3 # end of a third block, planned for 3 but only ran 2 tests

Which isn’t the prettiest solution, I’ll admit, but it does solve the interleaving problem. The parser can take the test numbers in any order and put them back together.

The other proposal along these lines is Test Blocks, which are similar to what Test::More::subtest() does:

TAP version 14
1..4
begin 1 Object creation
  1..2
  ok 1 Object created OK
  ok 2 Object isa Flunge::Twizzler
end 1 Object creation
ok 2 Clone OK
begin 3 Methods
  1..4
  ok 1 has twizzle method
  ok 2 has burnish method
  ok 3 has spangle method
  not ok 4 has frob method
end 3 Methods
ok 4 Resources released

But this doesn’t solve the interleaving problem.

I had mentioned these in a Lighting Talk at YAPC::NA 2012, and immediately got myself warnocked. So this is me standing up and asking to get this moving again :)