The Wisdom of TAP Numbering in a JavaScript World

2019-08-26

TAP started as a simple way to test the Perl interpreter. It worked by outputting a test count on the first line, followed by a series of “ok” and “not ok” strings on subsequent lines. The hash character could be used for comments.

1..4
ok
ok
not ok # Better check if the frobniscator is working
ok

There’s been some additions over the years, but a lot of output of individual tests still looks a lot like this. Multiple tests could be wrapped together with prove. Here’s what it looks like to run that on Graphics::GVG:

$ prove -I lib t/*
t/001_load.t ............. ok     
t/002_pod.t .............. ok     
t/010_single_line.t ...... ok   
t/020_many_lines.t ....... ok   
t/030_circle.t ........... ok   
t/040_rect.t ............. ok   
t/050_color_variable.t ... ok   
t/060_num_variable.t ..... ok   
t/070_ellipse.t .......... ok   
t/080_regular_polygon.t .. ok   
t/090_int_variable.t ..... ok   
t/100_glow_effect.t ...... ok   
t/110_comments.t ......... ok   
t/120_point.t ............ ok   
t/130_include.t .......... skipped: Implement include files
t/140_include_vars.t ..... skipped: Implement include files
t/150_block_var.t ........ ok   
t/160_poly_offset.t ...... ok   
t/170_ast_to_string.t .... ok   
t/180_two_parses.t ....... ok   
t/190_meta.t ............. ok   
t/200_renderer.t ......... ok   
t/210_two_meta.t ......... ok   
t/220_named_params.t ..... ok   
t/230_renderer_class.t ... ok   
All tests successful.
Files=25, Tests=91, 10 wallclock secs ( 0.10 usr  0.05 sys +  8.86 cusr  0.57 csys =  9.58 CPU)
Result: PASS

Since TAP is just text output, it’s easy to implement libraries for other languages. With prove‘s --exec argument, we just pass an interpreter. Here’s an example of a TypeScript project of mine:

$ prove --exec ts-node test/*
test/activator_do_nothing.ts ........ ok   
test/activator_multi.ts ............. ok   
test/authenticator_always_false.ts .. ok   
test/authenticator_always.ts ........ ok   
test/authenticator_multi_fails.ts ... ok   
test/authenticator_multi.ts ......... ok   
test/logger.ts ...................... ok   
test/reader_fh.ts ................... ok   
test/reader_mock.ts ................. ok   
test/sanity.ts ...................... ok   
All tests successful.
Files=10, Tests=12, 17 wallclock secs ( 0.05 usr  0.01 sys + 30.40 cusr  1.16 csys = 31.62 CPU)
Result: PASS

Although the TAP package for JavaScript has a nice little runner all its own, which includes an automatic coverage report:

$ node_modules/.bin/tap --ts test/**/*.ts
 PASS  test/activator_do_nothing.ts 1 OK 5s
 PASS  test/authenticator_always_false.ts 1 OK 5s
 PASS  test/activator_multi.ts 2 OK 5s
 PASS  test/authenticator_always.ts 1 OK 5s
 PASS  test/authenticator_multi_fails.ts 1 OK 5s
 PASS  test/authenticator_multi.ts 1 OK 5s
 PASS  test/logger.ts 1 OK 5s
 PASS  test/reader_fh.ts 2 OK 5s
 PASS  test/sanity.ts 1 OK 3s
 PASS  test/reader_mock.ts 1 OK 3s

                         
  🌈 SUMMARY RESULTS 🌈
                         

Suites:   10 passed, 10 of 10 completed
Asserts:  12 passed, of 12
Time:     14s
--------------------------|----------|----------|----------|----------|-------------------|
File                      |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
--------------------------|----------|----------|----------|----------|-------------------|
All files                 |    93.04 |    78.57 |    82.86 |    93.33 |                   |
 doorbot.ts               |      100 |       80 |      100 |      100 |                   |
  index.ts                |      100 |       80 |      100 |      100 |                26 |
 doorbot.ts/src           |     91.4 |    77.78 |    81.82 |    92.31 |                   |
  activator_do_nothing.ts |      100 |      100 |      100 |      100 |                   |
  activator_multi.ts      |      100 |      100 |      100 |      100 |                   |
  authenticator_always.ts |      100 |      100 |      100 |      100 |                   |
  authenticator_multi.ts  |      100 |       80 |      100 |      100 |                22 |
  read_data.ts            |      100 |      100 |      100 |      100 |                   |
  reader.ts               |    28.57 |      100 |       25 |    33.33 |       39,40,41,44 |
  reader_fh.ts            |    81.25 |        0 |       50 |    81.25 |          43,53,54 |
  reader_mock.ts          |      100 |      100 |      100 |      100 |                   |
--------------------------|----------|----------|----------|----------|-------------------|

Which, of course, can be dropped into the scripts -> test section of your package.json.

There’s a feature of TAP that’s been falling out of use in recent years, even in Perl, and that’s test counts. Modern TAP allows you to put the test counts at the end, which can be done in Test::More by calling done_testing(). In the tap module for Node.js, you can simply not set a test count at all and it does it for you. Here’s an example of that with a sanity test, which just makes sure we can run the test suite at all:

import * as tap from 'tap';
import * as doorbot from '../index';

tap.pass( "Things basically work" );

Avoiding a test count seems to be the trend in Perl modules these days. After all, automated test libraries in other languages don’t have anything similar, and they seem to get by fine. If the test fails in the middle, that can be detected by a non-zero exit code. It’s always felt like annoying bookkeeping, so why bother?

For simple tests like the above, I think that’s fine. Failing with a non-zero exit code has worked reliably for me in the past.

However, there’s one place where I think TAP had the right idea way back in 1988: event driven or otherwise asynchronous code. Systems like this have been popping up in Perl over the years, but naturally, it’s Node.js that has built an entire ecosystem around the concept. Here’s one of my tests that uses a callback system:

import * as tap from 'tap';
import * as doorbot from '../index';
import * as os from 'os';

doorbot.init_logger( os.tmpdir() + "/doorbot_test.log"  );

tap.plan( 1 );


const always = new doorbot.AlwaysAuthenticator();
const act = new doorbot.DoNothingActivator( () => {
    tap.pass( "Callback made" );
});
always.setActivator( act );

const data = new doorbot.ReadData( "foo" );
const auth_promise = always.authenticate( data );

auth_promise.then( (res) => {} );

If the callback to run tap.pass() never gets hit, this test will fail. If you had removed the call to tap.plan( 1 ) towards the start, it would pass as long as it compiles and doesn’t otherwise hit a fatal error. Which is a pretty big thing to miss, since the callback is critical to the functionality of this particular module. Even if I knew it worked now, it might not in regression testing later.

Most (all?) other automated test frameworks around JavaScript have this problem. Sometimes, you can get around it by writing in clever ways, but it is far too easy to write test code that can erroneously declare success. Besides, shouldn’t your tests be written in the most obvious, straightforward way? TAP had the answer decades ago.