During the process of putting together The Definitive Guide To Catalyst I got into a few arguments of principle with co-authors and the technical reviewers. The one where they pointed out that I'm a lazy boy and don't put explicit return statements into short methods I lost soundly, and the code was corrected. However, the debate over test planning was much more interesting.
The vast majority of perl tests are written using Test::More or some variant thereof. The "usual" approach, or at least the commonly taught one, is to declare the number of tests you're intending to run at the top of the file via:
use Test::More tests => 23;
or
use Test::More; ... plan tests => 23;
However, in the book I write my tests by doing:
use Test::More qw(no_plan);
which means that whatever tests are run, are run, and how many were actually run doesn't get checked at the end. And oh my but did the reviewers go mental with it, I got a comment along the lines of "I do hope this was just because you were showing the simplest code, and you wouldn't do this in production.".
Well, actually, yes, I would.
I claim that numeric plans don't actually gain you anything, promote false lazyness that can cause bogus test suites, and don't scale well to large teams - and being too lazy to write entire applications myself, I care very much about things scaling to large teams.
So, first, the case for the opposition. Why are numeric test plans good? Well, first, they protect you against code doing:
exit 0;
somewhere in your test suite and causing half of it not to be run.
No, wait, what?! When was the last time you saw code that did that? Certainly in applications perl, which is what I spend most of my time writing, that's simply not something that's done. Not a single library I've used in the past several years has done that, and no junior developer at any shop I've worked with would make that sort of silly mistake more than once.
In fact, there've even been occasions where I wanted to cut off the end of a test suite and have intentionally added that exit statement and had its effects been precisely what I wanted.
Second, a numbered plan means that if a loop test doesn't do the right thing then it'll get caught - something like gimme_a_list() returning the wrong number of elements in:
foreach my $member (gimme_a_list()) {
ok($member->is_shiny, 'Member '.$member->name.' is shiny');
}
Well, yes, it would catch gimme_a_list() returning nothing. But if we actually care about the number of elements returned, WHY THE HELL AREN'T WE TESTING FOR WHAT?! I mean, if it matters then you should be writing:
my @list = gimme_a_list();
cmp_ok(scalar(@list), '==', 3, 'gimme_a_list() returned three entries');
foreach my $member (@list) {
...
This not only keeps important information near where it's needed, it also traps the case where one loop test runs one too many times and another one too few (no, really, I've seen this ...). So in my opinion, that argument doesn't hold water either.
Perhaps most importantly from my POV, a whole-file test plan regularly causes merge conflicts - if the test file starts out with 71 tests, and I add three in my branch, and somebody else adds four in theirs, then we're going to get a conflict on merge and the developer merging will have to resolve it to the correct final value of 78 tests. And now we've got an apparent conflict in the versioning history that is completely meaningless.
Worse still, if we both add three tests each then there will be no merge conflict but the resulting test file will now fail. Congratulations, You Broke The Build - and you didn't even realise you were doing it, because yours and the other guy's tests are both bloody correct!
I hope you're now at least convinced that I have an argument about why no_plan is A Good Thing in a lot of cases. Better still, there's now a safe cure for the 'exit 0' problem - if you have Test::Simple 0.88 or greater (which is what contains Test::More), you can now do:
use Test::More; # no_plan not required ... done_testing;
and if your test script bails out part way through you'll get an appropriate error.
If you're planning to buy the book from Amazon, please consider following one of the following links to buy it from amazon.com, or to buy it from amazon.co.uk - and if you're elsewhere then please substitute your appropriate country TLD into one of those links. That way the affiliate revenue will go to the Enlightened Perl Organisation rather than amazon's back pocket, as well as my share of the revenues - which I'm donating to Enlightened Perl anyway>.
-- mst, out