Testing Questions

[Andy Armstrong]

On 14 May 2007, at 13:02, Pete Sergeant wrote:
> Some assorted questions about testing:
>
> * Do you aim for 100% code coverage? Pro: everytime I manage it, I
> tend to zap a bunch of edge-case bugs; Cons: tends to lead to fragile
> tests that know far too much about how the thing you're testing is
> implemented

100% is nice. Boosts confidence, shakes out 'but that can never  
happen' cases. I'd stop short of making it a requirement though.  
Given finite time there are definitely situations where that time is  
better spent doing something else.

> * To what degree do you try and separate programmer-generated unit
> tests, and tests generated by a test engineer? Do you make any
> distinction between regression tests and acceptance tests?

Test engineer? :)

> * At what point are you actually writing your tests? Before you start
> writing code? During the time you write code? Once the code is  
> written?
> Something else?

They usually happen around the same time. Either I'll start on the  
module and then write the first test as soon as the module is  
minimally functional or the test will come first.

I start with a sketch. Sometimes the sketch turns into a module and  
the test comes along slightly behind. Sometimes the sketch turns into  
the test and the module that satisfies that test follows.

> * Does anyone else add extra targets to their make files? I tend to  
> have
> a 'test_coverage' target that runs the tests using Devel::Cover.  
> Anyone
> do anything else that's interesting here?

If you're using EUMM you can do that like this:

    eval 'use ExtUtils::MakeMaker::Coverage';
    warn "Optional ExtUtils::MakeMaker::Coverage not available\n" if $@;

> * I tend to use t::lib::FooBar 't/lib/FooBar.pm' to put testing
> libraries in. Anyone have a better suggestion? How do you share these
> between codebases?

t/lib is fine. If you need to share FooBar.pm then it's just a module  
in its own right no? A dependency.

> * Do you use subdirectories in t/ for different classes of test? We  
> have
> a project with 60 test files with 1400 tests in it, but getting  
> this to
> work involved some Makefile.PL trickery. Any thoughts?

What trickery? Generally make test will recurse into t and find any  
tests. prove needs the -r option to do the same.

> * I tend to name tests with preceeding numbers to make them easy to
> tab-complete to: t/foobar/030_logging_020_file_based.t - anyone  
> have an
> improvement on this?

It's nice to be able to run the 'does this module even load' tests  
before the more complex cases too. So yes to numbering.

> * Is anyone auto-generating tests for their APIs? Documentation? If  
> so,
> how?

I guess the first step to that is to make your tests as data-driven  
as possible. I'll typically have a big array called @schedule or  
somesuch - full of test cases and a little bit of code that chugs  
through them all. If you can add tests without writing any more test  
*code* that'd be a good sign.

Once you're doing that it's much easier to generate tests from  
whatever reference material you have.

-- 
Andy Armstrong, hexten.net