What is wrong on Test::More? / Test::Moreが抱える問題点とその解決策

YAPC::ASIA 2012

What is wrong
on Test::More?
Test::Moreが抱える問題点とその解決策

makoto kuwata <kwa@kuwata-lab.com>
http://www.kuwata-lab.com/
2012-09-27 (Fri)

copyright(c) 2012 kuwata-lab.com all rights reserved.

Agenda
✦ Testing without spec
仕様を書かずにテストしてる

✦ Not structured tests
テストが構造化されてない

✦ Needs test plan
事前にテストプラン (＝テスト数) を必要とする

✦ No ﬁxture feature
フィクスチャ機能がない

✦ Hard to distinguish assertions
どれがアサーションなのか分かりにくい


Section 1:

Testing without spec
仕様を書かずにテストしている


Point

Write your test
according to spec,
not your code.
テストは、コードではなく仕様をもとに書け。


Sample: Test::More (Perl)

use Test::More tests => 4;

is f(0), 0;
is f(1), 1;
is f(2), 1;
is f(3), 2;


Sample: RSpec (Ruby)

describe 'f()'
it "calculates fibonacchi sequence" do
f(0).should == 0
f(1).should == 1
f(2).should == 1
f(3).should == 2
end
end


Sample: unittest (Python)

import unittest

class FooTest(unittest.TestCase):
def test_calculates_fiboacchi_seq(self):
"""Calculates Fibonacchi sequence"""
self.assertEqual(0, f(0))


Goal of test

✦ Test::More:

Does that code run correctly?
そのコードは意図した通りに動くか？

✦ RSpec:

Does that code satisfy the spec?
そのコードは仕様を満たしているか？


Difference between Test::More and RSpec

## Test::More
like $html, qr'<h3>Hello</h3>';

Higher-level information
より高水準の情報
## RSpec
it "contains section title" do
html.should =~ %r'<h3>Hello</h3>'
end


Solution: spec()
https://gist.github.com/3797929

sub spec {
my ($text, $block) = @_;
$block->();
}

## usage
spec "page contains section title", sub {
ok(render() =~ qr`<h1>Hello</h1>`);
};


Spec First

Step 1. Write specifications
spec "...specification1...";
spec "...specification2...";
#=> not ok 1 - ...spec... # TODO
#=> not ok 2 - ...spec... # TODO

Step 2. Add assertions according to spec
spec "...specification...", sub {
assertion1;
assertion2;
};

Solution: spec()

sub spec {
return $block->() if $block;
TODO: {
local $TODO = ": not implemented yet";
ok(undef);
}
}


Meaning of output lines
As is - 'ok' when assertion passed, 'not ok' when
failed
アサーションが成功したらok、失敗したらnot ok

ok 1 - assertion1
not ok 2 - assertion2

To be - 'ok' when spec satisﬁed, 'not ok' when not
コードが仕様を満たしたらok、満たさなければnot ok

ok 1 - specification1
not ok 2 - specification2


Spec : Assertion = 1 : N
✦A speciﬁcation can contain some
assertions
1つの仕様に複数のアサーションを書いてよい

spec "returns pair of integer", sub {
is scalar(@$ret), 2;
like $ret->[0], qr/^d+$/;
like $ret->[1], qr/^d+$/;
};


Spec : Assertion = 1 : N

As is Output lines per assertion
アサーションごとに出力行
ok 1
ok 2
ok 3

To be
ok 1 - returns pair of integer

Output lines per spec
仕様ごとに出力行


Solution: OK()

sub OK {
my ($expr) = @_;
unless ($expr) {
my ($pkg, $file, $lineno) = caller();
die "AssertionFailed"
." at $file line $lineno.n";
}
}


Solution: spec()
my $spec = undef;
my $num = 0;

sub spec {
$spec = $text;
$num++;
eval { $block->(); };
my $err = $@;
if (! $err) {
print "ok $num - $textn";
} else {
print "not ok $num - $textn";
$err =~ s/^/# /mg;
$err .= "n" if $err !~ /nz/;
print STDERR $err;
}
}


Conclusion of this section
✦ Write test based on spec, not on code
テストは、コードに対してではなく、仕様に対して書く

✦ 'Spec specified?' instead of 'Run correctly?'
「正しく動作するか？」ではなく「仕様を満たしているか？」

✦ Spec ﬁrst, assertion second
仕様を先に書いて、そのあとにアサーションを書く

✦ Spec : Assertion = 1 : N
1つの仕様が複数のアサーションを含んでよい

✦ Output line per spec, not assertion
出力行のok / not okはアサーション単位ではなく仕様単位に出す


Section 2:

Not structured tests
テストが構造化されてない


Point

Test should have structure.
Because spec has structure.
テストには構造がある。なぜなら仕様に構造があるから。


Sample: Speciﬁcation document

クラス：Calendar
メソッド：isLeapYear(int year): boolean
動作詳細：
・100で割り切れる場合、
・400で割り切れる場合はtrueを返す
・それ以外はfalseを返す
・4で割り切れる場合はtrueを返す
・それ以外はfalseを返す


Test code as spec document

Is your test code available
as spec document?
そのコードは仕様書としてほんとに利用できるの？


Sample: Test::More

use Test::More tests => 2;
use Foo;

$foo = Foo->new();
is(Foo->new()->meth1(), 11);
is(Foo->new()->meth2(), 12);


Sample: RSpec
Test target (class, method, ...)
require 'rspec' テスト対象 (クラス、メソッド、…)
describe Foo do
describe '#bar()' do
context 'when arg is provided' do
it "returns length" do
Foo.new.methd1([0,1,2]).should == 3
Foo.new.methd1([]).should == 0
end
end
end
end


Sample: RSpec
Test condition or situation
require 'rspec' 条件や状況
describe Foo do
end
end
end
end


Sample: RSpec

require 'rspec'
describe Foo do
end
end
end Speciﬁcation
仕様
end


Sample: unittest (Python)

import unittest

class FooTest(unitteset.TestCase):

def test_bar_1(self):
"""returns length of arg passed"""
self.assertequal(3, Foo().bar([1,2,3]))

def test_bar_2(self):
"""returns 0 when arg is not passed"""
self.assertEqual(0, Foo().bar())


Sample: Test::Unit2 (Ruby)

require 'test/unit'
require 'foo'

class FooTest < Test::Unit::TestCase
class MethTest < self
def test_returns_length_of_arg
n = Foo.new.bar([1,2,3])
assert_equal 3, n
end
end
end
ref: http://www.clear-code.com/blog/2012/4/25.html

Sample: subtest() (Test::More)

use Test::More tests=>1;

subtest "package Foo", sub {
plan tests=>1;
subtest "sub bar()", sub {
plan tests=>2;
ok (1+1 == 2);
ok (1-1 == 0);
};
};


Sample: subtest() (Test::More)

$ perl homhom.t
1..1
1..1
1..2
ok 1
ok 2
ok 1 - sub bar()
ok 1 - package Foo


Solution: subtest() alternatives
Test target
テスト対象
print "1..2n";
topic 'package Foo', sub {
topic 'sub meth1()', sub {
case_when 'arg is passed', sub {
spec "1+1 should be 2", sub {
OK(1+1 == 2);
};
spec "1-1 should be 0", sub {
OK(1-1 == 0);
};
}; Condition or situation
}; 条件や状況
};


$ perl homhom.t
1..2
# * package Foo
# * sub meth1()
# - when arg is passed
ok 1 - 1+1 should be 2
ok 2 - 1-1 should be 0



my $depth = 0;

sub topic {
my ($name, $block) = @_;
my $indent = ' ' x $depth;
print "# $indent* $namen";
$depth++;
$block->();
$depth--;
}



my $depth = 0;

sub case_when {
my ($condition, $block) = @_;
print "# $indent- $conditionn";
$depth++;
$block->();
$depth--;
}


Conclution of this section
✦ Testhas structure, because spec has
structure.
テストには構造がある。なぜなら仕様に構造があるから。

✦ xUnit
focuses on test automation,
RSpec focuses on test structure.
xUnitは自動化のための道具、RSpecは構造化のための道具

✦ Output
of subtest() suck.
Deﬁne your own subtest alternatives.
subtest()は出力が残念すぎる。自前関数お勧め。


Section 3:

Needs test plan
事前にテストプラン (=テスト数) を必要とする


Sample: Test::More


is(1+1, 2);
is(1-1, 0);

$ perl homhom.t
1..2
ok 1
ok 2


Sample: subtest()


subtest 'package Foo', sub {
plan tests=>1;
subtest 'sub meth()', sub {
plan test2=>1;
is(1+1, 2);
is(1-1, 0);
};
};


Pros of test plan
✦ Detect unexpected test ﬁnishing
テストの異常終了が検知できる

• Compare number of ('ok' + 'not ok') with test
plan
出力されたokやnot okの数と、宣言されたテスト個数とを比較

✦ Necessary to report test progress
テスト実行時に進行状況を知るのに必要

• Especially for 'prove' command
特にproveコマンドで


Cons of test plan
✦ Too messy to keep currect value
正しい値に更新し続けるのが面倒すぎる

• Update test plan when you add assertions
assertion関数を追加したら忘れずに更新

• back to top when you add at the end of ﬁle
ファイルの末尾にテストを追加したら、先頭に戻ってテスト数
を更新


Sample: done_testing()


is(1+1, 2);
is(1-1, 0);
done_testing();

$ perl homhom.t
ok 1
ok 2
1..2


done_testing() and subtest()


subtest "package Foo", sub {
subtest "sub meth1()", sub {
ok (1+1 == 2);
ok (1-1 == 0);
done_testing();
};
done_testing();
};
done_testing();
(No need to call done_testing() in subtest() since Test::More 0.95_01)


$ perl homhom.pl
ok 1
ok 2
1..2
ok 1 - sub meth1()
1..1
ok 1 - package Foo
1..1


Pros of done_testing()
✦ No need to speicfy test plan!
テスト数を指定しなくていい！


Cons of done_testing()
✦ Need to expand TAP spec
TAP仕様に拡張が必要

• Simplicity of TAP has gone
もはやTAPの簡易性は損なわれた

✦ 'Prove' prints '?' as number of tests
proveコマンドで全体のテスト数が「?」に

• Degeneration of interface
インターフェースとしては退化


Off Topic: Doubt about TAP
✦ If TAP accepts test plan after running tests,
テスト数がわかるのがテスト終了後でいいなら

• 'End of test' indicatior is necessary for TAP, but
test plan is not, is it?
テストの終わりが分かる何かがあればテスト数いらなくね？

• Index number of test is not necessary, is it?
そもそも ok や not ok に通し番号いらなくね？

ok 1 # ..spec..
ok 2 # ..spec..
<<TEST END>>


The root cause of problem
✦ Impossibleto count number of tests
before running tests
テスト数を事前に数えられない

✦ To be
あるべき姿

• Step1. Count and print number of tests
テスト数を数えて出力

• Step2. Run tests
テストを実行


Solution: Intermediate data structure
As is:
is 1+1, 2; ok 1
is 1-1, 0; ok 2
1..2

To be:
topic
is 1+1, 2; topic 1..2
is 1-1, 0; ok 1
spec ok 2
spec
Easy to
count tests

Cons of intermediate data structure
✦ Easyto count and ﬁlter tests before
running tests
テスト実行前にテストを数えたりフィルタするのが簡単にできる

✦ No need to extend TAP speciﬁcation
TAPの仕様を拡張しなくてよい

• No more done_testing()
done_testing()なんていらない

• No more nesting like subtest()
subtest()のような入れ子対応はいらない


Sample: xUnit

class FooTest(TestCase):
def test1(self): ...
def test2(self): ...

## build intermediate data structure
suite = TestSuite()
suite.add(FooTest('test1'))
suite.add(FooTest('test2'))

## run tests
TestRunner().run(suite)


Solution: topic() and spec()

topic 'class Foo', sub {
topic 'sub meth1()', sub {
case_when "arg is given", sub {
spec "1+1 should be 2", sub {
OK(1+1 == 2);
};
}; Change to build tree
};
};
Traverse tree
run_all();


Solution: topic()

my $NODES = [];

sub topic {
my ($name, $block) = @_;
my $node = {name=>$name, prefix=>'*',
children=>[]};
push @$NODES, $node;
my $bkup = $NODES;
$NODES = $node->{children};
$block->();
$NODES = $bkup;
}


Solution: spec()

my $NODES = [];

sub spec {
push @$NODES, [$text, $block];
}


Solution: _count_specs()

sub _count_specs {
my ($nodes) = @_;
my $n = 0;
for (@$nodes) {
if (ref($_) eq 'HASH') { # topic
$n += _count_specs($_->{children});
} elsif (ref($_) eq 'ARRAY') { # spec
$n += 1;
}
}
return $n;
}


Solution: run_all()
sub run_all {
print "1..", _count_specs($NODES), "n";
_run_all($NODES, 0, 0);
}

sub _run_all {
my ($nodes, $depth, $num) = @_;
for my $x (@$nodes) {
if (ref($x) eq 'HASH') { # topic
print "# $indent$x->{prefix} $x->{name}n";
$num = _run_all($x->{children}, $depth + 1, $num);
}
elsif (ref($x) eq 'ARRAY') { # spec
my ($text, $block) = @$x;
$num++;
_run_spec($text, $num, $block);
}
}
return $num;
}

Conclustion in this seciton
✦ Don't count tests manually. Use computer.
テスト数を手動で数えるのはやめてコンピュータにさせよう

✦ Noneed to expand TAP speciﬁcation.
Both done_testing() and subtest() are
wrong.
TAPの仕様拡張は不必要。done_testing()もsubtest()も間違い

✦ Intermediate data structure solves the
problem.
テストを表す中間データ構造を作れば万事解決


Section 4:

No ﬁxture feature
フィクスチャ機能がない


What is ﬁxture?

"A test ﬁxture (also known as a test context) is the
set of preconditions or state needed to run a test.
The developer should set up a known good state
before the tests, and return to the original state
after the tests."
http://en.wikipedia.org/wiki/XUnit

"テストを実行、成功させるために必要な状態や前提条件の集合
を、フィクスチャと呼ぶ。これらはテストコンテキストとも呼
ばれる。開発者はテストの実行前にテストに適した状態を整
え、テスト実行後に元の状態を復元することが望ましい。"
http://ja.wikipedia.org/wiki/XUnit

Fixture method
✦ xUnit
• setUp() / tearDown()

✦ RSpec
• before() / after()

✦ Test::More
• (nothing!)


Fault of setUp/tearDown
All tests in a class must share a setUp/tearDown.
sub setUp { my ($self) = @_;
$self->man = User->new(gender=>'M');
$self->woman = User->new(gender=>'W');
}
sub test1 { my ($self) = @_;
my $user = $self->man;
...
}
sub test2 { my ($self) = @_;
my $user = $self->woman;
...
}

Fault of setUp/tearDown
Separate TestCase class?
package FooTestCase;
sub setUp { ... }
sub testFoo { ... }

package BarTestCase;
sub setUp { ... }
sub testBar { ... }

No. Test structure should follow speciﬁcation
reason, not ﬁxture reason.


Another approach on fixture
Define fixture method for each test data.
sub fx_man {
return User->new(gender=>'M'); }
sub fx_woman {
return User->new(gender=>'W'); }

spec "test for man", sub {
OK(fx_man()->{gender} eq 'M'); }
spec "test for woman", sub {
OK(fx_woman()->{gender} eq 'W'); }

More flexible than setUp/tearDown


Cons of the approach

spec "returns length of file", sub {
my $file = fx_file("homhom");
OK(file_length($file) == 6);
unlink($file);
};

Troublesome task to do teardown
for each ﬁxture data
ﬁxtureごとに忘れずに解放処理を行うのは面倒


Solution: at_end()

sub fx_file {
my ($content) = (@_);
$file = "test-".rand().".txt";
write_file($file, $content);
at_end { unlink($file); };
return $file;
}
Register task called at end of test
テストの終わりに実行される処理を登録する


Solution: at_end()

spec "returns length of file", sub {
my $file = fx_file("homhom");
OK(file_length($file) == 6);
unlink $file;
};

No need to teardown for each test
テストごとの終了処理を書く必要がなくなる


Solution: at_end()

our @_CLOSURES = ();

sub at_end(&) {
my ($closure) = @_;
push @_CLOSURES, $closure;
}


Solution: _run_spec()

sub _run_spec {
my ($text, $num, $closure) = @_;
eval { $closure->(); };
my $s = $@ ? "ok" : "not ok";
print "$s $num - $textn";
my $@ = undef;
for my $clos (reverse(@_CLOSURES)) {
$clos->();
}
@_CLOSURES = ();
}


Conclusion in this section
✦ No ﬁxture feature in Test::More
Test::Moreにはﬁxture機能がない

✦ SetUp()/tearDown() are not so good
setUp()/tearDown()も良くはない

✦ Use end_at() which registers teardown
task in setup
作成時に解放処理を登録できるat_end()が便利


Section 5:

Hard to distinguish assertions
どれがアサーションなのか分かりにくい


Assertions in xUnit

Consistent naming rule for assertion methods.
アサーションメソッドに一貫した命名規則がある

Easy to distinguish assertions in test code
テストコード中でアサーションを見分けるのが簡単

assertTrue()
assertEqual()
assertMatch()
assertSet()
...


Assertions in Test::More

No naming rule for assertion functions.
アサーションメソッドに一貫した命名規則がない

Hard to distinguish assertions in test code
テストコード中でアサーションを見分けるのが困難

ok()
is()
like()
eq_set()
cmp_ok()
...


Solution: AssertionObject class
Collect assertion functions in a class.
アサーション関数を1つのクラスに集約

package AssertionObject;

## assertion methods
sub num_eq { ... }
sub str_eq { ... }
sub match { ... }
...


Solution: OK()
Change OK() which returns AssertionObject.
AssertionObjectインスタンスを返すようにOK()を変更

sub OK {
my ($actual) = @_;
return AssertionObject->new($actual);
}
Easy to distinguish assertions
## usage どれがアサーションかすぐ分かる
OK (1+1)->num_eq(2);
OK ("a")->str_eq("a");
OK ("9")->match(qr/^d+/);


Solution: Operator overload

package AssertionObject;
use overload
'==' => &num_eq,
'eq' => &str_eq;

package main;
OK (1+1) == 2; // same as num_eq()
OK ("a") eq "a"; // same as str_eq()

Shortcut to assertion methods
アサーションメソッドを簡単呼び出し


ok() vs. OK()

ok(2 == 1);
not ok 1
# Failed test at homhom.t line 4.

OK(2) == 1;
not ok 1
# AssertionFailed at homhom.t line 4.
# $actual == $expected: failed
# actual: 2
# expected: 1


is() vs. OK()

is() :
is("1.0", 1.0); #=> not ok

OK() :
OK ("1.0") eq 1.0; #=> not ok
OK ("1.0") == 1.0; #=> ok

You can choose 'eq' or '=='.
文字列演算子と数値演算子を選べる


Test::More vs. OK()

Test::More OK()
ok(1+1 == 2); OK (1+1) == 2;
isnt(1+1, 0); OK (1+1) != 0;
cmp_ok(1+1, '>=', 1); OK (1+1) >= 1;

Consistent usage
一貫した使い方


Conclusion in this section
✦ Noconsistent naming rule of assertion
functions in Test::Unit
Test::Moreにはアサーション関数に一貫した命名規則がない

✦ Hard to distinguish assertions in test
テストコード中でどれがアサーションか見分けるのが困難

✦ Solution:
AssertionObject class and
operator overload
解決策：AssertionObjectクラスと演算子オーバーロード


Oktest.pm - a new style testing library
http://search.cpan.org/~kwatch/Oktest/lib/Oktest.pm

use strict;
use warnings;
no warnings 'void'; # RECOMMENDED!
use Oktest;
topic "ClassName", sub {
topic "method_name()", sub {
spec "...detail...", sub {
OK (1+1) == 2;
OK ('a' x 3) eq 'aaa';
};
};
};
Oktest::main() if $0 eq __FILE__;
1;

What is wrong on Test::More? / Test::Moreが抱える問題点とその解決策

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to What is wrong on Test::More? / Test::Moreが抱える問題点とその解決策

Similar to What is wrong on Test::More? / Test::Moreが抱える問題点とその解決策 (20)

More from kwatch

More from kwatch (20)

Recently uploaded

Recently uploaded (20)

What is wrong on Test::More? / Test::Moreが抱える問題点とその解決策