Did you know that CPAN comes with a free QA team? CPAN Testers is a distributed, grass-roots project with over 6.5 million test reports. This talk describes how the project benefits Perl developers and offers four important practices for any large-scale, volunteer QA effort

1. Free QA!
David Golden
dagolden@cpan.org
www.dagolden.com
OSCON ▪ July 22, 2010

2. Free as in beer
Free as in speech

3. Quality
+
Assurance

4. Quality is an ideal

5. Quality Assurance is
what you can measure

6. CPAN Testers is Free QA for CPAN
The objective of the group is to test
as many of the distributions on CPAN as
possible, on as many platforms as possible.
The ultimate goal is to improve the
portability of the distributions on CPAN,
and provide good feedback to the authors.

7. CPAN is 15 years young
~135,000 uploads from ~5000 authors
~23,000 unique libraries uploaded
~21,000 active libraries available
~1,000 uploads every month
~200 new libraries every month

8. CPAN Testers is 11 years young
7.8 million reports
1000+ CPAN Testers
Peak month: 400,000+ reports

9. Learning
from
Crisis

10. Four ideas
for success

11. Culture
Fun
Mashups
Scale

12. Values Tools
Culture

13. $ perl Makefile.PL
$ make
$ make test
$ make install

14. $ perl Makefile.PL
$ make
$ make test
$ make install

15. CPAN (1995)
CPAN.pm (1996)
$ cpan File::Spec
(runs 'make test')

16. We built tests
into our tools

17. Thousands of people
running CPAN tests
... and throwing them away

18. CPAN Testers (1999)
cpan-testers@perl.org
Reports submitted "by hand"

19. CPANPLUS (2002)
Can submit reports
"automatically"
when you test or install

20. A step change in participation
0
5
10
15
20
25
30
35
40
45
50
2000
2001
2002
2003
2004
2005
2006
2007
2008
Reports/
Upload
A. Tang, “Becoming a CPAN Tester
with CPANPLUS” (2002)

21. We built tests
and test reporting
into our tools

22. But still just ~5 reports per upload
0
5
10
15
20
25
30
35
40
45
50
2000
2001
2002
2003
2004
2005
2006
2007
2008
Reports/
Upload

23. Crisis #1: Lack of coverage
CPAN Testers no longer a toy
but not yet useful for much

24. Fun
Community Competition

26. Step 1:
Make it easy to join

27. CPAN::YACSmoke (2005)
POE::Component::CPAN::YACSmoke
(2006)
CPAN::Reporter (2006)

28. Step 2:
Build a community
and give feedback

29. http://stats.cpantesters.org/
(2006)
cpan-testers-discuss@perl.org
(2007)
Barbie starts a monthly blog
(2007)

30. Competition

31. September, 2007
"The heat is on for Chris Williams,
he only thought Dave Cantrell
was chasing his heels, but
Andreas Koenig has also stepped
up his game too. It has been a
staggering month for the sheer
volume of reports..."
- Barbie

32. Climb the leaderboard

34. Test more or weirder
Perls and platforms

35. Perl version and OS coverage

36. The result of -Ofun
0
5
10
15
20
25
30
35
40
45
50
2000
2001
2002
2003
2004
2005
2006
2007
2008
Reports/
Upload

37. Zooming out
0
50
100
150
200
250
300
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Reports/
Upload

38. … rise of the machines …

39. CPAN::Reporter::Smoker (2008)
POE::Component::SmokeBox (2008)
CPANPLUS::YACSmoke (2009)
minismokebox (2009)

40. Reports by #1 tester each month
1
10
100
1,000
10,000
100,000
1,000,000
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Manual
Reporting
Automated
Reporting
Dedicated
Smoking
Smoke
Clusters

41. More testers & reports per tester
0
50
100
150
200
250
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
Number of
Testers
Reports per Tester
Manual
Reporting
Automated
Reporting
Dedicated
Smoking
Smoke
Clusters

42. Crisis #2: Spam
~15,000 reports a month (2006)
~250,000 reports a month (2008)
By email

43. Cc: AUTHOR@cpan.orgJoe Tester
To: cpan-testers@perl.org
2006: ~5 reports/upload

44. "Stop SPAMMING me!"
2008: ~120 reports/upload
WTF?!?

45. NOT fun for CPAN authors
Disabled author CC's
Notification through
a central service
Added author contact preferences

46. Mashups
Innovation Insight

47. Open data
allows
innovation

48. Test matrix
matrix.cpantesters.org

49. Dependency tree
deps.cpantesters.org

50. "Volatile 100" "FAIL 100"

51. Regression analysis and failure
correlations
o OS, architecture, perl version, perl
configuration
o test output, e.g. “failed at line …”
o tester, test framework
analysis.cpantesters.org – but offline! 

52. Crisis #3: Email overload
Reports sent by email
Reports re-distributed via NNTP

53. Success? Failure?
455,875
2,080
1,163
953
555
525
206
117
95
89
1 10 100 1000 10000 100000 1000000
perl.cpan.testers
perl.cpan.uploads
perl.perl5.porters
perl.daily-build.reports
perl.beginners
perl.modules
perl.perl5.changes
perl.moose
perl.module-authors
perl.perl6.language
November 2009 perl.org mailing list monthly volume

54. The way perl.org sees email
perl.cpan.testers
everything else
99%

55. November 2009: throttling back

56. Scale
Anticipation Leverage

57. Scale is not always
a technical problem!

58. Visualize
wild success

59. But don't build
for it right away

60. Email worked!
NNTP worked!
For 10 years

61. 1. Anticipate the breakpoint
2. Do something about it

62. FAIL
We should have seen
it coming sooner

63. We built 2.0
by designing an API for
server components
and implementing
a version of it on AWS

64. Clouds give leverage

65. Technical scale
Administrative scale

66. Volunteer time
is more scarce than
nominal cloud costs

67. CT 2.0 running in beta since April

68. Culture within values and tools
Fun from community and competition
Mashups spur innovation and insight
Scale needs anticipation and leverage

69. Thank you for coming!
Your QA task:
Rate this talk, leave comments and
tell me what could be better