Mutation testing is the true test for whether or not your source code does the right thing, and whether it's properly tested by your unit tests. But while we're on a killing spree against the mutants of our source code, let's reflect a bit on what these mutants really represent, and why we want to include the unit tests that kill them in our test suite. What's the real point of boundary testing? Do we really need every statement in the source code? Have we tested all calculations? How do we know that we got all the access modifiers on our methods right? These are some of the questions that we would like to reflect on for a moment. We'll start this talk with a short introduction to mutation testing, but only to set the stage. The core of this talk consists of a walk-through of Java code and unit tests, and a reflection on TDD and the quality of our unit tests. At the end of this talk you should be more suspicious about both your source code and your unit tests.

1. 1

2. JavaZone 2016
www.slideshare.net/filipvanlaenen/dial-m-for-mutation
Filip van Laenen
fvl@computas.com
@filipvanlaenen
Dial M for Mutation

3. 3

4. 4

5. https://www.quora.com/What-are-the-funniest-desire-path-examples-you-know/answer/Katie-Sonder5

6. https://www.reddit.com/r/funny/comments/35vomd/oh_the_irony/6

7. Hadi Hariri, “The Silver Bullet Syndrome”, JavaZone 2016
7

8. 8

9. 9

10. https://blog.codecentric.de/en/2016/02/sensible-mutation-testing-dont-go-killing-spree-2/
10

11. 11

12. MUTATION TESTING
12

13. Mutation Testing
• Unit tests testing the
source code
13

14. Mutation Testing
• Unit tests testing the
source code
14

15. Mutation Testing
• Unit tests testing the
source code
• Make a change to the
source code
15

16. Mutation Testing
• Unit tests testing the
source code
• Make a change to the
source code
• Failing unit test
16

17. Mutation Testing
• Unit tests testing the
source code
• Make a change to the
source code
• Failing unit test
• Many failing unit tests
17

18. Mutation Testing
• Unit tests testing the
source code
• Make a change to the
source code
• Failing unit test
• Many failing unit tests
• Infinite loops, compilation
errors, run-time errors,
etc…
18

19. Mutation Testing Book
• Free e-book
• Work in progress
• leanpub.com/mutationtesting
19

20. Mutation Testing Tools
• PITest
• Java
• http://pitest.org/
• Mutant
• Ruby
• https://github.com/mbj/mutant
20

21. GETTERS, SETTERS,
CONSTRUCTORS AND BUILDERS
21

22. Getters and Setters
• How difficult can it be?
• If I had a dime for every
time…
22 https://www.flickr.com/photos/mussikatz/16015546324/

23. Detach from the Input
• Isolate inner state from
input parameters
• Detach from input
parameters
• Arrays
• Collections
• Objects in general
23 https://www.flickr.com/photos/macieksz/503731332/

24. Detach the Output
• Don’t expose internal
state
• (Deep) copies
• Unmodifiable collections
24 https://www.flickr.com/photos/kelloggphotography/2099315965/

25. Unit Testing Getters, Setters, Constructors and Builders
• Are getters, setters, constructors and builders wired together
correctly?
• Primitive types, but also objects
• Are setters, constructors and builders detaching correctly from
input parameters?
• Arrays, collections, objects
• Are getters returning (deep) copies or unmodifiable versions?
25

26. BOUNDARY CHECKS
26

27. Unit Testing a Square
• How many unit tests do
you need?
27 https://www.flickr.com/photos/jasoncooper/13204428724/

28. Unit Testing a Square
• How many unit tests do
you need?
• 0?
28 https://www.flickr.com/photos/jasoncooper/13204428724/

29. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
29 https://www.flickr.com/photos/jasoncooper/13204428724/

30. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
• 2?
30 https://www.flickr.com/photos/jasoncooper/13204428724/

31. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
• 2?
• 4?
31 https://www.flickr.com/photos/jasoncooper/13204428724/

32. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
• 2?
• 4?
• 8?
32 https://www.flickr.com/photos/jasoncooper/13204428724/

33. 33

34. [-1, 1] × [-1, 1]
34

35. [-1, 1] × [-1, 1]
KISS implementation:
boolean isInside(double x, double y) {
return -1 <= x && x <= 1
&& -1 <= y && y <= 1;
}
35

36. [-1, 1] × [-1, 1]
KISS implementation:
boolean isInside(double x, double y) {
return -1 <= x && x <= 1
&& -1 <= y && y <= 1;
}
36

37. “Naive” Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
37

38. “Naive” Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
→ 9 out of 17 mutants killed
38

39. “Naive” Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
→ 9 out of 17 mutants killed
39

40. “Naive” Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
→ 9 out of 17 mutants killed
40

41. Adding a Corner Case
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
41

42. Adding a Corner Case
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
→ 11 (+2) out of 17 mutants killed
42

43. Adding Corner Cases
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
43

44. Adding Corner Cases
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
→ 13 (+2) out of 17 mutants killed
44

45. Adding Corner Cases
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
→ 13 (+2) out of 17 mutants killed
45

46. Adding Corner Cases
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
→ 13 (+2) out of 17 mutants killed
46

47. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
47

48. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
→ 14 (+1) out of 17 mutants killed
48

49. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
49

50. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
→ 15 (+1) out of 17 mutants killed
50

51. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
51

52. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
→ 16 (+1) out of 17 mutants killed52

53. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
53

54. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
54

55. “Naive” Unit Tests?
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
55

56. Useless Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
56

57. Useless Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
57

58. Useless Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
58

59. Six Unit Tests
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
59

60. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
• 2?
• 4?
• 8?
• 6
60 https://www.flickr.com/photos/jasoncooper/13204428724/

61. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
• 2?
• 4?
• 8?
• At least 6
61 https://www.flickr.com/photos/jasoncooper/13204428724/

62. [-1, 1] × [-1, 1]
KISS implementation:
boolean isInside(double x, double y) {
return -1 <= x && x <= 1
&& -1 <= y && y <= 1;
}
62

63. [-1, 1] × [-1, 1]
A mutant (currently) not generated by
PIT:
boolean isInside(double x, double y) {
return -1 <= x && Math.floor(x) <= 1
&& -1 <= y && y <= 1;
}
63

64. [-1, 1] × [-1, 1]
A mutant (currently) not generated by
PIT:
boolean isInside(double x, double y) {
return -1 <= x && Math.floor(x) <= 1
&& -1 <= y && y <= 1;
}
64

65. [-1, 1] × [-1, 1]
A mutant (currently) not generated by
PIT:
boolean isInside(double x, double y) {
return -1 <= x && Math.floor(x) <= 1
&& -1 <= y && y <= 1;
}
65

66. [-1, 1] × [-1, 1]
And other mutants…
66

67. [-1, 1] × [-1, 1]
And other mutants…
…which may require domain
knowledge to construct…
67

68. Binding the Square
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-2, 0));
assertFalse(square.isInside(0, 2));
assertFalse(square.isInside(2, 0));
assertFalse(square.isInside(0, -2));
68

69. Binding the Square
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-1.01, 0));
assertFalse(square.isInside(0, 1.01));
assertFalse(square.isInside(1.01, 0));
assertFalse(square.isInside(0, -1.01));
69

70. Unit Testing a Square
• How many unit tests do
you need?
• 0?
• 1?
• 2?
• 4?
• 8?
• At least 6
70 https://www.flickr.com/photos/jasoncooper/13204428724/

71. 71
https://www.flickr.com/photos/jasoncooper/13204428724/
Notice the difference

72. CONDITIONALS
72

73. Testing Conditionals
• Involves three things:
• Exercising the conditional
• Running the conditional
clause
• Verifying the functionality
of the conditional clause
73 https://www.flickr.com/photos/borkurdotnet/6797142125/

74. Testing Conditionals
• Involves three things:
• Exercising the conditional
• Running the conditional
clause
• Verifying the functionality
of the conditional clause
If you can’t reach the clause,
consider removing the code!
74 https://www.flickr.com/photos/borkurdotnet/6797142125/

75. Mutating Conditionals
• Three ways to mutate a
conditional:
• Negate
• Replace with false
• Replace with true
75

76. Mutating Conditionals
• Three ways to mutate* a
conditional:
• Negate
• Replace with false
• Replace with true
* Or get it wrong
76

77. Mutating Conditionals
• Three ways to mutate* a
conditional:
• Negate
• Replace with false
• Replace with true
* Or get it wrong
77

78. PRESENCE AND NECESSITY
OF EVERY EXPRESSION
78

79. Conditions in Isolation
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-1.01, 0));
79

80. Unused Code
• Missing unit tests?
• Code that can be deleted?
80

81. LOGS AND EXCEPTION MESSAGES
81

82. Testing Log Messages
• Why shouldn’t you?
• Ways to get log messages
wrong:
• Absent
• Wrong log level
• Wrong message content
• Exceptions
82 https://www.flickr.com/photos/waltstoneburner/7946581522/

83. How to Test Logging
• Create an in-memory
appender
• Set up the condition for
the message to be logged
• Check for:
• Its presence
• Log level
• Message content
83 https://www.flickr.com/photos/waltstoneburner/7946581522/

84. How to Test Exceptions
• Set up the condition for
the exception to be
thrown
• Check for:
• Its presence
• Exception type
• Message content
84 https://www.flickr.com/photos/7304439@N06/1453921573/

85. CONSTANTS
85

86. “
86
Πάντα εῥ ῖ – Everything flows
https://en.wikipedia.org/wiki/Heraclitus

87. Constant Constants
• Some constants are
pretty constant:
• 0, 1, 2, e, π, c, …
• Others may change:
• 18 (age of majority)
• 16, 21, 25, …
• Or are magic numbers :
• 42, population sizes, …
87 https://www.flickr.com/photos/theilr/16621361510/

88. Reuse or Redefine?
• Constant constants:
• Reuse
• Changing constants:
• Redefine
• Magic numbers:
• Redefine
• Double-entry
bookkeeping
88 https://www.flickr.com/photos/52814185@N02/6643480765/

89. Unit Testing Constants
• Constant constants:
• Implicitly
• Changing constants:
• Implicitly
• Magic numbers:
• Explicitly
89 https://www.flickr.com/photos/quasimondo/97503572

90. ACCESS MODIFIERS
90

91. Testing Public APIs
• Functionality ≠ public API
• Testing in the same
package:
• Common practice
• JUnit, TestNG, …
• Maven conventions
• IDEs
• “Interface exhibitionism”
91 https://www.flickr.com/photos/neesam/28674222645/

92. Testing Public APIs
• Test functionality in the
same package
• Use the default access
modifier when needed
• Test the public API from a
different package
• To figure out which
interfaces, classes and
methods should be public
92 https://www.flickr.com/photos/neesam/28674222645/

93. Access Modifier Mutator
• I could wish for an access
modifier mutator:
• On every interface, class,
method, field, etc…
• public → protected
• protected → (package)
• (package) → private
93 https://www.flickr.com/photos/neesam/28674222645/

94. CONCLUSION
94

95. https://www.reddit.com/r/funny/comments/35vomd/oh_the_irony/95

96. 96

97. Useless Unit Tests
assertTrue(square.isInside(0, 0));
assertFalse(square.isInside(2, 2));
assertTrue(square.isInside(1, 1));
assertTrue(square.isInside(-1, -1));
assertFalse(square.isInside(-1.01, 0));
assertFalse(square.isInside(0, 1.01));
assertFalse(square.isInside(1.01, 0));
assertFalse(square.isInside(0, -1.01));
97

98. JavaZone 2016
www.slideshare.net/filipvanlaenen/dial-m-for-mutation
Filip van Laenen
fvl@computas.com
@filipvanlaenen
Questions?
Comments?

99. Thanks!