How to test the quality of unit tests and know the fake unit tests

1. TejaSoft Innovations Pvt. Ltd.
nagendra.raja@tejasoft.com,
+91-9886723872

3. UNIT TESTS QUALITY
BENDING
#techdebt
By recognizing Fake Tests

4. TYPES OF TESTING

6. PURPOSE OF UNIT TESTS
To know what
developers have
understood as method
level requirment
Avoid repeat manual
pre-commit testing by
Developers
To make sure
regressions are caught
if the main code
changed
To make sure
Developers get to
know bugs much much
before QA team can
report
Bring in Automation to
speed up
development
Help Developers
Identify the Unit Level
Mistake (reduce
debugging time) & Fix
bugs with ease

7. UNIT TESTING SMELLS/QUALITY GAPS
• Empty Unit Tests
• No Asserts
• Wrong Asserts – They should have failed but pass
• @Ignore and commented test code
Fake Unit
Tests
• Even after 100% Cover Coverage, still many production or QA reported issues
• New Code that conflicted past requirement should have failed past Unit tests, but it
does not
Tests are
Not Enough
• Too Much of Mocking
• Flickering
• Interdependencies
• Long Running
• Magic Numbers
Complexity

10. Vaccine created DEFENCE MECHANISM
SHOULD KILL (Unit Tests should fail) mutated
VIRUSES – then Vaccine is effective
If Mutated VIRUS survives (all unit tests pass),
then Unit Tests are not enough.

11. WHAT DOES MUTATION MEAN?
Mutation - Change Main CODE
and see if Unit Tests catch (fail)
What is mutation testing?
Bugs, or mutants, are automatically inserted into your
production code. Your tests are run for each mutant. If your
tests fail then the mutant is killed. If your tests passed, the
mutant survived. The higher the percentage of mutants killed,
the more effective your tests are.

12. HOW TO CREATE THE MUTANTS!
Operator mutations About 30 different patterns like replacing + by - , * by ** and similar, but also > by >=
Keyword mutations Replacing true by false , in by not in and similar
Number mutations You can write things like 0b100 which is the same as 4, 0o100, which is 64, 0x100 which is
256, .12 which is 0.12 and similar. The number mutations try to capture mistakes in this area.
mutmut simply adds 1 to the number.
Name mutations The name mutations capture copy vs deepcopy and "" vs None .
Argument mutations Replaces keyword arguments one by one from dict(a=b) to dict(aXXX=b)
or_test and and_test and ↔ or
String mutation Adding XX to the string
Collections mutations Add Junk Elements or Remove Few elements from List or Array
Value Modify Modify the VALUE of results of method calls
Inject method calls Add get or setter methods multiple times in place of one call
Comment the CODE Remove some code or stop the methods being called
https://pitest.org/quickstart/mutators/

13. 3 TYPES OF MUTATION TESTING
1. DECISION MUTATION
control statements are to be changed
2. STATEMENT MUTATION
developer cut and pastes a part of a code of which
the outcome may be a removal of some lines
3. VALUE MUTATION
values of primary parameters are modified

14. WARNING
May take FOREVER TO RUN

15. CODE - WORKSHOP

16. ‘KATA’ MODULE – CODE WALKTHROUGH
1. Company and Employee classes
2. CompanyRunner – Explain & Execute Main – With Code
Coverage
3. CompanyTest – Explain & Execute Unit Tests – With Code
Coverage
4. Run PITEST kata – Explain, Execute and Show the HTML
Report

17. ‘FIX’ MODULE – CODE WALKTHROUGH
1. CompanyRunnerFixed – Explain & Execute Main – With
Code Coverage
2. CompanyTestFixed – Explain & Execute Unit Tests – With
Code Coverage
3. Run PITEST fix – Explain, Execute and Show the HTML
Report
4. CompanyFixed and EmployeeFixed classes

18. CODE COVERAGE VS MUTATION
Code coverage doesn't tell you everything about the effectiveness of your tests.
Think about it, when was the last time you saw a test without an assertion, purely
to increase the code coverage
Imagine a sandwich covered with chocolate paste
Code coverage would tell you the bread is 80% covered with paste, does not
differentiate what kind of paste is it.
Mutation testing, on the other hand, would tell you it is actually chocolate paste and
not... well... something else if paste is not chocolate

19. REGRESSIONS VS MUTATION - TESTING
1. Regressions are unintended
Since the vast majority of tests have been added due to previous bugs hence, if
they find a problem, you have regressed to a previous state (in which the
problem once again exists). High Failure Rates is, failure of the main code quality
2. Mutation are intended
In Mutation testing framework would intentionally introduce small errors (called
mutations) into your application and test failure are expected. More failures rate
is high quality of Unit Testing

20. REFERENCES
LANGUAGE Mutation Testing Frameworks
Java 1. PIT (GitHub)
https://github.com/vmzakharov/mutate-test-kata - Big Thanks to @vmzakharov, as code and insights of what all I shared is his CODE
only (#tejasoft https://github.com/nagkumar/mutate-test-kata only refactored it)
Different Mutators: https://pitest.org/quickstart/mutators/, https://pitest.org/quickstart/commandline/
2. Jumble - http://jumble.sourceforge.net/
3. µJava - http://cs.gmu.edu/offutt/mujava/
4. Jester - http://jester.sourceforge.net/
5. Javalanche - https://github.com/david-schuler/javalanche
6. tsDetect : https://github.com/TestSmells/TestSmellDetector - Code Smells Detector
C / C++ Mull
C# Visualmutator, Nester
JavaScript Stryker
PHP Infection (formerly called humbug)
Ruby Mutant
Rust mutagen
Swift Muter
Python https://pypi.org/project/mutmut/

21. MAINTAIN DISCIPLINE
Do not cut corners, else
end up with code and tests
of “usual (IT Services)”
quality

22. Q&A
What Gives at most Engineering Kick & Confidence?
Production No Production Fires - Code Just
Works
Developers Unit/Mutation Tests Keep Failing
QA Team Developers accept BUG as Bug
Better code Each Day