Several experience reports illustrate that mutation testing is capable of supporting a “shift-left” testing strategy for software systems coded in textual programming languages like C++. For graphical modeling languages like Simulink, such experience reports are missing, primarily because of a lack of adequate tool support. In this paper, we extend MUT4SLX, a tool for automatic mutant generation and test execution of Simulink models based on block diagrams. The tool is extended to support mutation operators for Stateflow models, which, to the best of our knowledge, are not supported by any other tool. The current version of MUT4SLX has 8 operators that are modeled after realistic faults (mined from an industrial bug database) and are fast to inject (by only replacing the parameter values). An experimental evaluation on four sample projects shows that MUT4SLX is capable of performing mutation analysis reasonably fast, but mutant execution is always more time-consuming.

1. MUT4SLX
Extensions for Mutation Testing
of Stateflow Models
Speaker — Prof. Serge Demeyer
Authors — Sien Nuyens, Halil Ibrahim Ceylan,
Onur Kilincceker, Mutlu Beyazıt, Serge Demeyer

2. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
DevOps — Continuous Testing
2

3. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Build Pipeline
3
version
control
build
developer
test
deploy
(internal)
automated
system
tests
deploy
(production)
measure &
validate
`
`
merge
conflicts
compiler
warnings
regression
faults
deployment
warnings
unmet
requirements
deployment
warnings
key
performance
indicator
M
u
t
a
t
i
o
n
T
e
s
t
i
n
g

4. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Testing The Quality Assurance
4
© Brussels Airlines

5. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Inject Synthetic Items
5
© "The Good, the Bad and the Ugly: Evaluating Convolutional Neural Networks for Prohibited Item Detection Using Real and
Synthetically Composite X-ray Imagery”. Neelanjan Bhowmik, Qian Wang, Yona Falinie A. Gaus, Marcin Szarek, Toby P. Breckon

6. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
6
Mostly textual programming languages
(C++, C, C#, Java, Python, …)

7. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Industry 4.0: Cyber-Physical Systems
7

8. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
MatLab Ecosystem
Simulink
• sensors and actuators (continuous signals)
• embedded (control) software
StateFlow
• reactive systems
• combinatorial & sequential decision logic
8

9. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Microwave (StateFlow)
9
input = stream of values
• start boolean;
• door_closed boolean;
• clear boolean;
• steps_to_cook int output = stream of values
• mode int

10. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Unit Testing: Streams of Values with Tolerated Deviation
10
public static void assertArrayEquals(String message,
float[] expected,
float[] actuals,
float delta)

11. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Test Design with Signal Builder
11
Assert
Compare expected
against actual signal

12. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Assert with SignalBuilder: Compare Signals
12
Assert
Compare expected
against actual signal

13. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
MUT4SLX: Mutation Testing for Simulink …
13

14. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
MUT4SLX: Mutation Testing for Simulink and StateFlow
14
New!
VST2024

15. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
MUT4SLX Overview
15
Model
Working Directory
Target System
Mutation Operators
Set
Configuration
Start Matlab
Open Working
Directory
Open Model
Select
Mutation
Operators
Find Specific
Block Type for
Mutation
Generate
Mutants
Load
JSON
File
Save Mutants
on CSV File
Pre-run Test
Apply Mutation
on Model
Run-Test
Check Test
Results
Prepare
Report
Detect
Test
Engine
MUT4SLX
Yes
No
Untested
Mutant in
CSV?
Save Report
in HTML format
MUT4SLX
Fast mutant injection by
manipulating via the API
Automatic test execution via Signal
builder and Simulink test.

16. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0. 16

17. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Mutation Operators: Industrial Collaboration
17
# Name Description
1 ROR Relational Operator Replacement
2 LOR Logical Operator Replacement
3 ASR Arithmetic Sign Replacement
4 MMR Min-Max Replacement
5 ICR If Condition Replacement
6 TOR Trigonometric Operator
Replacement
7 MOR Math Operator Replacement
8 PMR Product Multiplication
Replacement
9 POR Product Operator Replacement
10 FIR For Index Replacement
11 FLR For Limit Replacement
12 UDO Unit Delay Operation
13 STR Switch Threshold Replacement
14 SCR Switch Criteria Replacement
15 CR Constant Replacement
# Name Description
1 ROM Relational Operator Mutation
2 LOM Logical Operator Mutation
3 BOM Bitwise Operator Mutation
4 MOM Math Operator Mutation
5 NOM Non-bitwise Operator Mutation
7 CCM Control Chart Mutation
8 CDR Control chart Duration Mutation
9 TOM Transition Order Mutation
New!

18. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0.
Validation: 4 small models
18
System # mutants # killed
mutants
mutation
score
mutant
generation
time
mutant
execution
time
Cruise Control 63 33 52.38% 1.4 sec 11.79 min
Microwave 30 21 70% 0.7 sec 7.68 sec
Aircraft Elevator 2243 - - 66.1 sec -
Automatic Transmission 53 - - 1.35 sec -
System #
subsystems
Total Block
Count
Number of
States
Number of
Transition
Number of
tests
Cruise Control 1 9 7 23 15
Microwave 3 17 4 11 1
Aircraft Elevator 37 825 29 104 0
Automatic Transmission 17 65 9 14 0
It works …
… fast!

19. © Copyright Serge Demeyer 2024. Licensed under CC BY_NC 4.0. 19
https://github.com/haliliceylan/MUT4SLX