This document discusses integrating formal proofs and testing in software verification strategies. It describes how executable contracts can be used to test requirements at different levels, from low-level to high-level, more easily than traditional testing. A hybrid verification approach is proposed that uses both formal proofs and testing, with contracts protecting assumptions, to gradually introduce formal methods and gain their benefits while maintaining assurance from traditional testing.

1. Integrating Proof &
Testing in Verification
Strategies
Cyrille Comar

2. Agenda
• Place of Testing in Safety Standards
• Programming with executable Contracts
• Executable Contracts vs Formal Contracts
• Reducing Testing with Formal Proofs

3. Place of Testing in DO-178
DO-178 define mainly 6 sets of objectives related to
• Planning (7)
• Development (7)
• Verification (43)
• Configuration Management (6) ü Various reviews & analysis
• Quality Assurance (5) ü Requirements-Based Testing
• Certification Liaison (3)
SLR HLR LLR

4. Requirements in DO-178
• There are 3 levels requirements
• System Requirements (SLR)
• High-Level Requirements (HLR)
• Low-Level Requirements (LLR)
LLRs
SLRs HLRs
Allocated
to
Software
Software
Architecture

5. Requirements- Based (RB) Testing in DO-178
RB
Test Generation
LLR
HLR
Hardware /
Low-Level Software Software
Tests Integration Integration
Tests Tests
RBTest
Coverage Analysis
Statements
Structural Decisions
Coverage Analysis
MCDC

6. Robustness vs functional test cases in DO-178
• 2 kinds of test cases
• Normal Range Test Cases
• Robustness Test Cases
normal
“
…
ability
of
the
software
to
respond
to
inputs
and
conditions
…
”
abnormal
From outside From inside
(response should be specified in SLR/HLR) (Software error)
Integrated software
comp1 comp2 comp3

7. Low-Level Testing is long & difficult
• Is it as relevant as HLR based testing ?
• LLR verification is important but is “testing”
the most appropriate one?
subp2
subp1
subp5
subp3
subp4
Test subp6
harness Subp3
Subp4
stub

8. Programming by Contract
• Precondition
• Postcondition
• A CONTRACT is • Invariant
• Predicate
• A logical property • Assertion
• subprogram
• type
• Associated with a program entity • statement
• It can be verified
• It can be relied upon

9. Programming By Contract ( Ada examples)
procedure Square_Root (Input : Integer; Res : out Natural)
with
pre => Input >= 0,
post => Res * Res <= Input
and (Res+1)*(Res+1) > Input;
Implicit contract
Explicit contract type Positive_Stack is private
with
Invariant => Is_Empty (Stack)
or else Top (Stack) > 0;
subtype Odd_Positive is Natural
with
Predicate => (Odd_Positive/2)*2 /= Odd_Positive;

10. What can Contracts be used for?
• Adding user-defined dynamic checks at runtime
• A way of expressing constraints, properties
LLRs
“When a low level requirement can be expressed as a
contract, it can be tested much more easily”

11. Robustness Testing & Contracts
• Internal Robustness issues are related to Run-Time Errors
• Ada guarantees (most cases of) RTEs è run-time exception
• Preconditions are better than “defensive coding”
• The more runtime checks the better Robustness Testing is

12. Requirement Testing & Contracts
• LLR is expressed as a subprogram contract
• Successful execution of Postcondition è Test successful
• No need for collecting & verifying output

13. Executable Contracts vs Formal Contracts
procedure
Square_Root
procedure
Square_Root
(Input
:
Integer;
Res
:
out
Natural)
(Input
:
Integer;
Res
:
out
Natural)
with
-­‐-­‐#
pre
Input
>=
0;
pre
=>
Input
>=
0,
-­‐-­‐#
post
Res
*
Res
<=
Input
post
=>
Res
*
Res
<=
Input
-­‐-­‐#
and
(Res+1)*(Res+1)
>
Input;
and
(Res+1)*(Res+1)
>
Input;
(Ada
2012)
(SPARK
2005)
• Execution of “executable” contracts can be disabled
• The syntactic difference is not relevant
• The same contracts can be interpreted in 2 different world
Executable boolean expression First order logic formula

14. Requirement based Verification
& Formal Contracts
• LLR is expressed as a subprogram formal contract
• Successful proof of Postcondition è LLR verified for any input
• Approach allowed by DO-333
• Proof of absence of RTEs provides for some Robustness verification

15. Hybrid Verification (1)
• Some LLRs are tested
• Some LLRs are proved
• Is the combination as “strong” as all LLRs are tested?

16. Hybrid Verification (2)
Modular Formal verification is based on assumptions:
• My Precondition is never violated by Callers
• Postconditions of callees are true after the call
• all outputs are known
• Direct or Indirect
• Explicit (out parameters) or Implicit (globals)
• No surprises (e.g. unexpected aliasing)

17. Hybrid Verification (3)
Tested_Function
calls Proved_Function
Precondition must be executed
Proved_Function
calls Tested_Function
Postcondition must be executed
• Executable contracts “protect” assumptions of formal verification
• Globals can be computed by static analysis
• Surprises can be eliminated by subsetting

18. Conclusion: Hybrid Verification Benefits
• Helps with gradual introduction of Formal proofs
• The traditional 80 / 20 % rule is true for both formal
verification and testing
80%
remaining
easily
tested
80%
easily proved

19. Extra information
• Open-DO Initiative
• www.open-do.org
• http://www.open-do.org/projects/hi-lite/

20. Cyrille Comar
+33-1-49-70-67-16
comar@adacore.com
46 Rue d’Amsterdam
Paris
75009