Poster for the paper presented at the NASA Formal Methods Symposium (NFM) by Matthew Cleaveland on May 16, 2023. Abstract: Generating accurate runtime safety estimates for autonomous systems is vital to ensuring their continued proliferation. However, exhaustive reasoning about future behaviors is generally too complex to do at runtime. To provide scalable and formal safety estimates, we propose a method for leveraging design-time model checking results at runtime. Specifically, we model the system as a probabilistic automaton (PA) and compute bounded-time reachability probabilities over the states of the PA at design time. At runtime, we combine distributions of state estimates with the model checking results to produce a bounded time safety estimate. We argue that our approach produces well-calibrated safety probabilities, assuming the estimated state distributions are well-calibrated. We evaluate our approach on simulated water tanks.

1. Our approach is divided into two stages: design time and
run time
• Design time: Abstract the system as a probabilistic
automaton and use model checking to compute the
safety probability of each state of the automaton for a
fixed time horizon
• Run time: Combine state estimates from the system’s
state estimator with the table of safety probabilities to
produce a safety estimate for the system
Motivation
Case Studies
Problem Formulation
Future Work
Our Approach
• Autonomous systems are becoming commonplace
• Increasingly used in safety-critical applications
• Reasoning about their behaviors at run time is
incredibly important
• E.g. safety guarantees
We test our approach on a system of
water tanks with deterministic dynamics
and stochastic perception
• Accurately predicts safety violations
ahead of time
Matthew Cleaveland1, Oleg Sokolsky2, Insup Lee2, Ivan Ruchkin3
Conservative Safety Monitors of Stochastic Dynamical Systems
1Department of Electrical and Systems Engineering, University of Pennsylvania
2Department of Computer and Information Science, University of Pennsylvania
3Department of Electrical and Computer Engineering, University of Florida
Theoretical Guarantee
• Given: A closed loop dynamical system with a controller,
perception, and state estimator
• Goal: Produce well-calibrated and conservative run time
estimates of the probability of the system entering an
unsafe state over a fixed time horizon
• Conservative: Don’t over-estimate the probability of the
system being safe
𝑃𝑚𝑜𝑛𝑖𝑡𝑜𝑟 𝑠𝑎𝑓𝑒 ≤ 𝑃𝑠𝑦𝑠𝑡𝑒𝑚(𝑠𝑎𝑓𝑒)
• Well calibrated: The safety estimates closely match
the true safety of the system
𝑃𝑠𝑦𝑠𝑡𝑒𝑚 𝑠𝑎𝑓𝑒 𝑃𝑚𝑜𝑛𝑖𝑡𝑜𝑟 𝑠𝑎𝑓𝑒 = 𝑝 = 𝑝 ∀𝑝 ∈ [0,1]
State P(safe)
𝑡0 0.3
𝑡1 0.9
𝑡2 0.4
𝑡3 0.8
State P(safe)
𝑡0 0.3
𝑡1 0.9
𝑡2 0.4
𝑡3 0.8
෠
𝑃𝑠𝑎𝑓𝑒 = ∑𝑃 𝑠𝑦𝑠𝑡𝑒𝑚 𝑖𝑛 𝑠 ∗ 𝑃(𝑠𝑦𝑠𝑡𝑒𝑚 𝑤𝑖𝑙𝑙 𝑏𝑒 𝑠𝑎𝑓𝑒 𝑓𝑟𝑜𝑚 𝑠)
Estimated state
distribution
Run time monitor’s
safety estimate
• If our probabilistic automaton model is conservative
• And our state estimator is well-calibrated
• Then our run time safety estimates will be conservative
and well-calibrated
Reliability Diagram ROC Curve
Safe Trial Unsafe Trial
• Our method makes heavy use of model checking
• Use active learning to reduce the model
checking burden
• We assume the state estimator is well-calibrated
• Monitor for if the state estimator is well-calibrated