Safety-Certifying Open Source Software: The Case of the Xen Hypervisor

[Public]
Copyright (2023) Advanced Micro Devices, Inc.
Safety-Certifying
OSS Software : The case of Xen
• May 10th , 2023
• Stefano Stabellini & Senthil K Rajagopal

[Public]
Functional safety definition
• Safety critical embedded software applications are developed
for systems whose failures contribute to hazards in the system
for safety of life and environment.
• A functionally safe or safety critical system has a safety
function which maintains or transitions the system to safe
state. Safety function can be implemented by combination of
programmable electrical / electronic hardware and software.
• Safety certifications (IEC 61508, ISO 26262 etc..)
• Strict coding guidelines (MISRA C)
• Strict verification & validation (Testing) requirements
• Strict software process requirements

[Public]
Functional safety definition
“Functional safety is part of the overall safety that depends
on a system or equipment operating correctly in response to
its inputs.” (IEC 61508)
“Absence of unreasonable risk due to hazards caused by
malfunctioning behavior of E/E systems” (ISO 26262)
“Safety is freedom from unacceptable risk” (ISO 14971)
Safety is all about doing the job right with good
engineering practices

[Public]
Functional safety compliance
• Functional safety standards provide engineering guidelines to achieve functional safety
for programmable electrical / electronics hardware, software and the overall system.
• Safety standards like IEC 61508 and ISO 26262 recommend V model-based development
flow for hardware (ASIC / SoC/ FPGA) and software.
• Safety standards also provide options to qualify pre-existing software by satisfying
certain engineering requirements – Tailored software safety lifecycle by retroactive
engineering.

[Public]
Functional safety compliance – V model for newly developed software
Specification of software
safety requirements
Software architectural
design
Software unit design and
implementation
Software unit verification
Software integration and
verification
Testing of the embedded
software
Technical safety concept
System and item integration
and testing
Software verification
Software testing
Unit verification
System and item verification
V model-based safety
software development flow
to develop software from
the scratch
The following is the software development flow (V model) for safety related applications according to ISO 26262:

[Public]
Functional safety compliance for Pre-existing software
• Safety standards also provide options to qualify pre-existing software by
satisfying certain engineering requirements – Tailored software safety lifecycle by
retroactive engineering.
IEC 61508-3, 7.4.2.12 Route 3s ISO 26262-8, Clause 12

[Public]
Embedded Hypervisor – Xen
• Xen is the AMD Open-Source reference hypervisor for
embedded and automotive
• Both ARM and AMD x86
• AMD has an in-house engineering team to develop,
enhance, and support Xen for embedded and automotive
• Xen is delivered to customers today as reference and is
supported by Forum, Premium Technical support, and
engineering
• We have many customers using Xen on ARM in production
which require hard-real time isolation between VMs (Xen
cache coloring)

[Public]
Xen : Open-source community
Contributions to Xen by Company
• Xen Project is an Open-Source project under the Linux Foundation
• Well known and widely used in the industry
• Extremely strong review process and security process
• AMD x86 and ARM architectures already fully supported
• Xen is widely used in datacenter, cloud, client devices and more
• The Xen Open-Source Community is a diverse multi-
vendor community
• Maintainers from Amazon, ARM, Citrix, AMD, SuSE, and more
• Independent panel of experts with 15+ years of experience
• AMD has a team working closely with the upstream project
• Healthy long-term maintenance of the project

[Public]
Xen: Open-Source Community
9
Several security-focused
projects and products
based on Xen, including
QUBES OS

[Public]
Xen Hypervisor Rigorous Quality Process
• Xen has an extremely strong quality and validation process already in place
• Rigorous review process famous industry-wide
• Sometimes it is "too hard" to get patches accepted in Xen due to the strict review process
• Strong security process to handle security vulnerabilities with responsible disclosure
• Xen Security Process is best-in-class; OpenStack and other projects' security processes based on Xen’s
• Security and isolation are top priority of the project
• Traceability of commits: all communication done in public on xen-devel, with archives going back 20+ years
• Every commit is tested on a variety of hardware by 2 CI-loops before reaching the “master” branch
• Xen is already widely deployed in critical production environments
• Datacenter & Cloud: Amazon AWS, Rackspace, Citrix XenServer, Vates (multi-tenancy, AMD x86)
• Desktop: QubesOS, OpenXT (AMD x86 highly secure environments; mitigate common cause failures)
• Embedded: Xilinx Petalinux (ARM hard real-time isolation with cache coloring)

[Public]
Safety certifying Xen Hypervisor
• AMD is working on making Xen safety-certifiable for AMD platforms (ARM & AMD x86 )
• IEC 61508 SIL 3 (Systematic Capability 3 ) & ISO 26262 ASIL D
• Certification based on Xen upstream community processes and upstream codebase
• Not working with a private fork
• Ability to update the certification with limited efforts for newer Xen releases
• Certification docs & artifacts available for AMD customers
• Xen upstream community alignment with safety certifications requirements
• MISRA C
• Public interfaces documentation
• Gitlab-CI testing

[Public]
Safety certification scope of Xen Hypervisor
• Scope: IEC 61508 SIL 3 & ISO 26262 ASIL D
• Safety certifications for both Xen on ARM and AMD x86
• Stable AMD x86 hardware and ARM hardware platforms
• Core components in Xen
• x86: AMD-v, AMD-Vi, HPET, vPCI
• ARM: SMMUv3, Arch Timer, Hypervisor Extensions, vPCI
• Xen enabling components for VM-to-VM communication and PV Drivers
• grant table (memory sharing)
• event channels (notifications)
• No OS/hypervisor dependencies
• Dom0less/Hyperlaunch for both ARM and x86
• Domain creation done by Xen at boot based on provided
configuration
• Dom0 is not required; Out of Scope.
HW
HW partition
HW partition
Xen
Safe OS QM Linux
access access

[Public]
Xen hypervisor as a pre-existing software component shall take the following routes for
the safety certification:
Xen Hypervisor safety certification route
Established Open Source Development Process
Established Security Practices
Tailored software safety lifecycle
IEC 61508-3, 7.4.2.12 Route 3s– Assessment of non-compliant development
ISO 26262-8, Clause 12 – Qualification of software components
Organizational Supporting Processes
Independent Functional safety assessment
Xen
Hypervisor
Safety case
Certified
IEC 61508 SC 3
(Element level SIL 3)
ISO 26262 ASIL D
Xen Safe Hypervisor

[Public]
Xen Hypervisor safety certification plan
Xen Hypervisor software safety certification
ISO 26262:2018, Element level ASIL D
IEC 61508, Edition 2 Systematic Capability 3 (SIL 3)
Phase I
Safety
Concept
Review
Phase II
Final
Assessment

[Public]
Phase I
Safety
Concept
Approval
Safety concept review
Assessment of functional safety management
• Software safety requirements
• Software architecture specification
• Verification & validation plan (high level)
• Software verification plan
• Software validation plan
• Software tools classification
• Safety assessment plan
• Safety plan
• Software requirements management plan
• Project management plan
• Document management system
• Software development process
• Continuous quality improvement strategy
• Software configuration management plan
• Software change management plan
• Competence management plan
• Organizational safety culture tracking sheet
• Product security compliance

[Public]
Safety analysis
Software tools qualification
• Software FMEA report
Phase II
Final
Assessment
• Software tools qualification reports
Implementation / Coding
• MISRA C (based) Coding guidelines
• Coding guidelines conformance report
• Code review evidence
Software verification
• Software Verification specification
• Software Verification report
Software validation
• Software validation specification
• Software validation report
• Traceability reports
User guide & safety manuals
• General software user guide
• Software safety manual for IEC 61508 & ISO 26262
Safety case • Safety case

[Public]
Software safety engineering work products & evidence
S.No Work Products & Evidence Purpose
1 Software Safety Requirements (SSR) List of all functional requirements of the software
2 Software Architecture Specification (SAS) Document describes all the interfaces and design corresponding to the functional requirements
3 Coding:
• MISRA C Coding guidelines
• MISRA C Coding guidelines conformance matrix / code review report
Implementing (new features) and / or refactoring the code to meet good coding practices such as
MISRA C.
4 Software Verification :
• Software Verification Plan (Test Plan)
• Software Test Specification
• Software Test Report
To ensure all the developed software modules function correctly with their dependencies as per
the specification.
The test scope shall cover testing of all the interfaces, dynamic (e.g. code coverage), static testing
(e.g. control / data flow) , performance test, regression test, hardware test & fault injection test.
5 Software Validation :
• Software Validation Plan (Test Plan)
• Software Validation Test Specification
• Software Validation Test Report
• Traceability Reports
To ensure the developed software meets its original requirements and intended usage.
The test scope shall cover requirements test and testing the assumptions of usage.
Report also shall be generated to show the traceability between requirements and the test
results.
6 Tools Analysis :
Software Tools Classification & Qualification Report
To ensure tools used in development such as compiler and other testing tool do not introduce any
faults (bugs) in the target software, software tools are analyzed, classified and qualified (if
required) for the usage.
7 Software Failure Analysis :
Software Failure Mode Analysis (Software FMEA) Report with Mitigations
To ensure no additional hidden silent failures in the software, possible failures are identified, and
mitigations are implemented or proposed.
8 Process Control:
Plan documents for document / change/ configuration control and project
management
To ensure quality software is developed using standard software engineering metric.
9 Safety Management:
Safety plan, Training plan, safety manual and safety case documents
To meet the requirements of safety standards, assessment authority and customers.

[Public]
Software safety engineering work estimate
• Pre-existing software (open-source and in-
house) which has good development process in
place can be taken for safety certification.
• There will approximately be 65 % of effort in
verification & validation tests and safety analysis.
• There will approximately be 35 % of effort in
documentation.
• Experience says pre-existing software with
approximately 50K LoC can be made for safety
compliance in 24 to 30 months time frame.
More tests and quality documentation are good for any software project.
65%
35%
ENGINEERING EFFORT
Testing Documentation

[Public]
Xen Hypervisor safety certification deliverables
S.No Deliverables Phase
1 Xen safety concept report
Completion of
safety concept
review phase
2 Xen safety certificate
Completion of final
assessment
3
Report to the certificate with high
details and results
Completion of final
assessment
4 Technical report
Completion of final
assessment
Assessment Deliverables Customer Package by AMD

[Public]
Xen Hypervisor safety certification - Periodic audits
• Assessment body may conduct re-audit after
the certification of the software product as per
the requirements of the assessment body
• The frequency of the audit shall be negotiated – E.g.
once in 12-24 months
• Impact analysis shall be performed in case of any
product updates
• Impact analysis shall be performed by the
engineering team and shall be reviewed by the
safety manager and assessor
• Based on the output of the impact analysis, the
need and scope of reassessment will be decided

[Public]
The Xen path to safety: Impact
on the Xen Community
• Xen entering environments where safety is a
requirement
• Proof that Xen is safety-certifiable paves the way
for others in the community to safety-certify Xen
• Concrete positive impact to the Xen Open Source
project:
• More robust coding guidelines with MISRA C
• Better documentation
• Much more testing

[Public]
Safety/Community Alignment: MISRA C
• MISRA C: coding guidelines for safe and secure C programming
• The goal is to improve the codebase -- MISRA C compliance is never at the expense of quality
• MISRA C adoption in OSS needs to be for the benefit for the Open Source project and its community
• Involve the community in evaluating MISRA C guidelines
• Select the rules that make sense for your project
• Adopt the rules as part of your project’s coding style
• Use MISRA C code checkers to automatically scan new patches for MISRA C violations
• Make code reviews easier
• Helps contributors write better code

[Public]
23
• MISRA C Tailoring completed: ~140 rules considered relevant for Xen
• MISRA C Rules adoption in progress ~38/140 rules
• Xen is already following many MISRA C rules, just not officially
→ Add Xen Rules we already follow to CODING_STYLE
• Evaluate the remaining MISRA C rules for adoption
• Deviations are sometimes required
• Alternative strategies to address the underlying issues identified by MISRA C are sometimes possible
• MISRA C Deviation strategy
• Deviations are intentional and documented exceptions to the rule
• Tag deviations with in-code comments so that MISRA C scanners will “ignore” them
• Document the reason for the deviation appropriately in documenting-violations.rst
• A tool is provided to convert the Xen deviation tag into the tool-specific tag
• we can support multiple tools with a single tag
Xen code with
Xen tag
Xen code with
cppcheck tag
/* SAF-1-safe R8.6 linker defined symbols */
extern char _start[], _end[], start[];

[Public]
▪ MISRA C checkers are fundamental for spotting violations in new contributions
▪ MISRA C checkers integration with Xen Project Gitlab CI/CD in-progress

[Public]
Safety/Community Alignment: Documentation
• Safety requires detailed documentation of interfaces, architectures, and expected behaviors
• Helps OSS users adopting the software
• Helps contributors find their ways around codebase
• Helps maintainers keep supported interfaces stable
• Dcumentation of Xen public interfaces in progress
• Hypercalls
• Boot configurations
• Virtual hardware
• Physical hardware
• Documentation of key interfaces already exists but might need improvement
• Use Doxygen / RST and keep the document under GIT revision next to the code
• Easier to maintain
• Easier to keep in sync with the code
• Use the same contribution and maintenance model as the Xen codebase
• Maintainers to require documentation updates when a Xen interface changes

[Public]
Safety/Community Alignment: Testing
• Safety requires all features and interfaces to be tested
• Testing directly improve the quality and reliability of OSS releases
• Better testing leads to fewer bugs
• Enhance and Extend the existing Gitlab CI/CD infrastructure with more tests
• 118 Gitlab tests including build tests and runtime tests
• Of the runtime tests 5 tests are running on real hardware, ARM & x86
• The remaining runtime tests are running on QEMU as emulating environment
• Community member can contribute both tests and also gitlab-runners to run real hardware tests
• To help the community
• To verify that Xen keeps running on their platforms of choice
• Require tests updates when interfaces change
• Proposal: require addition of new Gitlab-CI tests for any new Xen feature?

[Public]
Conclusions
• Safety certifiability of upstream Open Source software is possible -- Xen has an example
• It greatly helps if the upstream project has good quality processes already in place
• The Open Source community needs to get behind Safety as a goal for the project
• It is key to get community buy-in
• Safety is not about paperwork but actual safety of the codebase
• Codebase improvements, protecting against common errors
• Far better testing
• Clear and comprehensive documentation

[Public]

Safety-Certifying Open Source Software: The Case of the Xen Hypervisor

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Safety-Certifying Open Source Software: The Case of the Xen Hypervisor

Similar to Safety-Certifying Open Source Software: The Case of the Xen Hypervisor (20)

More from Stefano Stabellini

More from Stefano Stabellini (16)

Recently uploaded

Recently uploaded (20)

Safety-Certifying Open Source Software: The Case of the Xen Hypervisor