The Xen Project is used by more than 10 million users, powers some of the largest clouds on the planet, and is starting to build momentum in embedded and safety-conscious market segments. It is also nearly 16 years old. The Xen Project’s success and longevity can be attributed to its flexible architecture, but more importantly to enabling community members to contribute ideas and code, even if they are not core to the project's main use-case. This has brought Xen far beyond server virtualization. Lars will share how the project has supported new technologies and ideas, which may include some really interesting things you might not know about Xen (especially around defense applications), and will derive best practices that may help other projects.

1. Thinking Outside of the Conceived
Tech Comfort Zone
Xen Project 15 Years
Down the Line
Presentation at
www.slideshare.net/xen_com_mgr

2. Lars Kurth
Community Manger, Xen Project
Chairman, Xen Project Advisory Board
Director, Open Source, Citrix
Contributors
Andrew Cooper: Committer, Xen Project ◉ Christopher Clark: Contributor, Xen Project
Mihai Donțu: Chief Linux Officer, Bitdefender ◉ George Dunlap: Committer, Xen Project
lars_kurth

3. What is Xen and Xen Project?
Versatile Virtualization Platform
Designed to be a component in a SW stack
Ease of use for end-users was never a design goal
Xen Hypervisor = “Engine”
Taken by integrators to build a product, service, …
Analogy: integrators build a “car” with Xen as “engine”
Xen Project
Development community with several sub projects
that develop technologies related to Xen
– Hypervisor
– PV Drivers
– Unikernel related projects: MirageOS, Unikraft

4. Rewind to the Beginning

5. Original Use Cases
Server Consolidation
Co-located hosting facilities,
Distributed web services
Cloud Computing
Secure Computing Platforms and
Application Mobility
High availability, Fault tolerance,
Checkpointing

6. Original Design Goals
Modular architecture
Via disaggregation
Flexible architecture
Via use of system services
High degree of customisability
Initial goal: server and possibility of
desktop products

7. OpenXT, SecureView
(desktop, laptops, tablets)
Defense Applications
Unikernels
Cyberchaff (HalVM)
Defense Applications
Xenon Hypervisor family, Magrana Server, …
Cloud Computing Amazon Web Services, Tencent, Alibaba Cloud, Oracle Cloud,
IBM SoftLayer, Rackspace, …
Server Virtualization
Linux Distros, Citrix Hypervisor, Huawei UVP, Oracle VM, XCP-ng
ARLX/Virtuosity OA, Bromium
uXen, Crucible Hypervisor
Embedded Defense /
Security Applications
Embedded/
Automotive Virtuosity, XILINX Xen Zynq, Perseus,
GlobalLogic Nautilus, EPAM Fusion
General purpose desktop and mobile Virtualization
XenClient, NxTop, Neosphere, Samsung, Qubes OS
VMI Bitdefender,
AIS, Zentific

8. So, how did we get to such a
diverse product eco-system?

9. Strong Security
mindset by main
Xen Developers

10. From the Beginning
Disaggregation was part of the original Xen conception
Influenced by microkernel thinking
VMn
Guest OS
Applications
VM0 (or Dom0)
Dom0 Kernel
Native Driver
System Services
XS TSDE
Scheduler MMU Timers Interrupts

11. From the Beginning
Disaggregation was part of the original Xen conception
Influenced by microkernel thinking
VMn
Guest OS
Applications
VM0 (or Dom0)
Dom0 Kernel
System Services
XS TS
Scheduler MMU Timers Interrupts
Driver Domain
Guest OS
Native Driver
Service Domain
Guest OS
System Services
DE
Disaggregate
Disaggregate

12. From the Beginning
Disaggregation was part of the original Xen conception
Influenced by microkernel thinking
The PV protocols were specifically designed for untrusted
backends and untrusted native device drivers
Particularly the grant tables
Security technologies were always in maintainers minds
They were up-streamed without resistance
Sometimes with a lot of support from maintainers

13. Today: Disaggregation used by …
OpenXT, SecureView
(desktop, laptops, tablets)
Defense Applications
Unikernels
Defense Applications
Xenon Hypervisor family, Magrana Server, …
Cloud Computing
Server Virtualization
ARLX/Virtuosity OA,
Crucible Hypervisor
Embedded Defense /
Security Applications
Automotive Virtuosity,
GlobalLogic Nautilus, EPAM Fusion
General purpose desktop and mobile Virtualization
Qubes OS
VMI

14. Tension: Core vs. non-Core Usage
Defense Applications
Unikernels
Defense Applications
Cloud Computing
Server Virtualization
Embedded Defense /
Security Applications
Automotive
General purpose desktop and mobile Virtualization
VMI
Until 2017, all key maintainers were working for
companies working on Server Virtualization, the core use-case
Today 30% of key maintainers
work for vendors that care about
embedded and mixed-criticality
use-cases bringing in new
innovations

15. Tension: Core vs. non-Core Usage
Many security technologies were not made easy to use
Disaggregation is a good example
Product schedule pressure, led to
– Core team members choosing the quick to implement / more performant
approach ➜ New features went into Dom0 first
– Maintainers reviewing non-Core contributions: takes effort to understand
and guide contributors to the best approach ➜ This did often not happen
E.g. Xen has no build capability for disaggregated domains
Led to duplication amongst downstreams
Increased the barrier to adoption for newcomers

16. Strong Security Mindset
Enabled non-Core Security
Contributions

17. From Military Clouds to
Desktop/Tablet Use Cases on x86

18. Xen Summit: 2007, New York, USA

19. Xen Summit: 2007, New York, USA
US DoD employees presented approaches to
Apply EAL5 Common Criteria Security Clearance for Xen
Xen Security Modules (XSM)
EAL5 CC related research led to the Xenon Separation VMM family
Today: widely used by DoD departments
XSM was subsequently upstreamed
Defense Applications
Xenon Hypervisor family, Magrana Server, …
Server Virtualization
Linux Distros, Citrix Hypervisor, Huawei UVP, Oracle VM, XCP-ng

20. 2012: The Xenon Separation VMM
Secure virtualization infrastructure for military clouds (pdf)
“ We reduce the size of the code base by dropping support
for selected features that are not needed to run commodity
operating systems, e.g. Windows 7 or Linux. Because the
internal design and construction of the base Xen VMM is
highly modular, removing some of the features does not
disable any of the remaining features.
The total effort required was about 3 engineers over a
period of 2 years, and includes the effort of keeping up with
the Xen community. ”

21. x86: Desktop / Tablet Use Cases
2009: Virtual Computer releases NxTop 1.0
NxTop is the first of 3 Xen based client solutions
Citrix and Neocleus release their first solutions in 2010
2012: Snowden-approved Qubes OS 1.0 releases
Today, Qubes OS 4.0 is the first technology to use PVHv2 in production
2013: Citrix and DoD collaborate to create SecureView
In 2015, XenClient is discontinued and released as OpenXT
SecureView continues to be developed on top of OpenXT
OpenXT, SecureView
(desktop, laptops, tablets)
Defense Applications
General purpose desktop Virtualization
XenClient, NxTop, Neosphere, Qubes OS

22. Reflection
Early & continued engagement of Security Researchers
Security feature contributions before being productized
Xen became an attractive platform for security research
Enabler:
The Xen community accepted contributions
Modular Architecture
Enabled specialised and certifiable products for defence use
But: Innovations were not attempted to be up-streamed

23. Reflection
Attempts to commercialize Xen on Desktop
Enabled a new products leveraging Xen’s security features
They also relied on removing upstream Xen code (on x86)
BUT:
None of these Xen capabilities were used in Server/Cloud
Related Story: GPU Passthrough & Virtualization
Were developed for Desktop Use Cases on Xen
TODAY:
GPU Virtualization is widely used in Server/Cloud

24. Virtual Machine Introspection:
from Research to Commercial
Server Products

25. Georgia Tech: 2007, Atlanta, USA

26. Georgia Tech: 2007, Atlanta, USA
2007: Virtual Machine Introspection for Xen hatched at
Georgia Tech
– 2003: Initial Research by Tal Garfinkel & Mendel Rosenblum
– Bryan Payne created a project called XenAccess
Later expanded scope to other hypervisors as LibVMI
2009: XenAccess and mem-events APIs were added
Used for security research and specialist security apps
Enabler:
The project accepted these changes with active help
from maintainers

27. Commercial interest in VMI
2013 to 2015: Intel launched its Haswell & Broadwell CPUs
Specifically VMFUNC, EPTP and #VE capabilities
New HW capabilities of Haswell & Broadwell generated
commercial interest in VMI by multiple vendors
2014 to 2015: The XenAccess and mem-events APIs were
re-architected into the VMI subsystem
In parallel: related companion technologies such as alt2pm
are being developed

28. Productization
2015 to 2016: Citrix and Bitdefender collaborated to bring
VMI to market through XenServer 7.0 and Bidefender HVI
Today: Similar products are available from AIS and Zentific
Today: ongoing contributions by security vendors to alt2pm
and #VE capabilities indicate future productization

29. Reflection
Few security features made it into Server/Cloud
For example: VMI & Live Patching
WHY?
HW capabilities were not enough
It was too early or the market wasn’t ready
Example: Dynamic Root of Trust related technologies
Complex and Use Case dependent
Example: Xen Security Modules - requires Specialist expertise;
configuration is Use Case dependent

30. From Desktop / Mobile
Use Cases to
Embedded / Automotive

31. 2008, Seoul, Korea

32. Samsung releases Secure Xen on Arm port
This Xen port used paravirtualization and was kept as a Xen
fork hosted by the Xen Project as requested by Samsung
2008 to 2014: Samsung regularly shows Xen demos
Dual mobile operating systems running on Samsung
devices and later Nexus 10
Demos and presentations covered
– Innovations in real-time capabilities based on Xen
– GPU sharing technologies based on Xen
2008, Seoul, Korea

33. This showed what is possible
Inspiring others to investigate and
research along similar lines

34. 2012, Cambridge, UK

35. 2012, Cambridge, UK
Xen on Arm reboot
Patches for upstream Xen on Arm support are posted
Armv7 and v8 are fully supported upstream by March 2014
The primary motivation for this work was to target Arm
based servers
Developers learned from the earlier Arm port:
– Clean and simple architecture
– Perfect match between Xen and Arm ISA
– Much smaller code size compared to Xen on x86

36. Upstream support and simplicity
of Arm port enabled embedded
Use Cases on Arm

37. The missing Evolutionary Link
From x86 to Arm
2010: Dornerworks contributes the ARINC653 Scheduler
Launches ARLX (initially for x86) taking a similar approach
to Xenon
2014: Dornerworks pioneers safety certification for Xen
OpenXT, SecureView
(desktop, laptops, tablets)
Defense Applications
General purpose desktop Virtualization
XenClient, NxTop, Neosphere, Qubes OS
ARLX/Virtuosity OA,
Crucible Hypervisor
Embedded Defense /
Security Applications

38. A new Evolutionary Wave
2014: Xen moves into embedded and automotive
The Xen Project launched an automotive initiative
Xen based distros and products are being developed
2015: First automotive Xen based platform
GlobalLogic introduces the Nautilus platform at CES 2015
2015: First embedded Arm-only Xen Distro
Xen Zynq Distribution for XILINX
The last few years: R&D
Similar order of magnitude to earlier waves

39. Sedimentation: from Software
to Hardware

40. Sedimentation:
Functionality implemented in Software being fully or partly
implemented in Hardware
Example: PV Interrupt and Timers ➜ using IO APIC and
Posted Interrupts
Silicon vendors have always worked closely with Xen
Xen’s had a significant impact on virtualization
related technology in Hardware
I wanted to tell this story, but:
It’s very difficult to explain
And much of the work was performed under NDAs

41. What have we learned?
What can you learn from Xen?

42. Culture enables Innovation
Accept changes for non-Core Use Cases
In our case this was not planned: we inherited the culture
But: do not sacrifice due diligence and future-proofing
Help people trying to adopt Xen for non-Core Use Cases
We were bad at this until about 3 years ago
Work with new ideas, forks and distributions
Developer Events: allow air-time for new ideas
Forks are not always bad: Samsung fork hosted by us
OpenXT (distro): close collaboration is key

43. Process enables Innovation
Feature/Configuration Classification
Experimental ➜ Tech Preview ➜ Supported
Marked non-Core functionality as Experimental or Preview
Maintainership
Contributors need to step up for larger non-Core Features
Proves longer term commitment by the contributor
Deprecation of Features
Intent to deprecate is announced on list: identify objections
Remove non-core Features, if stale or not used

44. Tension need to be Managed
Vendors have different Priorities
These can lead to conflicts
Non-Core vs. Core arguments can be ugly
Active community management is required
New Processes can lead to Tensions
Security Vulnerability Management led to problems
Who deals with a Vulnerability in non-Core code?
We didn’t understand this ➜ created tension and rejection
of non-Core features until we modified the process

45. Process enables Innovation
Security Support
The security team does not handle non-Core Vulnerabilities
– Supported Feature without Security Support
– Delegated Security Support (e.g. ARINC653 scheduler)
KCONFIG based Configuration Management
Larger non-core features are build or run-time disabled
Subprojects as incubators for New Ideas
Can be used to foster innovation on the periphery
Examples: MirageOS, Automotive Project, Unikraft

46. lars.kurth@xenproject.org
Picture by Lars Kurth

47. What is next?
Challenges we are facing today

48. Safety Certification
Driven by interest in embedded and automotive usage
So far the focus was on features and code size
BUT: vendors are actively looking for help from the project
to make building safety certified products sustainable on
top of Xen Project
2018: Experiments to identify impact on community
Experiments with MISRA C compliance ➜
Challenging because changes often lead to an emotional
response with far too much debate ➜ Change of
approach is needed

49. Safety Certification: Tensions^2
We may need to make significant changes
To coding standards: e.g. MISRA C
To existing processes: more requirements, design, etc.
documents ➜ enforce existing process, but vigorously
New processes and tools: not yet clear
Discussions suffer from lack of knowledge: what is really
needed?
High stakes: Risks if no workable solution is found
Vendors could go elsewhere
A fork of the project

50. Our Approach for now
Bring community leaders and “experts together”
Establish understanding of the needs of each side
Dispel myths: there are many
Identify gaps and what would need to change
Establish a change process and keep up momentum
Be prepared to adapt/fail/abandon
It may not be possible to find a compromise that works
for all stake-holders