My talk from Embedded World 2024 on Immutable Operating Systems.

1. Immutable Image-Based
Operating Systems
Presented by
Drew Moseley
Technical Solutions Architect
Toradex

2. WHAT WE’LL
COVER TODAY…
• Definitions
• Architecture
• Benefits
• Desktop Distro
• Embedded OS Architecture
• Demo(?)
AGENDA

3. WHAT WE DO
RELIABLEAND EASY-TO-USE EMBEDDED
SOLUTIONS FOR YOU
Arm® System on Modules
Reliable
Long-Term Maintenance
Scalable
From Stock
Production-Ready Software
Yocto-Based Linux
Windows Embedded Compact
Development Tools
Long-Term Maintenance
Ease-of-Use
Support
Ecosystem

4. Definitions
• Immutable1: not capable of or susceptibleto change
› Critical portions of the system are "read-only"
› Updates are performed with only well-defined mechanisms
› User data stored separately
› Applications generally use a different mechanism
1
https://www.merriam-webster.com/dictionary/immutable
2
https://www.merriam-webster.com/dictionary/image
• Image2: exact likeness
› Updating the entire "Operating System"
› Updating individual packages or applications "not supported"
3
https://www.merriam-webster.com/dictionary/atomic
• Atomic3: of, relating to, or concerned with atoms
› Incapable of being subdivided
› No chance of partially installed updates
Other names: Layered OS, Reprovisionable, Anti-hysteresis

5. Sidebar: Pets vs Cattle
• Coined by Randy Bias1
› Originally from Enterprise Computing
Space
• Desktop/Server:
› Pets - Individual laptops
› Cattle - Servers managed as code
• In Embedded:
› Pets - Weekend projects,
home automation
› Cattle - Large fleets of identical devices.
1 http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/

6. Conceptual Architecture
System
Operating System
(Image v1)
Bootloader
Kernel/DTB/Initramfs
"OS" Packages
User Data
User Applications

7. Conceptual Architecture
System
Operating System
(Image v2)
Bootloader
Kernel/DTB/Initramfs
"OS" Packages
User Data
User Applications
Operating System
(Image v2)
Bootloader
Kernel/DTB/Initramfs
"OS" Packages

8. Benefits
• Atomic versioning and updates of critical system components
› No more `apt --fix-missing --install` or related commands
• User components separately managed
› Better isolation of dependencies (ie containers)
› Fewer conflicts based on OS installed package versions
• Reproducibility
› The OS image is deterministic
› No configuration drift
• Better testing
› Exactly matching software on test and productiondevices
• Rollback capability
• More secure? Arguable

9. Drawbacks
• New/unfamiliar workflows
• Less flexible than traditional distros
• Do all your applications run in the
sandbox?
• Reboot required for any updates
› Mitigated by the app packaging system
• Is it really appropriate for desktop/laptop
use?

10. Technologies and Concepts
• libostree (https://ostreedev.github.io/ostree/)
› "Git for filesystems"
› Content-addressable objectstorage + hard links
• Multiple partitions
› Usually mounted read-only
› Symlinks for mutable config files
• Btrfs snapshots
• Declarative configuration
• Layering: https://coreos.github.io/rpm-ostree/

11. WHAT IS libostree?
"libostree is both a shared library and suite of command line tools
that combines a “git-like” model for committing and downloading
bootable filesystem trees, along with a layer for deploying them and
managing the bootloader configuration." 1
"git-like"
model
bootable
filesystem
trees
Bootloader
configuratio
n
1 https://github.com/ostreedev/ostree#libostree

12. OSTree BASICS
• File-based (!)
• Relies on non-root mount/“bind-mount”
- Normally the root of a file system is mounted as “the root”
- Linux allows to bind mount a subdirectory
• Initramfs mounts OSTree
- Pivot into bind mount/sub-directory
• Hardlinks are used to speed-up deployment and
minimize space usage
Source: https://medium.com/@1154_75881/what-is-the-difference-between-a-hard-link-and-a-symbolic-link-14db61df7707

13. Libostree filesystem layout
(Simplified)

14. "File system based on the copy-on-write principle
using B-trees, developed at Oracle since 2007"1
• Declared stable in Linux in 2013
• Subvolumes
• Atomically writable snapshots
• Cloning (multiple inodes pointing to the same
disk blocks)
BTRFS Snapshots
1
https://en.wikipedia.org/wiki/Btrfs

15. Applications
Containers: https://www.docker.com/ or https://podman.io/
Flatpak: https://www.flatpak.org/
Appimage: https://appimage.org/
Snaps: https://snapcraft.io/
Bundled with dependencies
"Distro-independent" Linux packages
Sandboxed from the host OS and other packages

17. Torizon Demo

18. Universal Blue
Based on Fedora Silverblue
"Cloud Native Linux Desktop Model"
• Base images generated by OCI containers
o RPM-OSTree
o BTRFS (snapshots?)
o Applications normally use Flatpak
Distrobox (https://distrobox.it/)
Linuxbrew (https://docs.brew.sh/Homebrew-on-Linux)
Many variants:
• Bluefin: GNOME Desktop
• Bluefin-DX: Bluefin + Cloud developer tools
• Built-in GPU drivers

19. Universal Blue Demo

20. NixOS
Reproducible
Declarative
Reliable
Package Manager
or
Full blown OS

21. VanillaOS
• Ubuntu Desktop based
• Dual A-B partitions

22. Survey of available systems
Desktop/Server
• Debian: Endless OS
• Ubuntu: VanillaOS
• Fedora: Silverblue
• Universal Blue
• NixOS
• GNU Guix
• Clear Linux
• Fedora CoreOS
• openSUSE Aeon (Gnome)
• openSUSE Kalpa (KDE)
• Flatcar Linux
• Bottlerocket OS
• Talos Linux (k8s)
• ChromeOS
Embedded
• Torizon
• Ubuntu Core
• Linux microPlatform
• BalenaOS
• SteamOS

23. References
• https://github.com/castrojo/awesome-immutable
• https://discord.gg/N4mswFw6ds
• https://blog.verbum.org/2020/08/22/immutable-%E2%86%92-reprovisionable-anti-hysteresis/
• https://www.torizon.io/
• https://www.torizon.io/open-source-community
• https://universal-blue.org/

24. THANK YOU
FOR YOUR INTEREST
www.toradex.com | www.torizon.io | developer.toradex.com
community.toradex.com | labs.toradex.com