The document discusses the use of FreeBSD as a hosting platform, detailing the evolution of the hosting architecture at punkt.de, which shifted from a Nanobsd setup to a jail-based architecture for better customer isolation and manageability. Key challenges include availability, performance, and cost, and the presentation highlights improvements such as faster, automated updates and easier provisioning of jails versus virtual machines. The document also outlines the technical specifics of the architecture, including the use of ZFS and Ansible for managing jails and backups.

1. FreeBSD as a Hosting Platform – Revisited
Patrick M. Hausen
EuroBSDCon 2017, Paris

2. Agenda
• Introduction
• Challenges in Hosting
• Our Old NanoBSD Setup
• Why a Jail Based Architecture
• How We Do it Today
• What We Would Like to Do in the Future

3. About Me
• Working in IT since 1986
• Minix 1.1 since 1989
• FreeBSD since 1993
• In charge of network and data centre operations at punkt.de

4. About Our Team
• mOps – the Magnificent Operators
• 3 (originally) operators
• 1 (originally) developer

5. About punkt.de
• Founded in 1996
• Started as an ISP
• Today:Hosting and development of web applications
• Roughly 100 Servers
• RIPE Member
• DENIC Member
• 2 development, 1 operations team

6. Challenges
• Availability
• Performance
• Cost
• Manageability
• … Scalability – of the entire data centre!

7. „Updates?
Never change a
running system!“

8. „Nobody wants
backup – everyone
wants restore.“

9. Our First Attempt: nanobsd
root
root
(alternate slice)
/cfg
/var
(customer data)
m0s1a m0s2a m0s3a m0s3d
mirror/m0
ada0 ada1

10. Advantages
• OS and packages are read–only
• Atomic updates
• Rollback (with exceptions)
• Identical software across all servers

11. Drawbacks
• We did not go all the way – image creation remained manual
• Reboot of the entire machine required
• Installation of additional ports afterwards is difficult
• Too little flexibility – PHP, MySQL versions …
We did address some of these:
• Image creation is now vagrant up
• Packages come from our own poudriere

12. Goals for the New Architecture
• Better isolation of customers on the same machine
• Individual configurations per customer (PHP, MySQL, …)
• N instances per physical machine
• Faster more reliable updates
• Fully automated

13. All the Rage Now …

14. So, Why Not a Hypervisor?
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Machine MachineMachineMachine

15. So, Why Not a Hypervisor?
• Each VM is managed like a separate system
• Memory and/or storage overprovisioning at least difficult
• Which one to pick?
• Storage is either fast or cheap

17. What Do You Want to Virtualize Today?
Machine Hypervisor
/sbin/init Jail
Single Server
Process
Docker

18. So …
• Our customers want
• The semantics of a VM
• We as operators want
• Fast provisioning
• Easy updates
• Our CEO wants
• Low cost (within reason)

19. Advantage Jails
• Look like a VM to the customer
• Low overhead
• Don’t require a running server process they depend on
• Look just like regular processes from the outside host
• Local filesystem semantics!
• Now come with virtualized network stack!
• Networking is straightforward and simple (if you know your basics)

20. VIMAGE
• Introduces the epair(4) virtual interface
• Essentially a virtual patch cable
• One end inside the jail, other end on the host system
• Bridge, route, NAT to your heart’s content

21. System Architecture
LAN
Jail Jail Jail Jail
Bridge Interface
vnet0 vnet0 vnet0 vnet0
igb0, ix0, …
IPv6 and IPv4 (subject to change)

22. Shameless Marketing Plug

23. What the Customer Gets
• Virtual proServer:
One jail instance on a large host.
• Dedicated proServer:
Own host with as many jails as he desires or the host can
bear.
• But it’s all the same technology!
Which makes it way easier for us.

24. Virtual proServer Host
• All SSD based
• All ZFS
• 256 GB of RAM
• 2x 10 Cores / 20 Threads
• 50 customer jails and twiddling it’s thumbs

25. Jail Management Tools
• Ezjail
• Warden („old“ FreeNAS jails)
• py-iocage („new“ FreeNAS jails)
We picked iocage and actively contribute to the current rewrite.
https://github.com/iocage/iocage

26. Peculiarities …
Template
Jail
Instance
Instance
Instance
copy-on-write
clones
One does not simply update …
Snapshot!

27. Blueprint Jails
• Not iocage templates!
• Regular jails with FreeBSD-X.Y-RELEASE (11.1 as of now)
• Contain all the software we think is relevant for the customer
PHP-FPM, MySQL/MariaDB, Elastic, NginX and Apache, …
• Use our own poudriere as the repo for pkg
• Created and configured with Ansible
• Not running! (after initial creation)

28. Instance Jail
• Empty jail in iocage
• Blueprint jail mounted on top – read-only, nullfs
• All the read-write directories are separate ZFS datasets
• Mountpoints are set to legacy
• Mounted at jail startup by iocage’s fstab feature

29. Filesystem Layout
/etc /usr/local/etc /tmp /var …
/iocage/jails/2017Q3-php70-es1/root (blueprint)
/iocage/jails/vpro0042/root (empty)
nullfs
RO
RW
zfs

30. A Jail’s fstab
/iocage/jails/2017Q3-php70-es1/root /iocage/jails/vpro0042/root nullfs ro …
zdata/iocage/jails/vpro0042/root/etc /iocage/jails/vpro0042/root/etc zfs rw …
zdata/iocage/jails/vpro0042/root/home /iocage/jails/vpro0042/root/home zfs rw …
zdata/iocage/jails/vpro0042/root/tmp /iocage/jails/vpro0042/root/tmp zfs rw …
zdata/iocage/jails/vpro0042/root/var /iocage/jails/vpro0042/root/var zfs rw …
[…]
/iocage/jails/2017Q3-php70-es1/root/usr/local/etc/rc.d
/iocage/jails/vpro0042/root/usr/local/etc/rc.d nullfs ro …
/iocage/jails/2017Q3-php70-es1/root/var/db/pkg
/iocage/jails/vpro0042/root/var/db/pkg nullfs ro …
zroot/vpro0042/mysql /iocage/jails/vpro0042/root/var/db/mysql zfs rw …
zroot/vpro0042/mysql/logs /iocage/jails/vpro0042/root/var/db/mysql/logs zfs rw …

31. Provisioning
proServer Host PXE (unattended)
Blueprint Jail Ansible, Poudriere
Instance Jail Ansible

32. Making Updating Great Again
• chroot <blueprint> pkg upgrade
• Actually we don’t do that, although we could …
• Immutable infrastructure!
• We create a new blueprint jail
• Then update all the dependent instances to use the new one

33. Backups
• Easy – ZFS snapshots
• Hourly, Daily, …
• sysutils/zfstools
• Differential clones to central (per rack) backup server
https://github.com/adaugherity/zfs-backup
• We have a port – will need some polishing to be included in the tree

34. What Now?
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Storage?

35. Self–Provisioning
• Would be nice ;-)
• Essentially a complete private cloud solution
• But first: an API (REST possibly)
• Then a UI can be done by a frontend developer
Possibly this already exists …

36. Questions?
?

37. Thanks!