Dieser Vortrag zeigt, welchen Herausforderungen im Hosting punkt.de in der jüngeren Vergangenheit gegenüber stand und welche Änderungen gegenüber unserer 2010 vorgestellten "NanoBSD"-Architektur wir seitdem umgesetzt haben. Der proServer hat auf der Seite des Hosting-Anbieters viele der erwünschten Eigenschaften eines "Private Cloud"-Produkts, stellt sich dem Kunden aber wie ein klassischer Root-Server dar. Für Anwendungen, bei denen eine solche Plattform gefordert ist, ein unschätzbarer Vorteil gegenüber reinen Container-Lösungen, die praktisch immer eine speziell angepasste Anwendungs-Architektur benötigen.
Machine Learning Model Validation (Aijun Zhang 2024).pdf
FreeBSD hosting
1. FreeBSD as a Hosting Platform – Revisited
Patrick M. Hausen
EuroBSDCon 2017, Paris
2. Agenda
• Introduction
• Challenges in Hosting
• Our Old NanoBSD Setup
• Why a Jail Based Architecture
• How We Do it Today
• What We Would Like to Do in the Future
3. About Me
• Working in IT since 1986
• Minix 1.1 since 1989
• FreeBSD since 1993
• In charge of network and data centre operations at punkt.de
4. About Our Team
• mOps – the Magnificent Operators
• 3 (originally) operators
• 1 (originally) developer
5. About punkt.de
• Founded in 1996
• Started as an ISP
• Today:Hosting and development of web applications
• Roughly 100 Servers
• RIPE Member
• DENIC Member
• 2 development, 1 operations team
10. Advantages
• OS and packages are read–only
• Atomic updates
• Rollback (with exceptions)
• Identical software across all servers
11. Drawbacks
• We did not go all the way – image creation remained manual
• Reboot of the entire machine required
• Installation of additional ports afterwards is difficult
• Too little flexibility – PHP, MySQL versions …
We did address some of these:
• Image creation is now vagrant up
• Packages come from our own poudriere
12. Goals for the New Architecture
• Better isolation of customers on the same machine
• Individual configurations per customer (PHP, MySQL, …)
• N instances per physical machine
• Faster more reliable updates
• Fully automated
14. So, Why Not a Hypervisor?
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Machine MachineMachineMachine
15. So, Why Not a Hypervisor?
• Each VM is managed like a separate system
• Memory and/or storage overprovisioning at least difficult
• Which one to pick?
• Storage is either fast or cheap
17. What Do You Want to Virtualize Today?
Machine Hypervisor
/sbin/init Jail
Single Server
Process
Docker
18. So …
• Our customers want
• The semantics of a VM
• We as operators want
• Fast provisioning
• Easy updates
• Our CEO wants
• Low cost (within reason)
19. Advantage Jails
• Look like a VM to the customer
• Low overhead
• Don’t require a running server process they depend on
• Look just like regular processes from the outside host
• Local filesystem semantics!
• Now come with virtualized network stack!
• Networking is straightforward and simple (if you know your basics)
20. VIMAGE
• Introduces the epair(4) virtual interface
• Essentially a virtual patch cable
• One end inside the jail, other end on the host system
• Bridge, route, NAT to your heart’s content
23. What the Customer Gets
• Virtual proServer:
One jail instance on a large host.
• Dedicated proServer:
Own host with as many jails as he desires or the host can
bear.
• But it’s all the same technology!
Which makes it way easier for us.
24. Virtual proServer Host
• All SSD based
• All ZFS
• 256 GB of RAM
• 2x 10 Cores / 20 Threads
• 50 customer jails and twiddling it’s thumbs
25. Jail Management Tools
• Ezjail
• Warden („old“ FreeNAS jails)
• py-iocage („new“ FreeNAS jails)
We picked iocage and actively contribute to the current rewrite.
https://github.com/iocage/iocage
27. Blueprint Jails
• Not iocage templates!
• Regular jails with FreeBSD-X.Y-RELEASE (11.1 as of now)
• Contain all the software we think is relevant for the customer
PHP-FPM, MySQL/MariaDB, Elastic, NginX and Apache, …
• Use our own poudriere as the repo for pkg
• Created and configured with Ansible
• Not running! (after initial creation)
28. Instance Jail
• Empty jail in iocage
• Blueprint jail mounted on top – read-only, nullfs
• All the read-write directories are separate ZFS datasets
• Mountpoints are set to legacy
• Mounted at jail startup by iocage’s fstab feature
32. Making Updating Great Again
• chroot <blueprint> pkg upgrade
• Actually we don’t do that, although we could …
• Immutable infrastructure!
• We create a new blueprint jail
• Then update all the dependent instances to use the new one
33. Backups
• Easy – ZFS snapshots
• Hourly, Daily, …
• sysutils/zfstools
• Differential clones to central (per rack) backup server
https://github.com/adaugherity/zfs-backup
• We have a port – will need some polishing to be included in the tree
35. Self–Provisioning
• Would be nice ;-)
• Essentially a complete private cloud solution
• But first: an API (REST possibly)
• Then a UI can be done by a frontend developer
Possibly this already exists …