Ceph claims to be a perfect unified file system for almost every purpose. In theory this can be true but in reality there are of course limitations. Following our way from Firefly (2014) to Luminous (2018) I will show our use cases, the shortcomings and the pitfalls we fell into! This will show Ceph in use for typical cloud and container stacks, simple media archives for applications and the devilish high available shared file systems.

1. STORAGE WARS
OSDC 2019

2. ACHIM
LEDERMÜLLER
WHOAMI
SYSTEMS ENGINEER
@READONLY444
HOSTING
INFRASTRUCTURE

3. MANAGED SERVICES

4. STORAGE
HOSTING

5. INTRODUCTION
TIM AL
🤑

6. WINTER 2012/2013
Are my VMs always
accessible?
Yes for sure 🧐
Hi
AL
TIM
🤔

7. WINTER 2012/2013
Is my data save if the the
disk array burns down?
I don’t think so, Tim.
😡
…but if we double the
price for your servers we can
buy some new licences 🤑 AL
TIM
🤔

8. 2013
Can’t you provide some
open source solution?
😊
Maybe we should try
GlusterFS!
TIM
🤔
AL

9. 2013
Can’t you provide some
open source solution?
😊
Maybe we should try
GlusterFS!
TIM
🤔
AL
10x 2 TB 10x 2 TB
10x 2 TB 10x 2 TB
10x 2 TB 10x 2 TB

10. 2013
🤯🤕🤢🤮
Can’t you provide some
open source solution?
😊
Maybe we should try
GlusterFS!
TIM
🤔
AL
10x 2 TB 10x 2 TB
10x 2 TB 10x 2 TB
10x 2 TB 10x 2 TB

11. DO TESTS
AT LEAST
WITH 80%
OF YOUR
MAXIMUM
CAPACITY

12. WINTER 2013/2014
🤬💩
Let’s try Ceph!
😕
Why? 😐
AL
TIM
😡

13. WINTER 2013/2014
🤬💩
Let’s try Ceph!
🐙🦑😍
😕
Why? 😐
AL
TIM
😡

14. Client
12MB
4 MB Slices
Object Storage Daemons

15. Client
12MB
4 MB Slices
Object Storage Daemons

16. Client
12MB
4 MB Slices
Object Storage Daemons

17. Client
12MB
4 MB Slices
Object Storage Daemons

18. Client
12MB
4 MB Slices
Object Storage Daemons

20. SUMMER 2014
Is my data save if your
dc burns down?
Ceph is replicated to an
other DC by default. 😏
AL
TIM
🤔

22. CRUSH

23. SUMMER 2014
… and if I delete
accidentally all my
volumes?
🥺😱 😒
Is my data safe if your
dc burns down?
Ceph is replicated to an
other DC by default. 😏
AL
TIM
🤔

24. FALL 2014
Why is my service slow?
No problem!
Every night for two minuts!
AL
TIM
🤔
😭😱

25. FALL 2014
Why is my service slow?
No problem!
Every night for two minuts!
I can fix this 😙
AL
TIM
🤔
😭😱
😎

26. Snapshots

27. Scrubbing

28. Backfilling

29. LIMIT
INTENSIVE
JOBS

30. LIMIT
INTENSIVE
JOBS
&
CLIENTS

31. TIM
WINTER 2014
TIM
Why is my mysql-slave
so far behind?
Due to redundancy of your data the latency of all components sums up. At least we have the time
going through the network to the primary OSD and the latency to slowest replica and the latency of the
Ceph code itself (sum of primary and slowest replica) and the latency of the spinning disks inclusive. At
this point we still ignore competing components like self healing and …Lorem ipsum dolor sit amet,
consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna
aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita
kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet,
consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna
aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita
😖 fix it
AL
😟

32. LATENCY
Client
OSD
OSD OSD

33. LESS DENSITY
2TB 2TB2TB 2TB2TB 2TB 2TB 2TB 2TB 2TB
1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB
1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB

34. ALL FLASH
SSD
SSD SSD
SSD SSD
SSD SSD
SSD
SSD
SSD SSD
SSD SSD
SSD SSD
SSD
SSD
SSD SSD
SSD SSD
SSD SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD
SSD SSD SSD SSD SSD SSD

35. WHAT
YOU
BUY
IS
WHAT
YOU
GET

36. ONCE A WHILE
TIM
We need additional 50 TB
When?
We already wrote
20 TB since last week.
😭
AL
😋

37. SET
SUITABLE
THRESHOLDS

38. OSDC 2018
Be honest, what have you
messed up already?
Nothing
🤥
🤨
🤫
Ceph covers our back!
AL
TIM
😁

39. USE
THE
DEFAULTS
&
CARE

40. THANKS!
AL

41. @netways
/netways
netways
www.netways.de
blog.netways.de
nws.netways.de