Romana, the open source project by Pani Networks, brings stunning simplicity to the usually so complex networking in OpenStack and Kubernetes. Using only native L3 routing and no overlays, along with automated distributed application of network policies and security rules, it provides operators with easy to understand and manage networking, while allowing network hardware to operate at its best and with full efficiency.
These slides were used during the OpenStack meetup in Auckland in May 2016, hosted by Catalyst IT.
2. romana.io Simplifying the network stack with Romana @romanaproject
Agenda
● “Cloud native”, why does it matter?
● A better network for cloud native architectures
● Demos
3. romana.io Simplifying the network stack with Romana @romanaproject
About us
● Team background:
– Data center networks
– Low-level traffic management
● Created L2 overlay network startup
– Bought by Cisco
● OpenStack networking
● There's got to be a better way
– Time is right
5. romana.io Simplifying the network stack with Romana @romanaproject
The past: Enterprise networking
● Full control
● Applications need L2 and L3
– May need hard-wired IP addresses
– Broadcasts
● Servers are pets, not cattle: “Careful!”
– VM migration
● Complex!
6. romana.io Simplifying the network stack with Romana @romanaproject
Cloud native applications
● Automate all the things!
– Infrastructure as code
– Cattle, not pets: “Meh... just kill it.”
– Workloads come and go quickly
– Build for resiliance
● IP is all you need
– No hardcoded IP addresses, discovery
– No special network requirements
– Basic IP connectivity
8. romana.io Simplifying the network stack with Romana @romanaproject
We have a mismatch
● Building cloud native applications…
● … on top of enterprise networking
– SDN controllers use overlay L2 domains
– VLAN, VXLAN, OVS, etc.
● Complexity and brittleness
– Lose benefits of simplicity
– Lose performance (encap, blinded hardware)
– Difficult to maintain and trouble shoot
9. romana.io Simplifying the network stack with Romana @romanaproject
The price you pay: Complexity
VXLAN Decap
VXLAN Decap
VXLAN Encap
VXLAN Encap
2 Top of Rack Round
Trips
East/West Traffic
Per Instance Security
10. romana.io Simplifying the network stack with Romana @romanaproject
The price you pay: Performance
Router
Endpoint A Endpoint B
Router
L2 overlay A
L2 overlay B
VRouter
11. romana.io Simplifying the network stack with Romana @romanaproject
Why do we do this to ourselves?
● We don't need any L2 features
● Except maybe traffic segmentation
– Multi tenancy
– Tiers and policies
13. romana.io Simplifying the network stack with Romana @romanaproject
Networking the way it was intended
● Use native L3 capabilities
● No overlays
● De-emphasize IP address ranges
● Still provide segmentation, multi tenancy
● Simple, clear and scalable network setup
14. romana.io Simplifying the network stack with Romana @romanaproject
Truly cloud native networking
● Project Romana
● Open source
● Apache 2.0 license
● Mostly written in Go
● Kubernetes and OpenStack
15. romana.io Simplifying the network stack with Romana @romanaproject
Truly cloud native networking
● Use only IP routing
– No overlays
– All workload addresses are 'real'
– Simplicity!
● Use smart addressing
– Encode tenant or segment in IP address
– Assign “virtual” addresses with host prefixes
– Massive (!) collapse of route table
● Routes are static
– No route updates, no broadcasts for new endpoint
16. romana.io Simplifying the network stack with Romana @romanaproject
Romana Architecture
● On each host: Agent
– Configures routes
– Connects endpoint interfaces
– Sets policy implementations
●
Controller: Cooperating microservices
– Each service with RESTful interface
– Specialized for different tasks
● Environment: Different integration points
– APIs, drivers for various parts of OpenStack or
Kubernetes
17. romana.io Simplifying the network stack with Romana @romanaproject
Romana Architecture
Host A Host B Host C
Agent Agent Agent
Tenant
Topology
IPAM
Root
Environment (OpenStack or Kubernetes)
Policy
19. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
Host B
eth0:
192.168.8.22
Host C
eth0:
192.168.8.33
20. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
21. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
22. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
23. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
24. romana.io Simplifying the network stack with Romana @romanaproject
Routing and route aggregation
Host A
eth0:
192.168.8.11
romana-gw:
10.0.0.1/16
10.0.0.5
10.0.1.7
10.0.1.19
10.0.5.3
Routes:
10.1/16 → 192.168.8.22
10.2/16 → 192.168.8.33
Host B
eth0:
192.168.8.22
romana-gw:
10.1.0.1/16
10.1.3.52
10.1.9.2
Routes:
10.0/16 → 192.168.8.11
10.2/16 → 192.168.8.33
Host C
eth0:
192.168.8.33
romana-gw:
10.2.0.1/16
10.2.0.16
10.2.3.81
10.2.4.6
Routes:
10.0/16 → 192.168.8.11
10.1/16 → 192.168.8.22
25. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
Rack A Rack B
26. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
27. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
Host A2 Routes
0.0.0.0 192.168.1.200→
10.68/14 192.168.1.1→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→
28. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: L2 under ToR
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
ToR A Routes
10.128/10 192.168.2.200→
10.68/14 192.168.1.1→
10.72/14 192.168.1.2→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→
Host A2 Routes
0.0.0.0 192.168.1.200→
10.68/14 192.168.1.1→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→
29. romana.io Simplifying the network stack with Romana @romanaproject
Larger network: Full L3
Host B1
Host B2
Host B3
Host B4
Host A1
ToR A ToR B
spine network
192.168.1.200 192.168.2.200
192.168.1.1
Host A2
192.168.1.2
Host A3
192.168.1.3
Host A4
192.168.1.4
10.68/14
10.72/14
10.76/14
10.80/14
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
10.132/14
10.136/14
10.140/14
10.144/14
Rack A Rack B
10.64/10 10.128/10
ToR A Routes
10.128/10 192.168.2.200→
10.68/14 192.168.1.1→
10.72/14 192.168.1.2→
10.76/14 192.168.1.3→
10.80/14 192.168.1.4→
Host Routes
0.0.0.0 192.168.1.200→
31. romana.io Simplifying the network stack with Romana @romanaproject
Romana: Traffic segmentation
● Tenant traffic separated:
– Tenants don't get whole CIDR prefix or L2 domain
– But fully isolated from other tenants' traffic
● Tenants can define segments:
– Like tiers, provide isolation and policies
● Use segment and tenant bits in IP addresses:
– Apply policies (iptables) based on that
– Segments can stretch across hosts
32. romana.io Simplifying the network stack with Romana @romanaproject
Semantic and topological addressing
3
1
3
0
2
9
2
8
2
7
2
6
2
5
2
4
2
3
2
2
2
1
2
0
1
9
1
8
1
7
1
6
1
5
1
4
1
3
1
2
1
1
1
0
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
The network prefix.
In this example, we
are using the 10/8
address space.
6
Host ID Segment ID
We currently
store tenant ID in
upper bits of
segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.
33. romana.io Simplifying the network stack with Romana @romanaproject
Semantic and topological addressing
3
1
3
0
2
9
2
8
2
7
2
6
2
5
2
4
2
3
2
2
2
1
2
0
1
9
1
8
1
7
1
6
1
5
1
4
1
3
1
2
1
1
1
0
9 8 7 6 5 4 3 2 1 0
0 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1
10
Network prefix bits
The network prefix.
In this example, we
are using the 10/8
address space.
6
Host ID Segment ID
We currently
store tenant ID in
upper bits of
segment ID.
4 67
Endpoint ID
Widths are configurable, don't have to use byte boundaries.
Encode the
tenant ID
34. romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Allowing traffic within tenant
10.0.0.5 10.1.0.12
iptables:
check src/dst addrs
“tenant/segment bits
must match”
Src: 10.0.0.5
Dst: 10.1.0.12
Same
tenant/segment bits
35. romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Isolating tenant traffic: Default
10.0.0.5 10.1.128.9
iptables:
check src/dst addrs
“tenant/segment bits
must match”
Src: 10.0.0.5
Dst: 10.1.128.9
Different
tenant/segment bits
Different
tenant
36. romana.io Simplifying the network stack with Romana @romanaproject
Host BHost A
Apply network policy between
segments (full isolation as default)
10.0.0.5 10.1.1.9
iptables:
Does policy chain
exist?
Otherwise: DROP
Src: 10.0.0.5
Dst: 10.1.1.9
Same tenant,
different segment
policy-chain:
From segment 0?
Protocol TCP?
To port 80?
38. romana.io Simplifying the network stack with Romana @romanaproject
Baking layered cakes
● Kubernetes on OpenStack? Why?
– On demand clusters
– Full tenant isolation
● Not all workloads fit into containers
– Seamless connection between pods and VMs
● Really nice with fully routed networking
– No double encapsulation
– Logical, efficient packet forwarding
43. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Install OpenStack
command line tools
44. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
$ neutron port-update
e925b70e-031e-4ef7-a27c-583b4b775290
--allowed-address-pairs type=dict list=true
mac_address=fa:16:3e:e1:df:59,ip_address=10.0.0.0/8
45. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
$ git clone https://github.com/romana/romana
$ cd romana/romana-install
$ ./romana-setup -p static -i my-inventory -s kubernetes install
46. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Romana
installer
47. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Kubernetes + Romana
Romana cluster
address range:
10/8
48. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - Overview
bar-1 bar-2foo
Kubernetes + Romana
Pods
with containers.
Pods have Romana
IP addresses.
49. romana.io Simplifying the network stack with Romana @romanaproject
Demo 1 - What you will see
● Creation of pods
● Network configuration
● Application of network policies
51. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
52. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
vm-workload
Legacy application
in VM
53. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - Overview
bar-1 bar-2foo
Kubernetes + Romana
vm-workload
Direct connection:
- No gateway
- No encap/decap
- No NAT
54. romana.io Simplifying the network stack with Romana @romanaproject
Demo 2 - What you will see
● Creation of pods
● Contact pod from VM
● See the packet route
55. Demo 3:
Romana + Kubernetes cluster
on top of Romana + OpenStack cluster
56. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
57. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
$ ./romana-setup -p static -i hw-inventory -s devstack install
58. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
HW1 HW2 HW3 HW4
OpenStack + Romana
Romana cluster 1
address range:
10/8
59. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
OpenStack VMs
VMs have
IP addresses
of
Romana cluster 1
60. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
$ ./romana-setup -p static -i vm-inventory -s kubernetes install
61. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3
Kubernetes + Romana
VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
Romana cluster 2
address range:
172.16/12
62. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - Overview
VM2 VM3
Kubernetes + Romana
VM1
HW1 HW2 HW3 HW4
OpenStack + Romana
Pods
with containers.
Pods have
IP addresses
of
Romana cluster 2
63. romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
64. romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
Remember this one?
2 Top of Rack
Round Trips
East/West
Traffic
Per Instance
Security
Without pure L3 network
layered clusters
would be even more
complex.
65. romana.io Simplifying the network stack with Romana @romanaproject
OpenStack + Romana
Kubernetes + Romana
Demo 3 - Overview
VM2 VM3VM1
HW1 HW2 HW3 HW4
But with Romana, networking
even in layered clusters becomes
really easy...
66. romana.io Simplifying the network stack with Romana @romanaproject
Demo 3 - What you will see
● Creation of pods
● Pods and VMs with fully routable addresses
● Ease of use showcase: Trouble shooting
67. romana.io Simplifying the network stack with Romana @romanaproject
Conclusion
● Cloud native architectures simplify things
● Need cloud native networking to enjoy benefits
● Romana:
– Cloud native without compromises
– Native network performance
– Mostly static config: Solid network
– Very easy to work with and understand
● Easy to try:
– Simple installers for Kubernetes and OpenStack
68. romana.io Simplifying the network stack with Romana @romanaproject
Thank you!
● Romana Links
– http://romana.io - Project home
– http://romana.io/blog - Blog
– https://github.com/romana/romana - Sources
● Contact
– @romanaproject - Twitter
– info@romana.io - Email
– https://romana.slack.com/ - Slack channel