Connecting Last Mile ISPs to Internet Exchange Points- BKNIX Case Study

Kittinan Sriprasert, BKNIX
15 May 2023
BKNIX Case Study
Connecting Last Mile ISPs to
Internet Exchange Points :

BKNIX Peering Forum 2023 | May, 15-16
Agenda
BKNIX at a glance
Local peering and Remote peering
Challenge of connecting remote location to IX
Design goal and solution
BKNIX Case Study
• Community networks
Conclusion & Future works
2

BKNIX at a glance
3
• BangKok Neutral Internet eXchange Point
• Layer-2 community-based exchange
• First neutral IXP in Thailand since 2015
• Non-for-profit IXP operator under THNIC Foundation
• Peak exchange traffic rate at 148G
• 50 ASN exchange members

Traditional peering
• aka Local peering or physical peering
• Routers resides at the same facility as IX switch
• The connection is simply just using cross-connect inside facility
• Most simple and Most affordable
4
Datacenter IX switch

BKNIX Peering Forum 2023 | May, 15-16 5
Remote peering
Datacenter
A
IX switch
Datacenter
B
Transport
network
• Routers resides at different facility as IX switch (Remote location)
• Remote connection is required
• Different degrees of remote locations
• Same city but different facility
• Different city, province
• Different country, continent
• Can be implemented using a dedicated connection or a virtual private network (VPN)
Dedicated connection

Challenge of connecting remote location to IX
6
Source: Public Infrastructure
• Readiness of public infrastructure
• underground, road, electrical system
• Cost of local loop circuit
• transmission equipment
• datacenter, building, land accessing
• Insufficient cable in that area
• Distance to IX facility
• IX may only be present in major city/province

Design goal and solution
• Cost effective and resource sharing
• Seamlessly integration
• Same experience as connecting directly to BKNIX port
• High performance
• High reliability
• Secure routing and connection
• Easy to scale
7

IPsec (IP Security)
• IPSEC provides secure communication
between 2 peers over IP networks
• Comprise of 2 main components:
• Authentication Header (AH) - Data Integrity
• Encapsulating Security Payload (ESP) - Data encryption
• Modes: Transport mode and Tunnel mode
• Security Associations (SA): establish and manage security parameters
• IPsec NAT-T support connection behind NAT network
8
Source : commons.wikimedia.org

OpenVPN
• Open-source VPN protocol
• TCP and UDP; UDP being faster and more suitable for real-time
applications
• SSL/TLS for encryption
• supports multiple platforms, including Windows, Linux, macOS, and
mobile devices
• supports multiple authentication methods, including certificates,
usernames and passwords, and two-factor authentication
9
Source : openvpn.net

Wireguard
• modern VPN protocol designed for simplicity and efficiency
• It is based on state-of-the-art cryptography, including Curve25519 for key
exchange, ChaCha20 for encryption and Poly1305 for message authentication
• WireGuard operates at the kernel level, making it faster and more lightweight
than traditional VPNs
• The protocol uses a single round-trip time (RTT) for key exchange, reducing
latency and improving performance
• WireGuard has a small codebase and is easy to audit, reducing the risk of
security vulnerabilities
• Work on UDP only (not support TCP)
10
Source : wireguard.com

Internal testbed setup
• Set up 2 routers, first router is local site and second router is from
remote site
• 2 routers connect to the same lan ; lan network representing internet
• Try to set up different vpn tunnels, IPsec, OpenVPN, Wireguard over
this network
11
LAN
10G
10G
Remote
Router
IX
Router
VPN Connection
(IPsec, OpenVPN, Wireguard)

Performance overview
Latency comparison on different packet size
12
ps=24 RTT (ms)
Min Avg Max
Raw 1.765 1.866 1.979
IPsec 1.711 1.880 2.144
OpenVPN 1.836 2.141 2.597
Wireguard 2.168 2.326 2.671
0
0.625
1.25
1.875
2.5
3.125
Raw IPsec OpenVPN Wireguard
ps=1300 RTT (ms)
Min Avg Max
Raw 1.846 1.935 2.121
IPsec 1.808 1.908 2.141
OpenVPN 1.960 2.171 2.522
Wireguard 2.169 2.448 2.989
Avg. RTT
Millisecond
ps=24
ps=1300

Performance overview (1)
Throughput test using iperf [tcp mode] - Raw
13
[ ID] Interval Transfer Bandwidth
[ 13] 0.0000-60.0070 sec 17.3 GBytes 2.48 Gbits/sec
[SUM] 0.0000-60.0096 sec 71.0 GBytes 10.2 Gbits/sec

Throughput test using iperf [tcp mode] - IPsec
14
[ 19] 0.0000-60.0132 sec 3.50 GBytes 501 Mbits/sec

Throughput test using iperf [tcp mode] - OpenVPN
15
[ 43] 0.0000-60.0903 sec 444 MBytes 62.0 Mbits/sec
[SUM] 0.0000-60.2435 sec 1.78 GBytes 254 Mbits/sec
[SUM] 0.0000-60.1985 sec 1.78 GBytes 254 Mbits/sec

Throughput test using iperf [tcp mode] - Wireguard
16

Throughput test using iperf [tcp mode] - Overall
17
10,200 Mbps
2,110 Mbps
1,890 Mbps
254 Mbps
0 3000 6000 9000 12000
Raw
Wireguar
d
IPsec
OpenVP
N
Mbps

BKNIX Case Study :
Community networks
18

Community network (1)
• Decentralized, community-owned and
managed communication infrastructure.
• Built using wired and wireless
technologies and open standards.
• Provides internet access, emergency
communications, and community-based
services.
• Promotes local ownership, decentralized
management, and resilience.
• Empowers underserved or remote areas
and supports community organizing.
19

Community network (2)
20
• Faces challenges including funding, technical complexity, and regulatory barriers.
• Has the potential to empower communities and promote digital inclusion.
Internet Internet
( )

Community network overview
21
ISP A
ISP B
Global
Internet
• Community set up local infrastructure or ad hoc networks
• Community network connect its gateway to ISPs to connect to Internet
• Connecting to multiple ISPs improve reliability of the network
Gateway

Connecting community network to IX
22
BGP
ISP B
ISP A
• Remote networks create remote connection through existing internet infrastructure.
• Remote connection is encrypted and protected over VPN.
• Dynamically advertise/withdraw prefixes or change policy over BGP.
• Peering with dual stack v4/v6 even underlying infrastructure is v4 only
IX ISP Community network
VPN Connection
(IPsec, OpenVPN, Wireguard)

Connecting community network to IX
23
BGP
VPN Connection
ISP B
ISP A
• Redundant connection through different ISPs
• Traffic engineering over multiple links (load sharing, active-standby)
Traffic flow

Mueang Tak
Mae Sot
Myanmar
Community Wireless
Mesh Networks
(CWMN)
BKNIX
Bangkok
Avg. Distance
450-500 KM
Research & THNIC
Academy Center
Testbed setup on CN (1)
Tak
24

Testbed setup on CN (2)
• Set up 2 routers, first router is local site and second router is from CN ;
• 2 routers connect to over the Internet through ISPs
• Try to set up different VPN tunnels, IPsec, OpenVPN, Wireguard over this network
25
BGP
ISP A
Average distance : 450 - 500 KM
VPN Connection

Performance overview on CN (1)
Latency comparison on different packet size on community network
26
ps=24 RTT (ms)
Min Avg Max
Raw 11.569 11.971 12.557
IPsec 11.203 11.536 12.321
OpenVPN 11.670 12.008 13.097
Wireguard 11.557 12.018 12.58
10.5
11
11.5
12
12.5
13
Raw IPsec OpenVPN Wireguard
ps=1300 RTT (ms)
Min Avg Max
Raw 12.105 12.613 13.331
IPsec 12.014 12.374 13.021
OpenVPN 12.247 12.887 13.693
Wireguard 11.924 12.416 13.336
Millisecond
ps=24
ps=1300
Avg. RTT

Latency comparison on diﬀerent packet size on community network
X
Avg
IPsec
OpenVPN
Wireguard
-4 -3 -2 -1 0 1 2 3 4
0.393
0.309
-3.634
Max
IPsec
OpenVPN
Wireguard
-4 -3 -2 -1 0 1 2 3 4
0.038
2.715
-2.325
Percent (%)
Percentage change of Raw (ps=24)
Percent (%)
Percentage change of Raw (ps=1300)
27

Throughput test using iperf [tcp mode] - Raw
28
[ 6] 0.0000-10.0503 sec 537 MBytes 448 Mbits/sec
[ 1] 0.0000-10.0635 sec 537 MBytes 447 Mbits/sec

Throughput test using iperf [tcp mode] - IPsec (aes128gcm)
29

Throughput test using iperf [tcp mode] - IPsec (chacha20poly1305)
30

Throughput test using iperf [tcp mode] - OpenVPN
31

Throughput test using iperf [tcp mode] - Wireguard
32

Throughput test using iperf [tcp mode] - Overall
33
447 Mbps
70 Mbps
44 Mbps
30 Mbps
17 Mbps
0 125 250 375 500
Raw
Wireguard
IPsec (chacha20pol
y1305)
IPsec (aes128gcm)
OpenVPN
Mbps

Conclusion
• Remote peering solution can connect remote network from remote
area to IX
• Remote connection can be done by VPN or IP Tunneling over
Internet
• Resource can be shared by using existing cables and devices
• Peer at IX dynamically via BGP using IPv4 and IPv6
• Although VPN has more computing and latency overhead but it is
more cost effective solution comparing to dedicated links
34

Future works
• Support more tunneling protocol
• Testing connection from more different vendors and devices
35

References
• https://www.watchguard.com/help/docs/help-center/en-US/Content/en-
US/Fireware/mvpn/general/ipsec_algorithms_protocols_c.html
• https://www.wireguard.com/papers/wireguard.pdf
• https://en.wikipedia.org/wiki/IPsec
• https://docs.strongswan.org
• https://commons.wikimedia.org
• https://community.openvpn.net/openvpn
• https://interlab.ait.ac.th/net2home-blog/
• https://academy.thnic.or.th/
36

BKNIX Peering Forum 2023 | May, 15-16 37
Thank you
Q&A?

Connecting Last Mile ISPs to Internet Exchange Points- BKNIX Case Study

Recommended

Recommended

More Related Content

Similar to Connecting Last Mile ISPs to Internet Exchange Points- BKNIX Case Study

Similar to Connecting Last Mile ISPs to Internet Exchange Points- BKNIX Case Study (20)

Recently uploaded

Recently uploaded (20)

Connecting Last Mile ISPs to Internet Exchange Points- BKNIX Case Study