What is the internet, really? It is a network of networks which are all interconnected to various degrees. Where they interconnect is typically at an internet exchange (IX) or internet exchange point (IXP). IX are important because they make the internet (and especially their user's networks) more resilient (robust and fault tolerant), more performant (higher bandwidth & lower latency), and quite often more cost effective. It's not difficult to connect to an IX but you do need to know the basic requirements, process, and best practices. You can also use a new open-source automation platform, called PeerCtl, to make connecting over an IX even easier. These slides are from a talk I gave on 4 May 2023 in Albuquerque, NM, USA. The talk covers: * What is the Internet, really? * What is an Internet Exchange (IX)? * Why are IX’ (and interconnection) important? * How-To Start Interconnecting

2. Agenda
● Who am I?
● What is the Internet, really?
● What is an Internet Exchange (IX)?
● Why are IX’ (and interconnection) important?
● How-To Start Interconnecting
● Q & A

3. Who am I?

4. Who let this guy in?

5. Who let this guy in?
● On computers since single digits
● 20 years in networking
● Designed, deployed, and operated SP, DC, and Enterprise networks
● Given talks on networking in 34 countries on 5 continents
● Holds 8 patents on network technologies
● Written 2 books on networking topics
● Co-founded FullCtl (Interconnection Software Provider)
● Co-founded IX-West / IX-Denver (Internet Exchange in Colorado)
● Director at OIX (ANSI accredited interconnection standards body)
● More: https://chrisgrundemann.com

6. The Internet

7. What is the Internet, really?
● TCP/IP
● A network of networks
● How we communicate
● How we learn
● How we heal
● How we do business
● How we access critical services
● How we build the future

8. The Internet - TCP/IP

9. The Internet The Web
http://internet-map.net/

10. The Internet The Web
http://internet-map.net/

11. The Internet - Undersea Cables
https://www.submarinecablemap.com/

12. The Internet - U.S. Long-Haul Fiber
https://pages.cs.wisc.edu/~pb/tubes_final.pdf

13. The Internet - A Network of Networks
https://www.opte.org/

14. The Internet - A Network of Networks
https://www.opte.org/

15. Internet Exchange (Points)

16. What is an Internet Exchange (IX)?
● A switch
● An organization
● A physical point of interconnection
● A “knot” tying networks together into the internet

17. What is an IX?
“While network providers might be the foundation of the internet,
internet exchange points (IX or IXP) can be considered the glue that
holds the internet together.”
“An IX is the physical location and infrastructure where networks
exchange internet traffic.”

18. What is an IX?

19. Why are IX’ important?
● Keep local traffic from having to leave local area
● Minimize latency, jitter, & packet loss (great for VoIP & video
conferencing)
● Reduced router/transport hops = greater reliability
● Maximize bandwidth: LAN speed local interconnects 1G, 10G, 100G
● Reduce costs for members by avoiding transit costs for traffic between
member networks
● Improve regional fault tolerance (i.e. if long-haul links out of SW are
severed/degraded, traffic can still flow locally uninterrupted)
● Direct access to major content sources (Google, CloudFlare, Meta, etc...)
● Provides additional traffic interconnect location to provide redundancy
for other regional peering fabrics

20. Lowering the Cost of Broadband
● IX versus IXP
● Local loop versus Internet access
● Fostering (real) competition

21. A (poorly drawn) Illustration
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y

22. Transit Providers
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y

23. Added Latency
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y

24. What about upstream failures?
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y
X

25. Reliant on non-local third-parties
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y
X

26. Multiple (unknown) failure conditions
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y
X

27. What if we add a local IX?
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y
ABQ
IX

28. ABQ
IX
Local traffic stays local.
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y

29. And the “big guys” become locals…
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y
https://ix-denver.org/members/
ABQ
IX

30. Plus, extra protection for local services
Local
ISP A Local
ISP B
Local
ISP C
Regional NSP
National ISP X
National ISP Y
https://ix-denver.org/members/
ABQ
IX

31. “According to ITU surveys, interconnection-related issues are ranked
by many countries as the single most important problem in the
development of a competitive marketplace for telecommunications
services…” (Intven et al. 2000)
https://regulationbodyofknowledge.org/faq/telecommunication-regulatio
n-interconnection/what-is-interconnection-and-why-is-it-important/

32. How-To Start Interconnecting

33. The Basics
● Transport
● Router
● BGP

34. The Basics
● Transport
● Router
● BGP
Data Center
(IXP)
Your AS

35. The Basics
● Transport
● Router
● BGP
Data Center
(IXP)
Router
Your AS

36. The Basics
● Transport
● Router
● BGP
Data Center
(IXP)
Router
Your AS
IX

37. The Basics
● Transport
● Router
● BGP
Data Center
(IXP)
Router
Your AS
IX
Route
Server

38. The Basics
● Transport
● Router
● BGP
Data Center
(IXP)
Router
Your AS
Internet
Carrier 1
Carrier 2
IX
Route
Server

39. The Basics
● Transport
● Router
● BGP
Data Center
(IXP)
Router
Your AS
Internet
Carrier 1
Carrier 2
IX
Route
Server
Buy / Sell

40. Best Current Operational Practice:
IX Connections

41. Physical Layer Guidelines
1. Direct connection to the Exchange – when at all possible, a Participant should connect directly from a Layer 3 interface into the shared switch fabric of
the Exchange. A direct connection minimizes the likelihood of any extraneous broadcasts or other not allowed control traffic traversing between autonomous
Layer 2 environments. Specifically, direct switch-to-switch connections between the Exchange and Participant should be avoided. The potential for impacting
the entire fabric and all Participants through spurious Layer 2 events is significantly higher than with a direct connection.
2. Confirm transmission characteristics of Layer 1 connectivity – when connecting to an Exchange for the first time, it is recommended to take a set of
baseline readings to use for later comparisons in the event of link issues. Specifically, link integrity should be verified and recorded.
a. Fiber Connection – Power readings on transmit and receive should be taken with an Optical Power Meter (OPM) and recorded for reference. Any
questionable readings should be addressed before putting traffic on the link. b. Copper Connection - Continuity testing should be performed and output
recorded for reference. As with fiber, any questionable readings should be addressed before putting traffic on the link. c. Both Fiber and Copper Connections
– once the physical connection has been established, further baseline data should be gathered at the interface. Specifically, a lack of errors
(CRC/Framing/Input/Output/Drops, etc.) should be observed and, if not, addressed before putting traffic on the link.
4. Properly label connections – all connection components should be properly labeled for easy identification using your standard naming conventions. This
includes fiber/copper runs, patch panels, interfaces and connected devices.
5. Create an As-Built of your connection - a current configuration document will become invaluable in an outage or for knowledge transfer, and should
include the following:
a. Link by link definition of circuit - each juncture point on the path between your interface and the Exchange should be recorded, using the connection
labels from (3) above. b. Create a contact list for all providers on the Layer 1 chain - at the very least a contact or list of contacts for the Exchange should be
associated with the As-Built and, in the event multiple carriers are involved, contact information should be included for each carrier and associated to their
point or points in the Layer 1 path.
https://github.com/Open-IX/BCOP/tree/master/IX_Connections

42. Data Link and Network Layer Guidelines
1. Properly configured IP and subnet mask (both IPv4 and IPv6) – the appropriate subnet mask must be configured on your connected interface. It is not
safe to assume that the mask configuration is similar across different Exchanges.
2. Ethertypes - Frames forwarded through an Exchange must only have one of the following types: a. 0x0800 – IPv4 b. 0x0806 – ARP c. 0x86dd – IPv6
3. All broadcast and Non-IP protocols must be disabled facing the exchange fabric, or filtered in some way when disabling is not an option. These include, but
are not limited to forwarded DHCP, MOP, Ethernet Keepalives, NetBIOS and IPv6 Router Advertisements.
4. All proprietary discovery protocols (CDP, FDP, etc.) must be disabled on exchange-facing interfaces.
5. IP Redirects must be disabled on exchange-facing interfaces.
6. IP Directed-broadcast must be disabled on exchange-facing interfaces.
7. Proxy Protocols - such as Proxy ARP and IPv6 Proxy Neighbor Discovery must be disabled on exchange-facing interfaces.
8. Layer 2 control traffic - in a switch-to-switch configuration, unless otherwise specified, spanning tree must be disabled on exchange-facing interfaces. In the
event this is not possible, all control traffic must be suppressed on exchange-facing interfaces.
9. All Interior Gateway Protocols must be disabled on exchange-facing interfaces.
10. Most Exchanges are very restrictive on the amount of MAC addresses on one connection. Usually the limit is 1. If the Participant has any device between the
router ports and the Exchange, he should make sure, that this device is completely quiet and not sending any unwanted packets (like stated earlier in this
Section). This might lead to hit the MAC security, which is configured on many Exchanges and disables the port, where the Participant is connected, for
traffic.
https://github.com/Open-IX/BCOP/tree/master/IX_Connections

43. Transport Layer Guidelines
1. Announcing Exchange IPv4/6 subnet via BGP - a participant must not announce the exchange subnets via
BGP.
2. Preconfigured Session Turn-UP - a Participant should not preconfigure and enable BGP peering sessions to
other Participants until both sides are ready to establish the connection.
3. A Participant should set a maximum prefix of ACL on peers whenever possible.
4. If the Exchange provides route server(s) and the Participant maintains an open peering policy, he should
connect to those, to get as much prefixes from other Participants as possible with less effort. Establishing
direct BGP sessions between Participants, which exchange mission-critical or lots of traffic, is also
important for redundancy.
5. The Participant's NOC should be aware of the Exchange's technical mailing list and be subscribed to it.
The subscription address must not be a ticket system with automated replies enabled.
6. If an existing peer of the Participant disconnects from the Exchange, the Participant should remove the
session towards that peer immediately to reduce ARP traffic on the Exchange fabric.
https://github.com/Open-IX/BCOP/tree/master/IX_Connections

44. Other Guidelines
1. Security - there are very few security measures available for connections between Participants and, given the bilateral
nature of any interconnection across the Exchange, this BCOP can only describe, but not recommend, any particular
method.
a. MD5 - Providers may use MD5 to secure their BGP session. b. MAC Access Control List (ACL) - A provider may use a
MAC ACL to ensure remote Participants are establishing BGP sessions from a known source. However, MAC ACLs can be
problematic. c. IP ACL's - a provider may use IP ACLs to define by IP those participants that are allowed to establish BGP
sessions with the Participant's IP address. d. Generalized TTL Security Mechanism (GTSM) - Providers should enable
GTSM on their peers whenever possible. e. Newer BGP security mechanisms are not addressed in this document revision.
2. PeeringDB - all Participants should have an updated entry in the PeeringDB.
3. Routing Registry - all Participants must have up-to-date objects in the Routing Registry (IRR) of their choice.
https://github.com/Open-IX/BCOP/tree/master/IX_Connections

45. Easy Button: PeerCtl

46. PeerCtl
Interconnection automation and workflow management.
● modern, modular, Unix philosophy services
● 100% API driven
● manage infrastructure
● automate provisioning
● source of truth for wider automation efforts
● open source or as a service
https://github.com/fullctl/peerctl

47. PeerCtl
Peering Toolkit
1. You select an IX. PeerCtl shows you all of the possible peers at that
location.
2. You select a peer. PeerCtl provides BGP configurations and email
templates to get connected.
3. You want more. PeerCtl shows you all of the other mutual
exchanges where you can interconnect with this peer.
https://github.com/fullctl/peerctl

48. PeerCtl - but wait, there’s more
● All kinds of “peers”
○ Bilateral, multilateral, PNI, transit, transport
● Source of truth for your Interconnection efforts
○ Sync with third party inventory via DeviceCtl
○ Stay virtual with PDB Ports
● Extensibility
○ PeerCtl BGP configurations are extensible and customizable with templates
○ Templates also allow you to extract peering data for other uses
○ Email templates with override also (and pdb contacts, etc)
● “show BGP summary”
○ View all sessions at once
● Import from network
○ Pull in existing BGP sessions
https://github.com/fullctl/peerctl

49. PeerCtl - Coming Soon
● Public Peering Portal
○ for third parties to use when requesting peering with your network
● “Where To Peer” Improvements
○ integrating traffic analysis to provide intelligent recommendations
● PeeringDB Integration Improvements
○ enable pushing updates to PeeringDB from PeerCtl
○ enable you to add networks to PeerCtl that are not listed in PeeringDB
● UX Improvements
○ enhanced alerting, along with interface design improvements
https://github.com/fullctl/peerctl

50. Questions???
● Eat local
● Drink local
● Shop local
● PEER local