The current world record for the most concurrent players in a single online multiplayer battle is 6,142, set by CCP Games in EVE Online. However, the sheer number of players meant that the gameplay experience when the record was set was distorted, slowed down past the point of enjoyment. But is there another way? Hadean set out to break the world record and show that its technology can support thousands of players in the same multiplayer battle without any compromises. This talk examines the record attempt, the challenges and ultimately, the outcome.

1. How We Tackled
A Gaming Record
Rashid Mansoor (CTO) & Aidan Hobson-Sayers (VP Product)

2. Today’s talk
1. Who we are and the gaming record we targeted
2. How our technology enabled us to do this
3. What we learned from EVE: Aether Wars
4. Introducing EVE Aether Wars: Phase Two

3. Who is Hadean?
● Hadean is a deep tech start-up founded
in 2015
● We are based in London and have
raised £12.35 million in funding to date
● We won a spot at Unbound as one of
the top 50 startups in Europe

4. Which record were we targeting?
6,142 concurrent players
in EVE Online’s Battle of 9-4RP2

5. What challenges did CCP Games face?
● EVE Online uses a game mechanic called Time Dilation (TiDi) to help
manage the lag in large scale battles
● Time Dilation slows the gameplay in EVE Online down to a vast degree
● Time Dilation heavily impacted the battle when the record was set in The
Battle of 9-4R

6. Aether Engine:
The world’s first distributed game engine
● Runs on entire compute clusters and data centres, and dynamically scale up and down
on the cloud
● Much the same way that UE4, Unity, or CryEngine run on gaming rigs
● Aether Engine lets game developers build complex and rich experiences
without limiting their creative vision

7. Aether Engine: only possible With HadeanOS

8. 1st Platform
Mainframe
2nd Platform
Client-Server
3rd Platform
Cloud
Mobile, Social, Cloud,
Big Data
Billions of users
Millions of apps
PCs, LANs, Internet
100s millions of users
10s thousands of apps
Mainframes, Minicomputers
Millions of users
Thousands of apps
To date, there have been three
generations of computing

9. 1st Platform
Mainframe
2nd Platform
Client-Server
3rd Platform
Cloud
Cloud, Edge, 5G
Billions of users,
apps, and sensors
Mobile, Social, Cloud,
Big Data
Billions of users
Millions of apps
PCs, LANs, Internet
100s millions of users
10s thousands of apps
Mainframes, Minicomputers
Millions of users
Thousands of apps
Cloud Edge 5G ???
We are now seeing the rise
of the 4th Platform

10. 1st Platform
Mainframe
2nd Platform
Client-Server
3rd Platform
Cloud
Cloud, Edge, 5G
Billions of users,
apps, and sensors
Mobile, Social, Cloud,
Big Data
Billions of users
Millions of apps
PCs, LANs, Internet
100s millions of users
10s thousands of apps
Mainframes, Minicomputers
Millions of users
Thousands of apps
4th Platform
Hadean Compute Model
The 4th Platform: powered by HadeanOS

11. Everyone gets a CPU
● Aether Engine consists of a
distributed Octree data structure
● This allows the simulation to
dynamically grow and shrink
across the cloud

13. Building an engine
with a familiar API
void initialise([...], cell_state &state) {
}
void initialise_cell([...], cell_state &state) {
}
void cell_tick([...], cell_state &state, float delta_time) {
}
struct physics_update_system {
typedef std::tuple<c_physics> components;
using ecs_type =
aether::constrained_ecs<cell_state, components>;
public:
void operator()([...], ecs_type &state, float delta_time) {
}
};
struct c_physics {
};

14. For Example - EVE: Aether Wars
void initialise(const aether_cell_state<octree_traits> &ae_state, cell_state &state) {
const auto cell = aether_state.get_cell();
build_aliens(state, cell);
build_player_manager(state, state);
build_asteroids(state, cell);
}
void initialise_cell(const aether_cell_state<octree_traits> &ae_state, cell_state &state) {
state.add_system<physics_init_system>();
state.add_system<physics_update_system>();
state.user_data = nullptr;
}
void cell_tick(const aether_cell_state<octree_traits> &ae_state, cell_state &state, float delta_time) {
state.tick(aether_state, delta_time);
}
struct physics_update_system {
// Per previous slide
};
struct c_physics {
float mass;
float size;
collision::sphere shape;
rigidbody::physics_state state;
};

15. Our ECS
● Data-driven design
● Achieve performance on coherent memory architectures
● Is being rapidly seen as the way for game programming
(e.g. Unity is moving in that direction and showing great
performance)

16. Hadean ECS: distributed by default
struct physics_update_system {
void operator()(const aether_cell_state<octree_traits> &aether_state, ecs_type &state, float delta_time) {
auto entities = state.local_entities<c_physics>();
for(auto p: entities) {
auto phys = p.get<c_physics>();
auto& state = phys->state;
state.step(delta_time);
}
check_collisions(entities, delta_time);
}
};
void physics_state::step(float delta_time) {
const float t = delta_time;
const float t_sqr = t * t;
const auto acceleration = force * inverse_mass;
linear_velocity = linear_velocity + (acceleration * t);
if (linear_velocity.norm() > max_linear_velocity) { // Cap the velocity
linear_velocity = linear_velocity.normalized() * max_linear_velocity;
}
position = position + (linear_velocity * t) + (acceleration * 0.5f * t_sqr);
// Angular velocity etc
[...]
}

17. Towards a Distributed Game Engine
Aether Sim
64 workers/box
M boxes
Dispatcher
E
dispatchers/box
O boxes
NA
State Muxers
S muxers/box
N boxes
NA 2500 Players
UK 2500 Players
EU 2500 Players
JP 2500 Players
1 Europe Central Azure Datacenter
R Edge Azure Datacenters
(UK, East Europe, Japan,
North America)
R Regions
(UK, East Europe, Japan,
North America)
UK
State Muxers
EU
State Muxers
JP
State Muxers
● 50MB/s compressed down per muxer box
● 9MB/s up total
● 230ms RTT worst case europe west to Japan
● 10GB/s relevanced down total
● 9MB/s up total
● 60ms RTT estimate
Home Users
Approximated/te
sted using AWS
Lambdas in the
four regions
Everything on Azure uses
- 8 NICs per box
- global peering
- accelerated networking
Constants
M (sim scale) = [4-10]
N (state scale) = ~4
O (event scale) = ~4
R (region scale) = 4
E (event load) = 8
S (state load) = ~8
Component counts
Workers = M*64 = ~256-640
Routers = E*O = ~32
Muxers = R*S*N = ~128
Forwarders = R = 4
Event forwarder
Event forwarder
Event forwarder
Event forwarder

18. ● Netcode is crucial for networked
games, but doesn’t directly
contribute to the creative vision
● For EVE: AW 2 we began building
a ‘netcode standard library’ for the
Engine to permit configurability at
every level
● We will dogfood this to enable an
easy ‘getting started’, with ultimate
configurability down the line
Intro NetCode Slide

19. Communicating State: Bottleneck
● A game world distributed across several servers poses some fresh
challenges
● Game State is huge
● Bandwidth to clients is very limited
● Solution: send neighbourhoods only; vary update rates

20. Communicating State: Details
● We use R-trees to spatially index entities around players
● We do rapid reads to compute ‘all neighbours in radius‘ for multiple fixed radii
● We send neighbors with different update rates depending on radial (or
spherical sector) distance and entity type
● The logic is highly extensible for game type and given other factors (e.g.
bandwidth, latency, QoS)

21. Communicating State: Other Problems
● R-trees may not be the optimal
solution -- we needed a quick
fix that is better than naive
approach
● Trade-off between speed of
update of the index vs speed
of reads (both critical to good
performance)
● Next experiment: Voronoi
tessellation on an octree (e.g.
Drost & Ilic, 2017)

24. ● Run on 768 CPU cores distributed across 8x client connection servers
● Players from 121 countries in a single game world
EVE Aether Wars: live playtest at GDC19

25. Key Learnings
For others who may venture down this path!

26. Our Learnings: Communication
● Communication is the biggest problem. Between cores, between servers,
between data-centres… and between people!
● Edge and cloud backbones are good but not a solve-all. We saw great
bandwidth and reliability but still had to deal with ping constraints

27. Our Learnings:
Netcode
● A careful balance between player
experience, bandwidth/latency,
server CPU/memory - no one size
fits all!
● We need to work on our distributed
debugging - stack traces that are
meaningful on a distributed engine
or game requires special tooling

28. Our Learnings: Planning
● Building tooling upfront so we can
check when and where things go
wrong
● Representative scale testing
shouldn’t wait - at scale, you need
bots that look like players so you
can get a feel for the actual
performance of your system

29. Our Learnings: Gameplay Constraints
● Engine performance at scale
is very much dependent on
gameplay
● You can get everything right
for a specific CPU/machine,
network and cloud but some
aspects can still be off due to
gameplay demands

30. Our Learnings: Logistics
● We ran the playtest during GDC -
this was suboptimal for securing
players to take part
● For Phase Two, we asked the
community what time they’d prefer
and used that to run the playtest

31. Our First Step
● We still have a lot to learn about
building engines and games in
general
● We also need to learn about the
demands of different game
genres… or even just different
games

32. Building With Our Partners
● We want to get our Aether Engine
SDK out so that we can see how the
community will break things and teach
us that we don’t know what we don’t
know
● We’ve come a long way with CCP
Games but we want to hear from you
as well!

33. So how did we do?

34. EVE: Aether Wars - Battle Report
● 3,852 human players participated
● A total of 14,274 ships in battle
● Players from 121 different countries, from Hawaii to Siberia
● A steady 50ms tick rate throughout the playtest

35. However...
● Gameplay was relatively simple: a
massive dog fight in space
● Despite the server holding strong,
bandwidth to the client and rendering
made the experience poor for some

36. Also...
● We learned that server/bandwidth usage cost would’ve been prohibitive for
real games
● Because of that we have done (and are continuing to do) massive
optimisations on the server end
● We’re already seeing some great results!

37. “It’s a huge milestone in the EVE: Aether
Wars project between Hadean and CCP
Games, as part of the continued drive to
explore the future of large-scale multiplayer
simulation.”
“Did EVE: Aether Wars succeed? As tech
demo, absolutely. Hadean proved right
then and there that they could have nearly
15,000 players simultaneously connected
and not crash the server.”
“Over the course of the action, some 88,988 ships
were destroyed, making other sci-fi sagas such as
Mass Effect look like a grazed knee by comparison.”
What did they say?

38. “We knew that Hadean’s tech held massive
potential and I’m ecstatic to see that potential
realised for the first time with EVE: Aether
Wars. I’m convinced that our partnership will
continue to break new grounds in virtual
worlds.”
Hilmar V. Petursson
CEO

41. EVE Aether Wars:
Phase Two
● Showcasing a compelling multiplayer
experience at enormous scale
● Building on the technological success of
Phase One and pushing the boundaries
even further
● A richer, deeper gameplay experience

42. ...and you can join in!
● The EVE Aether Wars: Phase Two playtest will take place later today at 19:00
CEST (lasting roughly an hour)
● All participants will receive an exclusive EVE Online skin bundle and the
chance to win a trip for two to Las Vegas and a Corsair PC rig
● Follow #EVEAetherWars if you can’t take part!
● Sign up at www.hadean.com/eveaetherwars

43. Visit us at
Hall 2.2 B-057
hadean.com @hadeanincaetherengine

44. Any questions?
hadean.com @hadeanincaetherengine