This webinar, will showcase the main activities and benefits of our Amazon OpenSearch Service Assessment offering. We'll take you through a real-life project where we partnered with a big MarTech player to enhance their Amazon OpenSearch solution for big data analytics while cutting over 60% of monthly costs. --- Get in touch: opensearch@kreuzwerker.de ---

1. Level Up your Amazon
OpenSearch Cluster in a Week
A Glimpse into kreuzwerker’s
Search Assessment Offering
for OpenSearch and Elasticsearch

2. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Your Hosts
Guido Lena Cota Olga Syrova

3. kreuzwerker
We unleash digital technology.

4. Founded 2012
AWS Partner since 2013
100% AWS
Berlin, Frankfurt,
Warsaw, Zurich
Migration & Modernization
Data & Analytics
Cloud Native Development
Managed Security Services
Cloud Financial Management

5. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
AUG
1
Search Assessment in a week
.INTRO CALL.
YOU present your
problem.
TOGETHER WE
set expectations
and timeline.
.PREPARATION.
YOU fill out our
questionnaire to
detail your use case
and prioritize pain
points.
.WORKSHOP.
TOGETHER WE
address the main
pain points and
lay down possible
remediations.
.REPORT.
WE present a
remediation plan
and highlight
further areas of
improvements.
(optional)
TOGETHER WE
implement the
remediations.
(migrations,
security and
compliance,
automation,
performance,
data pipelines,
ML/AI, …)
AUG
5
AUG
8
5
01 / About Us

6. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Search Assessment in a week
− How to assess OpenSearch
for Cost Efficiency
− How to approach Capacity
Planning
.WORKSHOP.
TOGETHER WE
address the main
pain points and
lay down possible
remediations.
.REPORT.
WE present a
remediation plan
and highlight
further areas of
improvements.
AUG
5
AUG
8
6
01 / About Us

7. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
© kreuzwerker 2023
AGENDA
About Us
Use Case
How to assess OpenSearch
for Cost Efficiency
How to approach Capacity
Planning
Q&A

8. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
Jazz™
affiliate marketing
campaigns management
performance analysis
…
8
02 / Use case
Mark.com
Online MarTech platform for B2B

9. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
REVENUE PUBLISHERS
Mark.com
9
02 / Use case

10. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
REVENUE PUBLISHERS
Mark.com
Price: $$
Bike™
10
02 / Use case

11. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
Price: $$
11
02 / Use case

12. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
Jazz™
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
Price: $$
12
02 / Use case

13. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
Jazz™
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
Price: $$$
13
02 / Use case

14. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
Jazz™
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
Price: $$$
14
02 / Use case

15. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
Jazz™
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
Price: $$$$$$
15
02 / Use case

16. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Mark.com
Online MarTech platform for B2B
Bike™
Cone™
Jazz™
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
REVENUE PUBLISHERS
Mark.com
Price: $$$$$$$$$$$...
16
02 / Use case

17. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
© kreuzwerker 2023
AGENDA
About Us
Use Case
How to assess OpenSearch
for Cost Efficiency
How to approach Capacity
Planning
Q&A

18. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
AWS Cloud
VPC
Amazon OpenSearch Service
1
2
3
1 Send data as JSON 2 Store and make data searchable 3 Query and analyze data
Master Nodes
Hot Data Nodes
Warm / Cold Nodes
How Amazon OpenSearch works
03 / How to assess OpenSearch for Cost Efficiency 18
GET clicks/_search
{
“query”: {
…
}
}

19. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Cost Factors
03 / How to assess OpenSearch for Cost Efficiency 19
− Node(s) Usage
− Storage
− Data transferred in/out

20. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Cost Factors
1. Use Case
2. Architecture
3. Resource Utilization
4. Index Design
Systematic Assessment
− Node(s) Usage
− Storage
− Data transferred in/out
03 / How to assess OpenSearch for Cost Efficiency

21. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Assess the Use Case
− Large-scale Aggregations for Analytics
− Big Data
− 60 GB/day, 150M docs/day
− Structured Data
− e.g., clicks, impressions, leads
− Time Series
− Retention Policies
03 / How to assess OpenSearch for Cost Efficiency 21

22. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Assess the Use Case
− Large-scale Aggregations for Analytics
− Big Data
− 60 GB/day, 150M docs/day
− Structured Data
− e.g., clicks, impressions, leads
− Time Series
− Retention Policies
Assessment Notes
✔large memory + disk needs
❑ ARCH: data tiers?
❑ INDEX: mappings and settings?
03 / How to assess OpenSearch for Cost Efficiency 21

23. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
event aggregator
(cron jobs)
data ingestors
(microservices)
event aggregator
(cron jobs)
data ingestors
(microservices)
dashboards
data ingestors
(microservices)
event aggregator
(cron jobs)
Assess the Architecture
AWS Cloud
VPC
Amazon OpenSearch Service
Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
Amazon CloudWatch
Mark.com
datacenter
read
index
read,
index
03 / How to assess OpenSearch for Cost Efficiency 22

24. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
event aggregator
(cron jobs)
data ingestors
(microservices)
event aggregator
(cron jobs)
data ingestors
(microservices)
Assess the Architecture
AWS Cloud
VPC
Amazon OpenSearch Service
Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
Amazon CloudWatch
Mark.com
datacenter
dashboards
data ingestors
(microservices)
event aggregator
(cron jobs)
read
index
read,
index
Assessment Notes
✔large memory + disk needs
✔ARCH: no data tiers
❑ INDEX: mappings and settings?
❑ ARCH: oversized cluster?
❑ ARCH: overprovisioned
nodes?
03 / How to assess OpenSearch for Cost Efficiency 22

25. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
event aggregator
(cron jobs)
data ingestors
(microservices)
event aggregator
(cron jobs)
data ingestors
(microservices)
Assess the Architecture
AWS Cloud
VPC
Amazon OpenSearch Service
Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
Amazon CloudWatch
Mark.com
datacenter
dashboards
data ingestors
(microservices)
event aggregator
(cron jobs)
read
index
read,
index
03 / How to assess OpenSearch for Cost Efficiency 22

26. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Assess the Resource Utilization
Heap Usage
%
CPU Usage
%
03 / How to assess OpenSearch for Cost Efficiency
Storage Usage
%
Data Nodes
23

27. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Heap Usage
%
CPU Usage
%
Storage Usage
%
Data Nodes
Assess the Resource Utilization
Assessment Notes
✔large memory + disk needs
✔ARCH: no data tiers; too many
or overprovisioned data nodes;
disks largely unused
❑ INDEX: mappings and settings?
❑ INDEX: sharding strategy?
03 / How to assess OpenSearch for Cost Efficiency 23

28. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Impact of sharding strategy on nodes
Amazon OpenSearch Service
Data Node 01 Data Node 02
Shards Allocation
Document Index Shards
03 / How to assess OpenSearch for Cost Efficiency 24

29. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Impact of sharding strategy on nodes
Amazon OpenSearch Service
Data Node 01 Data Node 02
Shards Allocation
index search
Number of shards affects CPU demand
- Index requests are sent to all shards
(primaries + replicas) of a write index.
- Search requests are sent to either a
primary or replica for every shard.
03 / How to assess OpenSearch for Cost Efficiency 25

30. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Impact of sharding strategy
Amazon OpenSearch Service
Data Node 01 Data Node 02
Shards Allocation
Shards size affects CPU and memory,
and impacts operations
- Small shards introduces CPU and
memory overhead.
- Storage skew may create bottlenecks.
03 / How to assess OpenSearch for Cost Efficiency

31. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Assess the Index Design
03 / How to assess OpenSearch for Cost Efficiency 26
21 indices
− on average:
− 35 shards among primaries and replica
− 500 MB per shard
− active indices:
− 4 indices involved for writes (142 shards)
− 14 indices involved for reads (200 shards)
> GET _cat/indices?v&h=index,pri,rep,docs.count,pri.store.size
index pri rep docs.count pri.store.size
clicks_day_2023-08-01 4 3 10957593 1.8gb
...

32. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Assess the Index Design
03 / How to assess OpenSearch for Cost Efficiency 26
21 indices
− on average:
− 35 shards among primaries and replica
− 500 MB per shard
− active indices:
− 4 indices involved for writes (142 shards)
− 14 indices involved for reads (200 shards)
Assessment Notes
✔large memory + disk needs
✔ARCH: no data tiers; too many
or overprovisioned data nodes;
disks largely unused
✔INDEX: over-sharding
❑ INDEX: mappings and settings?

33. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Assess the Index Design
03 / How to assess OpenSearch for Cost Efficiency 26
Mappings inefficiencies, e.g.,
✘ identifiers with numeric type → ✔ “keyword” type
✘ indexed fields unused in queries → ✔ ”index”: false
…
Settings inefficiencies, e.g.,
✘ refresh rate every second → ✔ at least 30s
✘ retention policies via cron jobs → ✔ index lifecycle policies
…
Assessment Notes
✔large memory + disk needs
✔ARCH: no data tiers; too many
or overprovisioned data nodes;
disks largely unused
✔INDEX: over-sharding; possible
optimizations for mappings and
settings

34. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Recap Assessment
Assessment Notes
✔large memory + disk needs
✔ARCH: no data tiers; too many
or overprovisioned data nodes;
disks largely unused
✔INDEX: over-sharding; possible
optimizations for mappings and
settings
03 / How to assess OpenSearch for Cost Efficiency 27

35. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
© kreuzwerker 2023
AGENDA
About Us
Use Case
How to assess OpenSearch
for Cost Efficiency
How to approach Capacity
Planning
Q&A

36. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
An Iterative Process
Estimate
Validate
Fine-Tune
Preparation
CAPACITY
PLANNING
04 / How to approach Capacity Planning 29

37. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
1. Reduce storage requirements
2. Reduce CPU and memory
requirements
“How we accelerate financial
and operational efficiency with
Amazon OpenSearch”
kreuzwerker.de blog
Prepare for Capacity Planning
✔large memory + disk needs
✔ARCH: no data tiers; too many
or overprovisioned data nodes;
disks largely unused
✔INDEX: over-sharding; possible
optimizations for mappings and
settings
Assessment Notes
04 / How to approach Capacity Planning
“Crafting Solutions with the
Elastic Stack: pragmatic takes
and lessons learned”
Elasticsearch Berlin Meetup
30

38. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
An Iterative Process
Estimate
Validate
Fine-Tune
CAPACITY
PLANNING
04 / How to approach Capacity Planning 31
A Hot nodes only
Potential cost saving: ~62%
B Hot + Warm nodes
Potential cost saving: ~72%
.SPOILER..
Proposed configurations:

39. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Capacity Planning / Estimate
1. Select instance family
2. Estimate storage requirements
3. Estimate CPU requirements
4. Determine the cluster
configuration to validate
04 / How to approach Capacity Planning 32

40. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Capacity Planning / Estimate
1. Select instance family
2. Estimate storage requirements
3. Estimate CPU requirements
4. Determine the cluster
configuration to validate
✔large memory + disk needs
✔ARCH: no data tiers; too many
or over-provisioned data nodes;
disks largely unused
✔INDEX: many write shards;
over-sharding; no lifecycle
policies; possible optimizations
for mappings and settings
Assessment Notes
Candidate:
R* instance type, optimized for
memory-intensive workloads
Supported instance types in Amazon OpenSearch Service
AWS Docs
04 / How to approach Capacity Planning 33

41. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Capacity Planning / Estimate
1. Select instance family
2. Estimate storage requirements
3. Estimate CPU requirements
4. Determine the cluster
configuration to validate
Best Practices
Formulas:
Index_Storage =
(GB_day * retention_days * (#replicas+1))
Total_Storage =
(∑index
Index_Storage) * service_overhead
, where service_overhead = 1.45
Recommendation:
Account for 25% of growth per node.
04 / How to approach Capacity Planning 34

42. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Capacity Planning / Estimate
1. Select instance family
2. Estimate storage requirements
3. Estimate CPU requirements
4. Determine the cluster
configuration to validate
Best Practices
Formula:
Total_vCPU = ∑index
(1.5 * #active_shards)
04 / How to approach Capacity Planning 35

43. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Capacity Planning / Estimate
1. Select instance family
2. Estimate storage requirements
3. Estimate CPU requirements
4. Determine the cluster
configuration to validate
Best Practices
Recommendations:
- Always.Choose.Graviton.
- Consider scalability costs
- Take availability into account
- Take heap requirements into account
04 / How to approach Capacity Planning
Amazon OpenSearch Service Pricing
AWS Docs
36

44. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Capacity Planning / Validate
04 / How to approach Capacity Planning
1. Select instance family
2. Estimate storage requirements
3. Estimate CPU requirements
4. Determine the cluster
configuration to validate
“On benchmarking
Elasticsearch: some advice, a
blueprint”
kreuzwerker.de blog
BENCHMARK IT!

45. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Back to Mark.com…
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
12x
0 Initial Setup
04 / How to approach Capacity Planning 37

46. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
9x
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
12x
0 Initial Setup 1 Reduce size
Potential saving:
~25%
04 / How to approach Capacity Planning 37
Back to Mark.com…
Apply
remediations

47. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
9x
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.4xlarge.search
12x
0 Initial Setup 1 Reduce size 2 Downgrade hot nodes
2a 2b
3x Master Nodes
c5.large.search
Hot Data Nodes
i3.2xlarge.search
9x
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.2xlarge.search
9x
Potential saving:
~25%
Potential saving:
~47%
Potential saving:
~58%
04 / How to approach Capacity Planning 37
Back to Mark.com…
Apply
remediations

48. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Additional cost-saving opportunities
2 Downgrade hot nodes
2a 2b
3x Master Nodes
c5.large.search
Hot Data Nodes
i3.2xlarge.search
9x
3x Master Nodes
c5.large.search
Hot Data Nodes
r5.2xlarge.search
9x
Potential saving:
~47%
Potential saving:
~58%
1. Choose Graviton instance types.
e.g., cluster of 9 r6g.2xlarge.search hot nodes
= ~8% cheaper than
2. Reserved Instance pricing model.
e.g., reserve master nodes + 4 hot data nodes
for one year with no upfront payment
= ~12% cheaper than
3. Introduce warm data tier.
e.g., replace 3 hot data nodes with one
ultrawarm1.medium.search node
= ~29% cheaper than
2b
04 / How to approach Capacity Planning 38
2b
2b

49. | Level Up your Amazon OpenSearch Cluster in a Week
© kreuzwerker 2023
Search Assessment is just the beginning
.PREPARATION.
.W
ORKSHOP.
.INTRO CALL.
.REPORT.
1 week
migrations
security
performance
data pipelines
semantic search
logs
capacity planning
ML
04 / How to approach Capacity Planning 39
compliance
automation

50. 50
Questions?

51. kreuzwerker GmbH
Ritterstr. 12
10969 Berlin
www.kreuzwerker.de
Fon +49 30 609 838 80
Fax +49 30 609 838 899
opensearch@kreuzwerker.de