An overview of the use-case to move a workload from on-prem to the cloud with kafka , serverless and making it secure.

1. Streaming data
with serverless at
de Volksbank
Bart Monhemius & Jacob Verhoeks
AWS Community Day 2023 , Utrecht

2. Who’s joined us for?

4. Jacob Verhoeks
Techlead – Mission Critical
Engineer @ Schuberg Philis
AWS Community Builder
Bart Monhemius
Solutions Architect – Senior
DevOps Engineer @ de Volksbank

5. Moving away from the mainframe

6. Decoupling the mainframe with streams

7. AWS Cloud and the
Bank
Cloud Strategy
Why Serverless?
Secure Enablement Platform
Landing zone

8. Cloud strategy
• Cloud first
• SaaS over PaaS over IaaS
• Cloud Native

9. Serverless – Maximize the work not done

10. Serverless –
Minimize the
carbon emitted

11. Serverless –
Enables autonomy

12. Serverless –
Minimize the
attack surface

13. Serverless –
Resilient by
default

14. Enablement
Platform
• Infrastructure as Code
• Terraform Cloud
• Git
• Guardrails, no insecure
config
• Central Network and
Firewall
• Much more

15. Start Building
• Standard way of working
• Building blocks in terraform
• Reusable modules
• Built-in security
• Easy DTAP

16. KMS is
your
friend
Customer
Managed
Keys
Strict policy
• Explicit define
Roles that can
encrypt and
decrypt
Exclude
Admin Role
Prevent
Admin
Access

17. DynamoDB
Encryption at Transit and rest with KMS
This service runs outside VPC and doesn’t have a
resource policy like S3
Can’t block outside access
Add VPC endpoint condition to the KMS key policy,
and make description only from roles inside the vpc.
Backup/Recovery with AWS Backup
Load from S3 only works with empty database

18. S3
KMS encryption everywhere
Via VPC Endpoints to avoid public
internet
Resource policy that require traffic
through vpc endpoint
• Carefull not to lockout management api calls
• Redshift Spectrum doesn’t support vpc
endpoints

19. API
Gateway
Only Rest-API supports VPC endpoint
No HTTP (v2) Api Gateway
mTLS support only for public api-
gateway via the custom domain
(cloudfront)

20. Lambda
Minimal development
overhead
Great for security
Cold starts remain a
challenge

21. Fargate
Containers without complex infrastructure
Use Secret values to pass environment variables
directly from SSM parameter Store or Secret
Managers
Sidecars, combine containers
ECR Registry with
Inspectorv2 Deepscanning

22. Streaming
& mTLS
Lambda support only PBES1 for Kafka MTLS
(Des/md5/sha1 from 1990’s)
Eventbridge Pipe (same as lambda)
Glue Streaming
• Missing group.id
• Expensive
Limited Sink Connectors

23. Implementation

24. Transaction
datastore

25. Transaction
datastore
• Low-latency
• High-throughput
• Fault-tolerant

26. Write Once
Read Many
DynamoDB as (semi-)immutable storage

27. Streaming
prevents nested
API calls

28. Aggregate
data
preemptively

29. CQRS and
read-only
API

30. Streaming
with
containers
High
throughput:
1.5k msg/s
Near real-time
Autoscaling
Serverless
with Fargate

31. Fast &
Resilient
Kafka
consumers

32. Minimizing
resources
with reactive
programming

33. High throughput
streaming
• At-least-once delivery
• Out-of-order processing

34. Streaming
with
Lambda?
👍Batching
👍Non-real-time processing
👍Low volumes
👎Lacks good mTLS

35. REST API
Access
• OpenAPI template
• Separate mTLS proxy
• Still missing in App Mesh

36. Minimizing Lambda cold starts with Quarkus
AWS Lambda

37. Lessons
learned
with
Lambda
Minimize I/O
Don't chain Lambda functions
Use containers for complex
operations
Needs a mTLS proxy

38. OpenTelemetry
– Flexible
observability

39. Cost
• Only the first CloudTrail is free
• The CloudTrail cost is higher than the
workload. KMS/ DynamoDB audit lines
• DynamoDB Auto-scale is costly on
startup but low after that with billion
records. InfrequentAccess can save
even more.
• Spot and auto scale to 0, save cost on
the non-prod environments

40. Conclusion
SERVERLESS WORKS FOR A HIGH
TRAFFIC IMPORTANT FINANCIAL
APPLICATION
ABLE TO PARSE HUGE PEAK LOADS
WITH A GOOD MONTHLY COSTS
ERRORS / THROTTLING / TIMEOUTS
ARE VERY LIMITED WITH THE FAST
AUTOSCALE DYNAMODB.
GREAT PROJECT, PUSHING THE
TECHNOLOGY BOUNDARIES AND
COMPLEMENT EACH OTHER TEAMS'
STRENGTH.

41. Future

42. AWS AppSync

43. Advanced search
• Full-text fuzzy search
• NLP/AI search
• DynamoDB Streams

44. Technical improvements
Streaming with
Lambda
AWS Graviton
Lambda
SnapStart

45. Questions?
Connect with us!
https://www.linkedin.com/in/jacobverhoeks
https://www.linkedin.com/in/bmonhemius
Looking for a new job?