Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community!

1. Dynamic Change Data Capture with Flink CDC and
Consistent Hashing

2. Hi!
Xiao Meng
● Software Engineer @ Goldsky
● Expert Data Engineer @ Activision
● Data, Infrastructure, and SRE
Yaroslav Tkachenko
● Founding Engineer @ Goldsky
● Staff Data Engineer @ Shopify
● Software Architect @ Activision
● Kafka, Flink, and all things
streaming!

3. Goldsky

4. Use case
➢ goldsky subgraph deploy poap-subgraph/1.0.0 --path .
➢ goldsky pipeline create def.json poap-subgraph-pipeline

5. Indexing Blockchain Data
Thousands of Schemas!

6. Postgres CDC 101

7. Postgres CDC 101

8. Our Requirements
● Replication Slot Management
○ RDS Aurora: default 20
○ One slot to rule them all: hard to scale
○ One slot per table/job: operational burden
● Snapshot should be lock-free and
scalable.
● Dynamic Schema
○ self-serve deployments generate new schema
and tables all the time

9. Consistent Hashing & Sharing Replication Slots

10. Our Requirements
● ✅ Replication Slot Management
● Lock-Free and Parallel Snapshot
● Scan Newly Added Tables

11. The Journey: Flink-CDC & Goldsky Timeline
2022
Release 2.3
Oracle & MongoDB CDC
Connector
Exploration
Flink-CDC 2.2
Incremental Snapshot
Framework based on MySQL
(flink-cdc base)
Debezium 1.6.4
Postgres CDC Connector PR
Scan Newly Added Tables PR
In Production
config-file driven
by our engineers
Scan Newly Added Tables PR
Self-Serve CDC
directly driven by
end-users!
2023
Mar
Oct
Nov
Dec
Jun
Jan
Release 2.4
Postgres CDC PR Merged🎉
Debezium 1.9.7
Roadmap 2.5
Sep

12. Flink-CDC: SQL/DataStream API

13. Foundation: FLIP-27 Source Interface & CDC-base

14. Implementation

15. Troubleshooting async code

17. Well, things didn’t work!
We noticed that the connector tried to use the
replication slot during snapshotting… But why?

18. Algorithm:
1. stop live event processing
2. read next chunk
3. resume live event processing
4. reconcile the events using
watermarks

19. Postgres IS challenges
Snapshot task can create a live replication one!
● PostgresScanFetchTask →
● SnapshotSplitReadTask (read chunk)
● ❓PostgresStreamFetchTask (execute backfill)
● SnapshotSplitReadTask (read chunk)
● ❓PostgresStreamFetchTask (execute backfill)
● SnapshotSplitReadTask (read chunk)
● …

20. Postgres IS challenges
Backfills are hard to scale.
● Replication slot can’t be re-used between snapshot and live modes (it’s
already active).
● When parallelism > 1, multiple snapshot tasks will try to use the same slot
concurrently.
● It’s SLOW! Unbearably. Backfills require seeking a specific LSN in the WAL
and it’s very slow when it happens after reading every chunk.

21. Postgres IS challenges
Backfills are hard to scale.
● You could create a backfill slot per task, but:
○ It’s still very slow.
○ You could end up with too many slots! And think about their lifecycle.
○ Can be brittle for large snapshots.

22. Postgres IS challenges

23. Flink to the rescue!
What if we re-implement some parts of the
incremental snapshotting algorithm with Flink?

24. Postgres
CDC source
[snapshot
mode]
Postgres
CDC source
[live mode]
Reconcile

25. Flink to the rescue!

26. Flink to the rescue!

27. Flink to the rescue!
Reconciliation example:
➢ K1: ✅ INSERT (L), ✅ UPDATE (L)
➢ K2: ✅ UPDATE(L), ❌ READ (S), …, ✅ UPDATE(L)
➢ K3: ✅ READ (S), ✅ UPDATE (L)

28. Flink to the rescue!

29. Results
● Fully self-serve, customer-driven
○ No need to know about Kafka, Kafka Connect, Debezium, CDC, data
pipelines, Postgres replication slots, etc…
● 400+ tables being ingested
● < 3sec p99 end-to-end latency

30. Key Takeaways
● Maintaining replication slots can be tricky!
○ Ensure you have the right monitoring & alerting.
● Use consistent hashing for managing a pool of resources.
● Flink has many low-level building blocks that can be used to customize any
workflow.
○ You can build your own flavour of incremental snapshotting!

31. Links
● DBLog: A Watermark Based Change-Data-Capture Framework
● Flink CDC
○ Demo: Streaming ETL for MySQL and Postgres with Flink CDC
○ Postgres Incremental Snapshotting PR
○ Postgres Connector

32. Bonus: dealing with sparse updates
● db1_customerA <= many writes
● db1_customerB <= few writes
● db2_customerC <= no writes
To “unlock” the third database:
> SELECT * FROM pg_logical_emit_message(true, 'heartbeat', 'hello');

33. Questions?
@sap1ens @xiaomeng