A presentation given by Kishore Banala, Senior Software Engineer, Netflix, at our 2024 Austin API Summit, March 12-13. Session Description: Extend the advantages of GraphQL beyond the UI layer by creating data streams that seamlessly transfer data from Federated GraphQL to your preferred destination. This presentation explores the myriad use cases that can be unleashed, such as Search, Analytics etc., sparing you from the complexity of extensive ETL jobs. Join us for an in-depth exploration of the advantages that arise from seamlessly connecting GraphQL with data streams, opening new dimensions of efficiency and capability.

1. UNLEASHING THE
POTENTIAL OF GRAPHQL
WITH STREAMING DATA
Kishore Banala Austin API Summit 2024

2. About Me
● Senior Software Engineer at Netflix, 2018 - Present
● Middleware Infrastructures team
● We’re working at the intersection of Workflows, Data and recently
Ontologies to empower Netflix Content and Studio Engineering
● https://www.linkedin.com/in/kishore-banala/

3. The key ingredients of today’s talk

4. Federation with GraphQL
Movies Subgraph Reviews Subgraph

5. Sample Movies and Reviews data

6. CDC?
● Short for Change Data Capture
● Tracks and captures modifications in data in real-time.
● Eg. by reading MySQL binlog
● Facilitates data synchronization, replication, and ETL processes.
● Debezium, Netflix DBLog, Kafka Connect etc.

7. Sample CDC data..

8. CDC is awesome, but..
● What if we enrich the data before going to sink?
● Remove the tight coupling with source data schema?

9. CDC + GraphQL.. How?

10. CDC + GraphQL.. How?

11. CDC + GraphQL.. why?

12. Does it has to be federated?

13. How we benefited..
https://netflixtechblog.com/how-netflix-content-engineering-makes-a-federated-graph-searchable-5c0c1c7d7eaf

14. How we benefited..
https://netflixtechblog.com/data-movement-in-netflix-studio-via-data-mesh-3fddcceb1059

15. and..
● Building caches
● Columnar datastores for Analytics
dashboards
● Views in Hive
● “Data Products”

16. Considerations…
● Abstractions to net let your users accidentally abuse the system.
● Define best practice for Data hygiene.
● Consider leveraging Data Lineage tooling
● Make Data Products discoverable
● Restrict users to only move data they own.
○ Unless absolutely required, It can easily get out of hand, when users
move federated data from multiple GraphQL servers to a sink.

17. Thanks to my team and many others who have been working
on this for a while now.

18. Thank you!