The PubSub model enables organizations to effectively create systems modeled on the concept of event publishers and subscribers. Leveraging a messaging service like Kafka or GCP PubSub is a method of developing and enabling the data application lifecycle given high throughput conditions. This talk will discuss practical tips for architecting and productionalizing scalable and latent data applications that leverage the PubSub model. Attendees will learn about common data messaging capabilities found through the PubSub model and how to leverage PubSub to optimize the performance of data systems with low-latency requirements.

1. How PubSub helped build Vox
Media’s multi-component,
high-volume data applications
Moses Musaelian | Vox Media
Current 2022

2. Introduction
● ML Engineer at Vox Media for over a year
● Working in Revenue Data Team

3. Introduction - You visit a site…

4. Bid Enrichment Flow

5. Bid Enrichment Flow - Trace

6. Google Pub/Sub - “message passing system for distributed applications in GCP”

7. Use Case #1: ML Service Design : “Latency Sensitive” Approach
Expect 800 bids per second!
Time out at 250ms!
“Lighten the load” off of our
front-facing API and let the
other components subscribe to
our message service

8. Use Case #2: DataFlow Logging (Viewability Data Application)
API Layer
Memorystore
Pubsub
Dataflow
BigQuery
Request Response

9. Use Case #3: Real Time Metrics from Pub/Sub (GCP Cloud Monitoring API v3)
Requirement → Don't write data to a
single time series faster than one point
each 5 seconds

10. Concluding Thoughts
API Layer
→ Metric send
→ Logging send
→ Worker send
API Layer
→ PubSub Send