Iterable is a marketing automation SaaS that operates at vast scale. Iterable processes a billion queue transactions per day representing the flow of users through programmatic marketing campaigns, which we call Workflow Campaigns. We're currently migrating this workload from RabbitMQ to Apache Pulsar. In this talk we'll go over how Workflow Campaigns work, and the features of Apache Pulsar that make it well-suited to this workload. Pulsar transactions ensure exactly once processing of these campaigns. Tiered storage allows us to keep a permanent record of every users' campaign history. The log of state transitions that we store in Pulsar topics allows us to create derived metrics for near-real-time analytics. Stream processing with Flink allows us to enforce simple count-based application limits on Workflows. Finally, we'll discuss the process of migration with no downtime.

1. Scalable Marketing Automation
Thomas Kim
Systems Architect, Iterable
Tom Wang
Staff Software Engineer, Iterable

2. OUR AGENDA
2
1. Background: Iterable’s Marketing Automation
Tools
2. Background: Current Implementation
3. Design on Pulsar
4. Flink and Beyond

3. Your Subtitle Goes Here (Delete if not needed)
Iterable’s Marketing Automation: “Workflows”
● Drag and drop interface to mix and match
filters and actions to make marketing
campaigns
● Both blast and real-time
3

4. Workflow Campaigns: Delay Nodes
4
Your Subtitle Goes Here (Delete if not needed)

5. Workflow Campaigns: Visitor Tracking
5
Your Subtitle Goes Here (Delete if not needed)
visitor id 1234 is here

6. Workflow Campaigns: UI Observability
6
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

7. 7
Workflow Campaigns: Entrance Limits

8. RabbitMQ
8
Your Subtitle Goes Here (Delete if not needed)

9. RabbitMQ Flow Control
9
Your Subtitle Goes Here (Delete if not needed)

10. Scale
10
● ~ 1B workflow node transitions every day
● ~ 100M new users entering into a workflow each day (double-check this?)
● At any point in time, hundreds of millions of users sitting somewhere in a workflow (in-flight, or in delay
nodes)
Your Subtitle Goes Here (Delete if not needed)

11. RabbitMQ + Redis + PG
11
Your Subtitle Goes Here (Delete if not needed)

12. Goal: Single Source of Truth
12
● Improves performance and correctness
Your Subtitle Goes Here (Delete if not needed)

13. ● Queue drives processing
● Implicitly holds truth of visitor tracking state
Phase 1: Drop-in Replacement
13
Your Subtitle Goes Here (Delete if not needed)

14. ● Already some benefits:
● Built-in support for scheduled messages
● Higher throughput
● Costs, compliance
Phase 1: Drop-in Replacement
14
Your Subtitle Goes Here (Delete if not needed)

15. Phase 2: Transactions
15
● Ack and re-enqueue need to be atomic
Your Subtitle Goes Here (Delete if not needed)

16. Phase 3: Derived UI State
16
Your Subtitle Goes Here (Delete if not needed)

17. Future: Flink and Beyond
17
● Use Flink, Pulsar Flink Connector, and Pulsar to process entrance limits
Your Subtitle Goes Here (Delete if not needed)

18. Entrance Limits with Flink
18
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

19. Recap
19
● Phased replacement of RabbitMQ and other data stores
● Improved correctness through transactions
Your Subtitle Goes Here (Delete if not needed)

20. Thanks
20
● Team members
● Iterable Pulsar team:
● Streamnative
Your Subtitle Goes Here (Delete if not needed)

21. 21
Questions?

22. YOUR TITLE GOES HERE
22
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

23. YOUR TITLE GOES HERE
23
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

24. YOUR TITLE GOES HERE
24
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

25. YOUR TITLE GOES HERE
25
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

26. YOUR TITLE GOES HERE
26
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

27. YOUR TITLE GOES HERE
27
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

28. YOUR TITLE GOES HERE
28
● Only need to use one column? Then this is the slide for you.
Your Subtitle Goes Here (Delete if not needed)

29. ● Are two columns more your style?
Then use this.
YOUR TITLE GOES HERE
Your Subtitle Goes Here (Delete if not needed)
29
● Are two columns more your style?
Then use this.

30. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Do you need three
columns instead of two?
Then use this.
● Do you need three
columns instead of two?
Then use this.
30
● Do you need three
columns instead of two?
Then use this.

31. YOUR DIVIDER SLIDE
TITLE GOES HERE
(MORE DIVIDER OPTIONS IN THE APPENDIX)

32. YOUR TITLE GOES HERE
Your Subtitle Goes Here (Delete if not needed)
● Use this slide if you have a stat that you
would like to use.
32
58%
This is where any info
pertaining to the stat
would go.
STAT CREDIT

33. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have two stats you want to use?
Then this is the slide for you.
33
58%
This is where any info
pertaining to the stat
would go.
STAT CREDIT
29%
This is where any info
pertaining to the stat
would go.
STAT CREDIT

34. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have stats that need to be displayed a little
differently? Use this slide if a bar graph is
more your style.
34
1 2 3 4

35. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have stats that need to be displayed a little
differently? Use this slide if a table is more
your style.
35
TITLE TITLE TITLE
Stat Stat Stat
Stat Stat Stat
Stat Stat Stat

36. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have an awesome image to go with your
content? Not a problem.
● Only want to use an image? Use the slide
after this!
36
REPLACE ME

37. REPLACE ME
37

38. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● One screenshot to show?
I’m the slide for you.
38
REPLACE ME

39. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● What about two?
39
REPLACE ME REPLACE ME

40. Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Is a laptop more your style? Don’t worry.
40
REPLACE ME

41. 41
REPLACE ME

42. 42
Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have some extra white space that you’d
like to fill up? Use the Iterable nodes!
● You can move, rotate, and
resize these images.

43. 43
Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have some extra white space that you’d
like to fill up? Use the Iterable nodes!
● You can move, rotate, and
resize these images.

44. 44
Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have some extra white space that you’d
like to fill up? Use the Iterable nodes!
● You can move, rotate, and
resize these images.

45. 45
Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have some extra white space that you’d
like to fill up? Use the Iterable nodes!
● You can move, rotate, and
resize these images.

46. 46
Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have some extra white space that you’d
like to fill up? Use the Iterable nodes!
● You can move, rotate, and
resize these images.

47. 47
Your Subtitle Goes Here (Delete if not needed)
YOUR TITLE GOES HERE
● Have some extra white space that you’d
like to fill up? Use the Iterable nodes!
● You can move, rotate, and
resize these images.

48. 48
This is where your quote would go.
You can adjust the height as necessary
to accommodate for anything
that’s longer than this.
First Name Last Name
Job Title, Company

49. Q&A / END SLIDE

50. APPENDIX

51. RED
#EF3D55
PURPLE
#6A266D
COOL GREEN
#59C1A7
CYAN
#34C3F2
LAVENDER
#C29DC9
MUSTARD
#F2CA1B
SALMON
#F36F4C
NOT BLACK
#4D4D4D
GREY
#999094
COLORS
51
Primary
Secondary
Grey

52. SEND TIME
OPTIMIZATION
MODERN
ARCHITECTURE
WEBINAR
RE-ENGAGE
MESSAGING
ANNOUNCEMENTS
CHANNEL
OPTIMIZATION
CAMPAIGNS
RESPONSE
FEED
DATA
TARGETED
MARKETING
TRANSACTIONAL
FREQUENCY
OPTIMIZATION
PERSONALIZATION
LOW-LEVEL
PERSONALIZATION
SEGMENTATION
SMS
CADENCE
SINGLE DATA
PLATFORM
GROWTH
AUDIENCE
CONNECTION ERROR
DESKTOP
QUOTE
ONLINE
SHOPPING
WORKFLOW
STUDIO
DATA
SILOS
AUDIENCE
CONNECTION
CONTENT
CHECKED
RE-ENGAGEMENT
TRUE
CROSS-CHANNEL
EMAIL
ERROR
SEARCH
MEDIUMS
HIGHLIGHT
AUTOMATION
EASY TO
USE
EMAIL
BRAND
LOVE
EVENT
DATA
WELCOME
USER PROFILE
DATA
1-TO-1
RELEVANCE
SMART
PHONE
CAMPAIGN
SUCCESS
DIRECTION
PROMOTIONS
VALUE
RED ICONS
52

53. SEND TIME
OPTIMIZATION
MODERN
ARCHITECTURE
WEBINAR
RE-ENGAGE
MESSAGING
ANNOUNCEMENTS
CHANNEL
OPTIMIZATION
CAMPAIGNS
RESPONSE
FEED
DATA
TARGETED
MARKETING
TRANSACTIONAL
FREQUENCY
OPTIMIZATION
PERSONALIZATION
LOW-LEVEL
PERSONALIZATION
SEGMENTATION
SMS
CADENCE
SINGLE DATA
PLATFORM
GROWTH
AUDIENCE
CONNECTION ERROR
DESKTOP
QUOTE
ONLINE
SHOPPING
WORKFLOW
STUDIO
DATA
SILOS
AUDIENCE
CONNECTION
CONTENT
CHECKED
RE-ENGAGEMENT
TRUE
CROSS-CHANNEL
EMAIL
ERROR
SEARCH
MEDIUMS
HIGHLIGHT
AUTOMATION
EASY TO
USE
EMAIL
BRAND
LOVE
EVENT
DATA
WELCOME
USER PROFILE
DATA
1-TO-1
RELEVANCE
SMART
PHONE
CAMPAIGN
SUCCESS
DIRECTION
PROMOTIONS
VALUE
PURPLE ICONS
53

54. SEND TIME
OPTIMIZATION
MODERN
ARCHITECTURE
WEBINAR
RE-ENGAGE
MESSAGING
ANNOUNCEMENTS
CHANNEL
OPTIMIZATION
CAMPAIGNS
RESPONSE
FEED
DATA
TARGETED
MARKETING
TRANSACTIONAL
FREQUENCY
OPTIMIZATION
PERSONALIZATION
LOW-LEVEL
PERSONALIZATION
SEGMENTATION
SMS
CADENCE
SINGLE DATA
PLATFORM
GROWTH
AUDIENCE
CONNECTION ERROR
DESKTOP
QUOTE
ONLINE
SHOPPING
WORKFLOW
STUDIO
DATA
SILOS
AUDIENCE
CONNECTION
CONTENT
CHECKED
RE-ENGAGEMENT
TRUE
CROSS-CHANNEL
EMAIL
ERROR
SEARCH
MEDIUMS
HIGHLIGHT
AUTOMATION
EASY TO
USE
EMAIL
BRAND
LOVE
EVENT
DATA
WELCOME
USER PROFILE
DATA
1-TO-1
RELEVANCE
SMART
PHONE
CAMPAIGN
SUCCESS
DIRECTION
PROMOTIONS
VALUE
COOL GREEN ICONS
54

55. SEND TIME
OPTIMIZATION
MODERN
ARCHITECTURE
WEBINAR
RE-ENGAGE
MESSAGING
ANNOUNCEMENTS
CHANNEL
OPTIMIZATION
CAMPAIGNS
RESPONSE
FEED
DATA
TARGETED
MARKETING
TRANSACTIONAL
FREQUENCY
OPTIMIZATION
PERSONALIZATION
LOW-LEVEL
PERSONALIZATION
SEGMENTATION
SMS
CADENCE
SINGLE DATA
PLATFORM
GROWTH
AUDIENCE
CONNECTION ERROR
DESKTOP
QUOTE
ONLINE
SHOPPING
WORKFLOW
STUDIO
DATA
SILOS
AUDIENCE
CONNECTION
CONTENT
CHECKED
RE-ENGAGEMENT
TRUE
CROSS-CHANNEL
EMAIL
ERROR
SEARCH
MEDIUMS
HIGHLIGHT
AUTOMATION
EASY TO
USE
EMAIL
BRAND
LOVE
EVENT
DATA
WELCOME
USER PROFILE
DATA
1-TO-1
RELEVANCE
SMART
PHONE
CAMPAIGN
SUCCESS
DIRECTION
PROMOTIONS
VALUE
CYAN ICONS
55

56. SEND TIME
OPTIMIZATION
MODERN
ARCHITECTURE
WEBINAR
RE-ENGAGE
MESSAGING
ANNOUNCEMENTS
CHANNEL
OPTIMIZATION
CAMPAIGNS
RESPONSE
FEED
DATA
TARGETED
MARKETING
TRANSACTIONAL
FREQUENCY
OPTIMIZATION
PERSONALIZATION
LOW-LEVEL
PERSONALIZATION
SEGMENTATION
SMS
CADENCE
SINGLE DATA
PLATFORM
GROWTH
AUDIENCE
CONNECTION ERROR
DESKTOP
QUOTE
ONLINE
SHOPPING
WORKFLOW
STUDIO
DATA
SILOS
AUDIENCE
CONNECTION
CONTENT
CHECKED
RE-ENGAGEMENT
TRUE
CROSS-CHANNEL
EMAIL
ERROR
SEARCH
MEDIUMS
HIGHLIGHT
AUTOMATION
EASY TO
USE
EMAIL
BRAND
LOVE
EVENT
DATA
WELCOME
USER PROFILE
DATA
1-TO-1
RELEVANCE
SMART
PHONE
CAMPAIGN
SUCCESS
DIRECTION
PROMOTIONS
VALUE
GREY ICONS
56

57. WHITE ICONS
57
SEND TIME
OPTIMIZATION
MODERN
ARCHITECTURE
WEBINAR
RE-ENGAGE
MESSAGING
ANNOUNCEMENTS
CHANNEL
OPTIMIZATION
CAMPAIGNS
RESPONSE
FEED
DATA
TARGETED
MARKETING
TRANSACTIONAL
FREQUENCY
OPTIMIZATION
PERSONALIZATION
LOW-LEVEL
PERSONALIZATION
SEGMENTATION
SMS
CADENCE
SINGLE DATA
PLATFORM
GROWTH
AUDIENCE
CONNECTION ERROR
DESKTOP
QUOTE
ONLINE
SHOPPING
WORKFLOW
STUDIO
DATA
SILOS
AUDIENCE
CONNECTION
CONTENT
CHECKED
RE-ENGAGEMENT
TRUE
CROSS-CHANNEL
EMAIL
ERROR
SEARCH
MEDIUMS
HIGHLIGHT
AUTOMATION
EASY TO
USE
EMAIL
BRAND
LOVE
EVENT
DATA
WELCOME
USER PROFILE
DATA
1-TO-1
RELEVANCE
SMART
PHONE
CAMPAIGN
SUCCESS
DIRECTION
PROMOTIONS
VALUE

58. DIVIDER #1

59. DIVIDER #2

60. DIVIDER #3

61. DIVIDER #4

62. DIVIDER #5

63. DIVIDER #6

64. DIVIDER #7

65. DIVIDER #8

66. DIVIDER #9

67. DIVIDER #10

68. DIVIDER #11

69. DIVIDER #12

70. DIVIDER #13

71. THANK YOU!