The document discusses Recursion Pharmaceuticals' migration of their image analysis pipeline from an on-premise Kafka Streams solution to a cloud-based workflow using Dagger, a workflow library built on Kafka Streams. The pipeline processes large volumes of high-throughput cell imaging data to extract features at different levels and perform downstream analysis. The migration addressed issues with the original system's batch processing approach and lack of real-time feedback. Examples are provided of implementing a sample workflow to extract site-level features using both low-level Kafka Streams APIs and the higher-level Dagger abstraction.

1. Ben Mabey
VP of Engineering
@bmabey
Discovering Drugs
with Kafka Streams
Scott Nielsen
Director of Data Engineering
K A F K A S U M M I T S F 2 0 1 9

2. Penn Teller

3. Penn TellerB
Scott

4. Decoding Biology
to Radically Improve Lives

5. © 2017 Recursion Pharmaceuticals
1000s of untreated
genetic diseases
Photo of our wall?

6. 0.00001
0.0001
0.001
0.01
0.1
1
10
100
1000197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000200120022003200420052006200720082009201020112012201320142015
Transistor Area (% of 1970 values)
Moore’s Law

7. 0.00001
0.0001
0.001
0.01
0.1
1
10
100
1000197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000200120022003200420052006200720082009201020112012201320142015
Transistor Area (% of 1970 values)
1
10
100
1971197219731974197519761977197819791980198119821983198419851986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010
R&D Spend / Drug (% of 2007 values)
Moore’s Law

8. 0.00001
0.0001
0.001
0.01
0.1
1
10
100
1000197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000200120022003200420052006200720082009201020112012201320142015
Transistor Area (% of 1970 values)
1
10
100
1971197219731974197519761977197819791980198119821983198419851986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010
R&D Spend / Drug (% of 2007 values)
Moore’s Law
Eroom’s Law

9. 0
10
20
30
40
50
60
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
Number of Drugs Approved in US (1993-2016)

10. How can we fix this?

11. RecursionPharma.com

12. RecursionPharma.com
Over 7 million per week

13. RecursionPharma.com
hoechst (DNA)

14. RecursionPharma.com
concanavalin A (ER)

15. RecursionPharma.com
mitotracker (mitochondria)

16. RecursionPharma.com
WGA (golgi apparatus, cell membrane)

17. RecursionPharma.com
SYTO 14 (RNA, nucleoli)

18. RecursionPharma.com
phalloidin (actin fibers)

19. RecursionPharma.com
combined

20. How do these pretty
pictures help?

21. Healthy child
Child with rare
genetic disease
(Cornelia de Lange
Syndrome)

22. Healthy child Healthy cells
Child with rare
genetic disease
(Cornelia de Lange
Syndrome)
Genetic disease
model cells
(Cornelia de Lange
Syndrome)

23. Healthy Disease

24. Healthy Disease Disease + Drug?

25. Public Dataset: http://rxrx.ai
Nature Article
Machine learning brings cell imaging promises into focus
https://tinyurl.com/ml-cells
Learn more…

26. How is this data
produced?

27. 308 wells/plate

28. 4 sites/well
308 wells/plate

29. 6 channels (images)/site
7,392 images per plate
4 sites/well
308 wells/plate

30. 6 channels (images)/site
7,392 images per plate
4 sites/well
308 wells/plate
~69GB per plate

31. Experiment A Experiment B Experiment C
Experiment D

32. Our “Series A” System

33. On-Premise

34. On-Premise

35. Stream images to S3
On-Premise

36. Generate thumbnails
Image metrics
Stream images to S3
On-Premise

37. Generate thumbnails
Image metrics
Stream images to S3
On-Premise

38. Generate thumbnails
Image metrics
Fire and forget
Stream images to S3
On-Premise

39. Generate thumbnails
Image metrics
Fire and forget
Experiment A
Stream images to S3
On-Premise

40. Generate thumbnails
Image metrics
Fire and forget
Experiment A
Stream images to S3
Extract Features
On-Premise
Process experiments
in batch

41. Generate thumbnails
Image metrics
Fire and forget
Stream images to S3
Extract Features
On-Premise
Process experiments
in batch

42. Generate thumbnails
Image metrics
Fire and forget
Stream images to S3
Extract Features
metrics, models,
reports, etc
On-Premise
Process experiments
in batch

43. Generate thumbnails
Image metrics
Fire and forget
Stream images to S3
Extract Features
metrics, models,
reports, etc
On-Premise
Process experiments
in batch

44. Generate thumbnails
Image metrics
Fire and forget
Stream images to S3
Extract Features
metrics, models,
reports, etc
On-Premise
Process experiments
in batch

45. Traditional, low throughput, biology

46. Traditional, low throughput, biology
~6-12 plates per week, ~400-800GB

47. © 2017 Recursion Pharmaceuticals
High-throughput
experiments
Robots
photo

51. 100
6.9TB

52. 100
6.9TB
300
20TB

53. 100
6.9TB
300
20TB
Kafka Streams solution
was launched

54. 100
6.9TB
300
20TB
700
48TB
1,300
90TB
1,700
118TB
1,900
132 TB
Kafka Streams solution
was launched

57. 100
6.9TB
300
20TB
700
48TB
1,300
90TB
1,700
118TB
1,900
132 TB

58. 100
6.9TB
300
20TB
700
48TB
1,300
90TB
1,700
118TB
1,900
132 TB
280 TB
Today

59. So what was wrong with the
original system?

60. Generate thumbnails
Image metrics
Extract Features
metrics, models,
reports, etc
On-Premise
Process experiments
in batch

61. Experiment A Experiment B Experiment C
Experiment D
Plates are not imaged in order

63. Migration Goals

64. Migration Goals
Move orchestration and processing to cloud.

65. Migration Goals
Move orchestration and processing to cloud.

66. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.

67. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.

68. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.
Preserve existing micro-services logic.

69. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.
Preserve existing micro-services logic.
Make cheaper.

70. Let’s take a look at the
logical pipeline that we
needed to implement…

71. Images / channel level

72. Images / channel level
image level metrics

73. Images / channel level
site (all channels/images)
thumbnails
image level metrics

74. Images / channel level
site (all channels/images)
thumbnails
site level features
image level metrics

75. Images / channel level
site (all channels/images)
thumbnails
site level features
image level metrics

76. Images / channel level
site (all channels/images)
thumbnails
site level features
image level metrics
site metrics

77. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
image level metrics
site metrics

78. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
image level metrics
site metricsmetrics

79. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
image level metrics
site metricsmetrics
plate level features metrics

80. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
experiment features
image level metrics
site metricsmetrics
plate level features metrics
Experiment A

81. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
experiment features
image level metrics
site metricsmetrics
plate level features metrics
metrics, models,
reports, etc
Experiment A

85. Kafka Streams
was just released…

86. Kafka Streams
was just released…

87. 🗡 dagger 🗡
workflow library
written on top of Kafka Streams
that orchestrates microservices

88. 🗡 dagger 🗡
workflow library
written on top of Kafka Streams
that orchestrates microservices
Dagger, ya know, because
it is all about the workflows
represented as directed
acyclic graphs, i.e. DAGs.

89. 🗡 dagger 🗡
workflow library
written on top of Kafka Streams
that orchestrates microservices

90. New workflow system in 2017?

91. New workflow system in 2017?
Not Invented Here syndrome?

92. Core logic in library
is ~2800 LOC
New workflow system in 2017?
Not Invented Here syndrome?

93. Core logic in library
is ~2800 LOC
All of our our DAGs,
including schema, task,
and workflow definition
~1700 LOC
New workflow system in 2017?
Not Invented Here syndrome?

94. Core logic in library
is ~2800 LOC
All of our our DAGs,
including schema, task,
and workflow definition
~1700 LOC
New workflow system in 2017?
Not Invented Here syndrome?

95. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
experiment features
image level metrics
site metricsmetrics
plate level features metrics
metrics, models,
reports, etc

96. Let’s look at a small workflow
using Kafka Streams initially…

97. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream

98. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
final KTable<String, ExperimentMetadata> experimentMetadata = builder.table(
EXPERIMENT_METADATA_TOPIC);
final KStream<String, ChannelLevel> images = builder.stream(
CHANNEL_IMAGES_TOPIC);
final KStream<String, Site> sites = images
.groupBy((exp, channel) -> channel.site())
.windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS)))
.aggregate(
() -> new AggState(),
(site, channel, agg) -> agg.observe(channel.site(), channel.channel),
(site, agg_a, agg_b) -> agg_a.merge(agg_b))
.join(experimentMetadata,
(agg, expMeta) -> agg.markCompleted(expMeta.numChannels))
.filterValues(agg -> agg.isComplete())
.mapValues(agg -> agg.site());
sites.to(SITE_IMAGES_TOPIC);

99. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
final KTable<String, ExperimentMetadata> experimentMetadata = builder.table(
EXPERIMENT_METADATA_TOPIC);
final KStream<String, ChannelLevel> images = builder.stream(
CHANNEL_IMAGES_TOPIC);
final KStream<String, Site> sites = images
.groupBy((exp, channel) -> channel.site())
.windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS)))
.aggregate(
() -> new AggState(),
(site, channel, agg) -> agg.observe(channel.site(), channel.channel),
(site, agg_a, agg_b) -> agg_a.merge(agg_b))
.join(experimentMetadata,
(agg, expMeta) -> agg.markCompleted(expMeta.numChannels))
.filterValues(agg -> agg.isComplete())
.mapValues(agg -> agg.site());
sites.to(SITE_IMAGES_TOPIC);

100. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
final KTable<String, ExperimentMetadata> experimentMetadata = builder.table(
EXPERIMENT_METADATA_TOPIC);
final KStream<String, ChannelLevel> images = builder.stream(
CHANNEL_IMAGES_TOPIC);
final KStream<String, Site> sites = images
.groupBy((exp, channel) -> channel.site())
.windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS)))
.aggregate(
() -> new AggState(),
(site, channel, agg) -> agg.observe(channel.site(), channel.channel),
(site, agg_a, agg_b) -> agg_a.merge(agg_b))
.join(experimentMetadata,
(agg, expMeta) -> agg.markCompleted(expMeta.numChannels))
.filterValues(agg -> agg.isComplete())
.mapValues(agg -> agg.site());
sites.to(SITE_IMAGES_TOPIC);

101. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
final KTable<String, ExperimentMetadata> experimentMetadata = builder.table(
EXPERIMENT_METADATA_TOPIC);
final KStream<String, ChannelLevel> images = builder.stream(
CHANNEL_IMAGES_TOPIC);
final KStream<String, Site> sites = images
.groupBy((exp, channel) -> channel.site())
.windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS)))
.aggregate(
() -> new AggState(),
(site, channel, agg) -> agg.observe(channel.site(), channel.channel),
(site, agg_a, agg_b) -> agg_a.merge(agg_b))
.join(experimentMetadata,
(agg, expMeta) -> agg.markCompleted(expMeta.numChannels))
.filterValues(agg -> agg.isComplete())
.mapValues(agg -> agg.site());
sites.to(SITE_IMAGES_TOPIC);

102. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
final KTable<String, ExperimentMetadata> experimentMetadata = builder.table(
EXPERIMENT_METADATA_TOPIC);
final KStream<String, ChannelLevel> images = builder.stream(
CHANNEL_IMAGES_TOPIC);
final KStream<String, Site> sites = images
.groupBy((exp, channel) -> channel.site())
.windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS)))
.aggregate(
() -> new AggState(),
(site, channel, agg) -> agg.observe(channel.site(), channel.channel),
(site, agg_a, agg_b) -> agg_a.merge(agg_b))
.join(experimentMetadata,
(agg, expMeta) -> agg.markCompleted(expMeta.numChannels))
.filterValues(agg -> agg.isComplete())
.mapValues(agg -> agg.site());
sites.to(SITE_IMAGES_TOPIC);

103. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
final KTable<String, ExperimentMetadata> experimentMetadata = builder.table(
EXPERIMENT_METADATA_TOPIC);
final KStream<String, ChannelLevel> images = builder.stream(
CHANNEL_IMAGES_TOPIC);
final KStream<String, Site> sites = images
.groupBy((exp, channel) -> channel.site())
.windowedBy(SessionWindows.with(Duration.ofHours(SESSION_WINDOW_HOURS)))
.aggregate(
() -> new AggState(),
(site, channel, agg) -> agg.observe(channel.site(), channel.channel),
(site, agg_a, agg_b) -> agg_a.merge(agg_b))
.join(experimentMetadata,
(agg, expMeta) -> agg.markCompleted(expMeta.numChannels))
.filterValues(agg -> agg.isComplete())
.mapValues(agg -> agg.site());
sites.to(SITE_IMAGES_TOPIC);

104. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Kafka Streams App External Service
task input topic

105. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Kafka Streams App External Service
task input topic

106. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Kafka Streams App External Service
task input topic
task output topic

107. How would you do the same
workflow in dagger?

108. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream

109. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Input topics & tables

110. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Input topics & tables
Stream operations

111. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Input topics & tables
Stream operations
Tasks

112. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
Input topics & tables
Stream operations
Tasks
Output topics

113. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

114. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

115. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

116. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

117. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

118. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

119. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}
Specify function to be used

120. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

121. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

122. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

123. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

124. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream
{"name": "extract-site-level-features",
"graph":
{"images-channel":
{"type": "topic-stream", "topic-name": "images_channels"}
"experiment-metadata":
{"type": "topic-table", "topic-name": "experiment_metadata"},
"images-site":
{"type": "stream-operation",
"key-schema": "long", "value-schema": "job_site_level",
"inputs": ["images-channel", "experiment-metadata"],
"function": "aggregations/images-site-grouping"},
"features-site":
{"type": "external-task",
"stream": "images-site",
"task-name": "extract-features"},
"features-output":
{"type": "publish",
"topic-name": "extracted_features",
"stream": "features-site"}}}

125. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
{:name "extract-site-level-features",
:graph
{:images-channel
{:type :topic-stream, :topic-name "images_channels"},
:experiment-metadata
{:type :topic-table, :topic-name "experiment_metadata"},
:images-site
{:type :stream-operation,
:key-schema :long, :value-schema "job_site_level",
:inputs [:images-channel, :experiment-metadata],
:function (fn [images-channel experiment-metadata] …),
:features-site
{:type :external-task,
:task-name "extract-features",
:stream :images-site},
:features-output
{:type :publish,
:stream :features-site,
:topic-name "extracted_features"}}}
images_site stream

126. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
{:name "extract-site-level-features",
:graph
{:images-channel
{:type :topic-stream, :topic-name "images_channels"},
:experiment-metadata
{:type :topic-table, :topic-name "experiment_metadata"},
:images-site
{:type :stream-operation,
:key-schema :long, :value-schema "job_site_level",
:inputs [:images-channel, :experiment-metadata],
:function (fn [images-channel experiment-metadata] …),
:features-site
{:type :external-task,
:task-name "extract-features",
:stream :images-site},
:features-output
{:type :publish,
:stream :features-site,
:topic-name "extracted_features"}}}
images_site stream

127. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
{:name "extract-site-level-features",
:graph
{:images-channel
{:type :topic-stream, :topic-name "images_channels"},
:experiment-metadata
{:type :topic-table, :topic-name "experiment_metadata"},
:images-site
{:type :stream-operation,
:key-schema :long, :value-schema "job_site_level",
:inputs [:images-channel, :experiment-metadata],
:function (fn [images-channel experiment-metadata] …),
:features-site
{:type :external-task,
:task-name "extract-features",
:stream :images-site},
:features-output
{:type :publish,
:stream :features-site,
:topic-name "extracted_features"}}}
Inline function directly
images_site stream

128. extract site features
images_channel topic
experiment_metadata topic table
extracted_features topic
images_site stream

129. 🗡( )
Dagger is a compiler

130. 🗡( ) Kafka Streams
Topology
Dagger is a compiler

131. 🗡( ) Kafka Streams
Topology
Dagger is a compiler

132. What would the entire pipeline
look like in dagger?

133. well level features
Images / channel level
site (all channels/images)
thumbnails
site level features
experiment features
image level metrics
site metricsmetrics
plate level features metrics
metrics, models,
reports, etc

135. 🗡 🗡
Our pipeline application that uses Dagger

136. How does the whole system
look like now?

137. Generate thumbnails
Image metrics
Extract Features
metrics, models,
reports, etc
On-Premise
Process experiments
in batch

138. On-Premise

139. On-Premise
Publish Image Events

140. On-Premise
Publish Image Events

141. On-Premise
🗡
🗡
Publish Image Events
Uploader

142. On-Premise
🗡
🗡
Publish Image Events
Uploader
dagger is used here too!

143. On-Premise
🗡
🗡
Publish Image Events
Uploader

144. On-Premise
🗡
🗡
Publish Image Events
Uploader

145. On-Premise
🗡
🗡
🗡 🗡
Publish Image Events
Uploader

146. On-Premise
🗡
🗡
🗡 🗡
Autoscaled WorkersPublish Image Events
Uploader

147. On-Premise
🗡
🗡
🗡 🗡
Microservices
Publishers & Consumers
Autoscaled WorkersPublish Image Events
Uploader

148. On-Premise
🗡
🗡
🗡 🗡
BigQuerySQL
🗡Transform & Load
🗡
Microservices
Publishers & Consumers
Autoscaled WorkersPublish Image Events
Uploader

149. Migration Goals

150. Migration Goals
Move orchestration and processing to cloud.✓

151. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.
✓
✓

152. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.
Preserve existing micro-services logic.
✓
✓
✓

153. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.
Preserve existing micro-services logic.
Make cheaper.
✓
✓
✓
✓

154. Migration Goals
Move orchestration and processing to cloud.
Faster feedback and less bursty workloads.
Preserve existing micro-services logic.
Make cheaper.
✓
✓
✓
✓
EC2 and Lambda -> Google Clould preemptibles.

155. Big data, small metadata…

156. Big data, small metadata…

157. Lessons learned…

158. Early Adopter Tax

159. Missed out on mature
workflow monitoring

162. On-Premise
🗡
🗡
🗡 🗡
🗡Transform & Load
🗡
Uploader
Easy deployment!

163. Kafka Streams App External Service
task input topic
task output topic
Durable Log FTW

164. Thank you!

165. 🗡

166. Come help us decode biology!
@RecursionPharma @bmabey