Orchestration is more than just scheduling. Just like an orchestra is coordinated by a conductor, a data team needs a centralized viewpoint of all the tools, workflows, and interactions that make up a data stack. In this talk, we'll define orchestration by what it enables for a data team while walking through the steps to easily coordinate workflows in a way that is both scalable and versatile. Watch to learn the steps to successfully deploy orchestration internally – including who to involve, what infrastructure to think about, and how to plan for failures. This talk was given at the Modern Data Stack Conference 2023 by Sarah Krasnik, Growth Lead at Prefect. To watch the talk, visit: https://youtu.be/0H_JEDo3dcQ To learn more about Prefect, visit: https://prefect.io

1. Orchestration
Education
Station
Sarah Krasnik
Prefect
MDSCon 2023

2. Agenda
Requirements of an
orchestrator
How to reliably implement
orchestration
01
02

3. Good orchestration
is not this
Source: Cartoon Box 301 by Frame Order
Then what is it?

4. Orchestration
Scheduling
What features
does an
orchestrator
need to have?

5. Orchestration
Scheduling
Failure
recovery
Logging
Infrastructure
Global
variables
Pass
dynamic data
Permissions
Dependency
management
Notifications
Features of an
MVP
Features to
produce delight
Connect to
other tools

6. You sit down to check out an
orchestrator. Now what?

7. Steps to onboard an orchestrator
1. Involve dependent teams and tools
2. Think about infrastructure and security early
3. Build for both production and local development
4. Plan for failures
Practical advice to build and deploy production workflows

8. Involve dependent teams and tools
Potential pitfalls
● Assuming orchestrators only influence the data team.
● Not wiring up the orchestrator with existing tools early in
development.
Actions for success
● Gather requirements from adjacent teams early.
● Ensure you can connect to related tools.
ETL
Transform &
store
Operationalize
Data science
…and more.

9. Infrastructure and security
Potential pitfalls
● Offloading infrastructure concerns to devops without
context
● Assuming only 1 person needs access forever
Actions for success
● Think about the production use case early and where
dataflows are going to run
● Comply with your organizational security policies

10. Build for both prod and dev
Potential pitfalls
● Testing only in production, forever
● Assuming only 1 person will ever need access
Actions for success
● Make it easy to develop locally, test automatically
● Comply with your organizational security policies

11. Plan for failures
Potential pitfalls
● Using private messages in Slack to debug
● Not setting up smart retries
Actions for success
● Automate failure alerting, looping in other teams
● Write idempotent code, documenting how to triage
failures

12. Thank you
Check out Prefect
www.prefect.io