In Data Engineer's Lunch #44, we will discuss Prefect and how it compares to Airflow when scheduling tasks. Accompanying Blog: https://blog.anant.us/data-engineers-lunch-44-prefect-for-workflow-management/ Accompanying YouTube: https://youtu.be/P184heuv8ws Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

1. Version 1.0
Prefect
In Data Engineer's Lunch #44, we will discuss Prefect and how
it compares to Airflowwhen scheduling tasks alongside a demo
from their documentation.
Josh Barnes
Jr. Engineer @ Anant

2. Prefect
● Workflow Management System
○ Retries, Logging, Dynamic Mapping, Caching, Failure Notifications
● Free to start: 10,000 tasks/month
● Prefect takes your code and transforms it into a robust, distributed pipeline.
○ Continue using existing tools, languages, infrastructure, and scripts.
○ Prefect builds a fully introspectable and customizable DAG definition for every
workflow, users are never required to interact with it if they don't want to. Instead,
Python is the API. Users define functions and call them as they would in any script, and
Prefect does the work to figure out the workflow structure.
● Prefect has more unit tests and greater test coverage than any other workflow engine,
including the entire Airflow platform.
● UI to visualize every aspect your workflow and even run off schedule tasks trying altered
variables.

3. Workflows and Tasks
● Workflows “Flows” represent the dependency structure between tasks, but do not
perform logic
● Prefect tasks are functions that have rules about when they should run.
○ Process data, Orchestrate external systems, call out to other environments or
languages.
○ Decorate a python function with @task and it’s been created.
○ Receives metadata about upstream dependencies prior to running.
○ Code agnostic without restriction to inputs or outputs.
● Both Tasks and Workflows produce State to reflect the behavior at any time.
● Designed to run at any time with concurrency.
● https://docs.prefect.io/core/task_library/overview.html#task-library-in-action

4. Where Prefect Improves upon Airflow
● DAGs which need to be run off-schedule or with no schedule at all
● DAGs that run concurrently with the same start time
● DAGs with complicated branching logic
● DAGs with many fast tasks
● DAGs which rely on the exchange of data
● Parametrized DAGs
● Dynamic DAGs

5. Flow Diagram

6. Demo
● https://github.com/PrefectHQ/prefect
● Simple Flow Deployed to Prefect Cloud

7. Resources
● https://docs.prefect.io/core/

8. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037