SlideShare a Scribd company logo
1 of 6
Python Dependencies
The Right* Way
Bazel + Python @ Beeswax
Ron Rothman
* For us. YMMV.
● 30 engineers
● monorepo
● 100 applications
● 300 kLoPyC
1.Global requirements.txt
2.rules_python
3.pipenv
We have: We tried:
● bazel test
● bazel run
● deployments
We need deps for:
Bazel
pip-tools
Virtualenvs
Wheels
Beeve
lockfiles
# requirements.lock
boto3==1.10.2
botocore==1.13.2 # via boto3, s3transfer
certifi==2019.9.11 # via requests
chardet==3.0.4 # via requests
docutils==0.15.2 # via botocore
futures==3.3.0 # via s3transfer
idna==2.8 # via requests
jmespath==0.9.4 # via boto3, botocore
python-dateutil==2.8.0 # via botocore
requests==2.22.0
s3transfer==0.2.1 # via boto3
six==1.12.0 # via python-dateutil
urllib3==1.25.6 # via botocore,
requests
py_binary(
name = "my_app",
deps = ["lib_a"],
requires = ["requests>=2.21"],
)
py_library(
name = "lib_a",
deps = ["lib_b"],
requires = ["requests>=2.18"],
)
py_library(
name = "lib_b",
deps = ["lib_c"],
requires = ["boto3~=1.9"],
)
py_venv(
name = "my_app_env",
targets = [":my_app"],
)
# requirements.in
boto3~=1.9
requests>=2.18
requests>=2.21
Needs improvement:
● Dev/CI workflow more complicated than before
● Bazel cache misses
● Dependency changes are not propagated automatically
Working well:
● 100% correct (and minimal) dependencies!
● Monorepo-friendly
● History of dependency changes (lock files)
● Wheels are easy
Thank You 🙏🏻
ron {at} beeswax.com

More Related Content

What's hot

Gitosis on Mac OS X Server
Gitosis on Mac OS X ServerGitosis on Mac OS X Server
Gitosis on Mac OS X ServerYasuhiro Asaka
 
Spatial script for my JS.Everywhere 2012
Spatial script for my JS.Everywhere 2012Spatial script for my JS.Everywhere 2012
Spatial script for my JS.Everywhere 2012Steven Pousty
 
Openshift GeoSpatial Capabilities
Openshift GeoSpatial CapabilitiesOpenshift GeoSpatial Capabilities
Openshift GeoSpatial CapabilitiesSteven Pousty
 
Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012
Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012
Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012Steven Pousty
 
Rustでパケットと戯れる
Rustでパケットと戯れるRustでパケットと戯れる
Rustでパケットと戯れるShuyaMotouchi1
 
A curious case of broken DNS responses (Coloclue Presents - Nov 2019)
A curious case of broken DNS responses (Coloclue Presents - Nov 2019)A curious case of broken DNS responses (Coloclue Presents - Nov 2019)
A curious case of broken DNS responses (Coloclue Presents - Nov 2019)Babak Farrokhi
 
Highload осень 2012 лекция 1
Highload осень 2012 лекция 1Highload осень 2012 лекция 1
Highload осень 2012 лекция 1Technopark
 
Infrastructure coders logstash
Infrastructure coders logstashInfrastructure coders logstash
Infrastructure coders logstashDavid Lutz
 
Searching the Now
Searching the NowSearching the Now
Searching the Nowlucasjosh
 
Spatial mongo for PHP and Zend
Spatial mongo for PHP and ZendSpatial mongo for PHP and Zend
Spatial mongo for PHP and ZendSteven Pousty
 
Mongo sf spatialmongo
Mongo sf spatialmongoMongo sf spatialmongo
Mongo sf spatialmongoSteven Pousty
 
Is ruby logger thread(process)-safe? at RubyConf 2013
Is ruby logger thread(process)-safe? at RubyConf 2013Is ruby logger thread(process)-safe? at RubyConf 2013
Is ruby logger thread(process)-safe? at RubyConf 2013Naotoshi Seo
 
A curious case of broken dns responses - RIPE75
A curious case of broken dns responses - RIPE75A curious case of broken dns responses - RIPE75
A curious case of broken dns responses - RIPE75Babak Farrokhi
 
nouka inventry manager
nouka inventry managernouka inventry manager
nouka inventry managerToshiaki Baba
 

What's hot (20)

skipfish
skipfishskipfish
skipfish
 
Gitosis on Mac OS X Server
Gitosis on Mac OS X ServerGitosis on Mac OS X Server
Gitosis on Mac OS X Server
 
Spatial script for my JS.Everywhere 2012
Spatial script for my JS.Everywhere 2012Spatial script for my JS.Everywhere 2012
Spatial script for my JS.Everywhere 2012
 
Go破壊
Go破壊Go破壊
Go破壊
 
Openshift GeoSpatial Capabilities
Openshift GeoSpatial CapabilitiesOpenshift GeoSpatial Capabilities
Openshift GeoSpatial Capabilities
 
Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012
Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012
Spatial Mongo and Node.JS on Openshift JS.Everywhere 2012
 
Rustでパケットと戯れる
Rustでパケットと戯れるRustでパケットと戯れる
Rustでパケットと戯れる
 
A curious case of broken DNS responses (Coloclue Presents - Nov 2019)
A curious case of broken DNS responses (Coloclue Presents - Nov 2019)A curious case of broken DNS responses (Coloclue Presents - Nov 2019)
A curious case of broken DNS responses (Coloclue Presents - Nov 2019)
 
Highload осень 2012 лекция 1
Highload осень 2012 лекция 1Highload осень 2012 лекция 1
Highload осень 2012 лекция 1
 
3 rd animation
3 rd animation3 rd animation
3 rd animation
 
Infrastructure coders logstash
Infrastructure coders logstashInfrastructure coders logstash
Infrastructure coders logstash
 
Searching the Now
Searching the NowSearching the Now
Searching the Now
 
Spatial mongo for PHP and Zend
Spatial mongo for PHP and ZendSpatial mongo for PHP and Zend
Spatial mongo for PHP and Zend
 
Lab report
Lab reportLab report
Lab report
 
Mongo sf spatialmongo
Mongo sf spatialmongoMongo sf spatialmongo
Mongo sf spatialmongo
 
Is ruby logger thread(process)-safe? at RubyConf 2013
Is ruby logger thread(process)-safe? at RubyConf 2013Is ruby logger thread(process)-safe? at RubyConf 2013
Is ruby logger thread(process)-safe? at RubyConf 2013
 
A curious case of broken dns responses - RIPE75
A curious case of broken dns responses - RIPE75A curious case of broken dns responses - RIPE75
A curious case of broken dns responses - RIPE75
 
faastCrystal
faastCrystalfaastCrystal
faastCrystal
 
Comets notes
Comets notesComets notes
Comets notes
 
nouka inventry manager
nouka inventry managernouka inventry manager
nouka inventry manager
 

Similar to Python Dependencies at Beeswax - BazelCon 2019 Lightning Talk - Ron Rothman

Pycon taiwan 2018_claudiu_popa
Pycon taiwan 2018_claudiu_popaPycon taiwan 2018_claudiu_popa
Pycon taiwan 2018_claudiu_popaClaudiu Popa
 
PyConUK 2018 - Journey from HTTP to gRPC
PyConUK 2018 - Journey from HTTP to gRPCPyConUK 2018 - Journey from HTTP to gRPC
PyConUK 2018 - Journey from HTTP to gRPCTatiana Al-Chueyr
 
Python 2 is dead! Drag your old code into the modern age
Python 2 is dead! Drag your old code into the modern agePython 2 is dead! Drag your old code into the modern age
Python 2 is dead! Drag your old code into the modern ageBecky Smith
 
Relational Database Access with Python ‘sans’ ORM
Relational Database Access with Python ‘sans’ ORM  Relational Database Access with Python ‘sans’ ORM
Relational Database Access with Python ‘sans’ ORM Mark Rees
 
Python仮想環境構築の基礎と ツールの比較
Python仮想環境構築の基礎と ツールの比較Python仮想環境構築の基礎と ツールの比較
Python仮想環境構築の基礎と ツールの比較cocodrips
 
Time series denver an introduction to prometheus
Time series denver   an introduction to prometheusTime series denver   an introduction to prometheus
Time series denver an introduction to prometheusBob Cotton
 
MongoDB Days UK: Using MongoDB and Python for Data Analysis Pipelines
MongoDB Days UK: Using MongoDB and Python for Data Analysis PipelinesMongoDB Days UK: Using MongoDB and Python for Data Analysis Pipelines
MongoDB Days UK: Using MongoDB and Python for Data Analysis PipelinesMongoDB
 
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...doughellmann
 
Pipenv - Melbourne Python User Group (MPUG)
Pipenv - Melbourne Python User Group (MPUG)Pipenv - Melbourne Python User Group (MPUG)
Pipenv - Melbourne Python User Group (MPUG)Rory Hart
 
2018 jPrime Deconstructing and Evolving REST Security
2018 jPrime Deconstructing and Evolving REST Security2018 jPrime Deconstructing and Evolving REST Security
2018 jPrime Deconstructing and Evolving REST SecurityDavid Blevins
 
Zn task - defcon russia 20
Zn task  - defcon russia 20Zn task  - defcon russia 20
Zn task - defcon russia 20DefconRussia
 
基于 Google protobuf 的 webgame 网络协议设计
基于 Google protobuf 的 webgame 网络协议设计基于 Google protobuf 的 webgame 网络协议设计
基于 Google protobuf 的 webgame 网络协议设计勇浩 赖
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scalePierre Souchay
 
Relational Database Access with Python
Relational Database Access with PythonRelational Database Access with Python
Relational Database Access with PythonMark Rees
 
Using PyPy instead of Python for speed
Using PyPy instead of Python for speedUsing PyPy instead of Python for speed
Using PyPy instead of Python for speedEnplore AB
 
[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래NAVER D2
 
Writing dumb tests
Writing dumb testsWriting dumb tests
Writing dumb testsLuke Lee
 

Similar to Python Dependencies at Beeswax - BazelCon 2019 Lightning Talk - Ron Rothman (20)

Pycon taiwan 2018_claudiu_popa
Pycon taiwan 2018_claudiu_popaPycon taiwan 2018_claudiu_popa
Pycon taiwan 2018_claudiu_popa
 
PyConUK 2018 - Journey from HTTP to gRPC
PyConUK 2018 - Journey from HTTP to gRPCPyConUK 2018 - Journey from HTTP to gRPC
PyConUK 2018 - Journey from HTTP to gRPC
 
Python 2 is dead! Drag your old code into the modern age
Python 2 is dead! Drag your old code into the modern agePython 2 is dead! Drag your old code into the modern age
Python 2 is dead! Drag your old code into the modern age
 
Relational Database Access with Python ‘sans’ ORM
Relational Database Access with Python ‘sans’ ORM  Relational Database Access with Python ‘sans’ ORM
Relational Database Access with Python ‘sans’ ORM
 
Introduction to robot framework
Introduction to robot frameworkIntroduction to robot framework
Introduction to robot framework
 
Python仮想環境構築の基礎と ツールの比較
Python仮想環境構築の基礎と ツールの比較Python仮想環境構築の基礎と ツールの比較
Python仮想環境構築の基礎と ツールの比較
 
Time series denver an introduction to prometheus
Time series denver   an introduction to prometheusTime series denver   an introduction to prometheus
Time series denver an introduction to prometheus
 
MongoDB Days UK: Using MongoDB and Python for Data Analysis Pipelines
MongoDB Days UK: Using MongoDB and Python for Data Analysis PipelinesMongoDB Days UK: Using MongoDB and Python for Data Analysis Pipelines
MongoDB Days UK: Using MongoDB and Python for Data Analysis Pipelines
 
Ladypy 01
Ladypy 01Ladypy 01
Ladypy 01
 
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
Rolling with the Times: Using wheels, pbr, and Twine for Distributing and Ins...
 
Pipenv - Melbourne Python User Group (MPUG)
Pipenv - Melbourne Python User Group (MPUG)Pipenv - Melbourne Python User Group (MPUG)
Pipenv - Melbourne Python User Group (MPUG)
 
2018 jPrime Deconstructing and Evolving REST Security
2018 jPrime Deconstructing and Evolving REST Security2018 jPrime Deconstructing and Evolving REST Security
2018 jPrime Deconstructing and Evolving REST Security
 
Zn task - defcon russia 20
Zn task  - defcon russia 20Zn task  - defcon russia 20
Zn task - defcon russia 20
 
基于 Google protobuf 的 webgame 网络协议设计
基于 Google protobuf 的 webgame 网络协议设计基于 Google protobuf 的 webgame 网络协议设计
基于 Google protobuf 的 webgame 网络协议设计
 
Consul administration at scale
Consul administration at scaleConsul administration at scale
Consul administration at scale
 
Relational Database Access with Python
Relational Database Access with PythonRelational Database Access with Python
Relational Database Access with Python
 
Python para equipos de ciberseguridad
Python para equipos de ciberseguridad Python para equipos de ciberseguridad
Python para equipos de ciberseguridad
 
Using PyPy instead of Python for speed
Using PyPy instead of Python for speedUsing PyPy instead of Python for speed
Using PyPy instead of Python for speed
 
[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래[1C2]webrtc 개발, 현재와 미래
[1C2]webrtc 개발, 현재와 미래
 
Writing dumb tests
Writing dumb testsWriting dumb tests
Writing dumb tests
 

Recently uploaded

Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)OPEN KNOWLEDGE GmbH
 
Buds n Tech IT Solutions: Top-Notch Web Services in Noida
Buds n Tech IT Solutions: Top-Notch Web Services in NoidaBuds n Tech IT Solutions: Top-Notch Web Services in Noida
Buds n Tech IT Solutions: Top-Notch Web Services in Noidabntitsolutionsrishis
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEOrtus Solutions, Corp
 
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfStefano Stabellini
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...Technogeeks
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)jennyeacort
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEEVICTOR MAESTRE RAMIREZ
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmSujith Sukumaran
 
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideBuilding Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideChristina Lin
 
Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)Hr365.us smith
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaHanief Utama
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024StefanoLambiase
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...OnePlan Solutions
 
Cloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStackCloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStackVICTOR MAESTRE RAMIREZ
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....kzayra69
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWave PLM
 

Recently uploaded (20)

Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)
 
Buds n Tech IT Solutions: Top-Notch Web Services in Noida
Buds n Tech IT Solutions: Top-Notch Web Services in NoidaBuds n Tech IT Solutions: Top-Notch Web Services in Noida
Buds n Tech IT Solutions: Top-Notch Web Services in Noida
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
 
Xen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdfXen Safety Embedded OSS Summit April 2024 v4.pdf
Xen Safety Embedded OSS Summit April 2024 v4.pdf
 
What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...What is Advanced Excel and what are some best practices for designing and cre...
What is Advanced Excel and what are some best practices for designing and cre...
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
 
Cloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEECloud Data Center Network Construction - IEEE
Cloud Data Center Network Construction - IEEE
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalmIntelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
 
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop SlideBuilding Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
Building Real-Time Data Pipelines: Stream & Batch Processing workshop Slide
 
Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)
 
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief UtamaReact Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
 
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
Dealing with Cultural Dispersion — Stefano Lambiase — ICSE-SEIS 2024
 
Unveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New FeaturesUnveiling the Future: Sylius 2.0 New Features
Unveiling the Future: Sylius 2.0 New Features
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
 
Cloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStackCloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStack
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....
 
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need ItWhat is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
 

Python Dependencies at Beeswax - BazelCon 2019 Lightning Talk - Ron Rothman

  • 1. Python Dependencies The Right* Way Bazel + Python @ Beeswax Ron Rothman * For us. YMMV.
  • 2. ● 30 engineers ● monorepo ● 100 applications ● 300 kLoPyC 1.Global requirements.txt 2.rules_python 3.pipenv We have: We tried: ● bazel test ● bazel run ● deployments We need deps for:
  • 4. # requirements.lock boto3==1.10.2 botocore==1.13.2 # via boto3, s3transfer certifi==2019.9.11 # via requests chardet==3.0.4 # via requests docutils==0.15.2 # via botocore futures==3.3.0 # via s3transfer idna==2.8 # via requests jmespath==0.9.4 # via boto3, botocore python-dateutil==2.8.0 # via botocore requests==2.22.0 s3transfer==0.2.1 # via boto3 six==1.12.0 # via python-dateutil urllib3==1.25.6 # via botocore, requests py_binary( name = "my_app", deps = ["lib_a"], requires = ["requests>=2.21"], ) py_library( name = "lib_a", deps = ["lib_b"], requires = ["requests>=2.18"], ) py_library( name = "lib_b", deps = ["lib_c"], requires = ["boto3~=1.9"], ) py_venv( name = "my_app_env", targets = [":my_app"], ) # requirements.in boto3~=1.9 requests>=2.18 requests>=2.21
  • 5. Needs improvement: ● Dev/CI workflow more complicated than before ● Bazel cache misses ● Dependency changes are not propagated automatically Working well: ● 100% correct (and minimal) dependencies! ● Monorepo-friendly ● History of dependency changes (lock files) ● Wheels are easy
  • 6. Thank You 🙏🏻 ron {at} beeswax.com

Editor's Notes

  1. Hi everyone...
  2. We're 30 / monorepo / 100 / 300k The problem... how to manage our Py 3p deps By third-party... [C]When we bāzel-test or bāzel-run some target... also deps of every other target in its xtive clo same in prod [C]When had one or two apps, we solved... reqs.txt ...this didn't scale well once critical mass by multiple engs rules_python - can't guarantee import correct version pipenv Which led us to roll...
  3. Our solution, which we call Beeve... set of tools pip-tools is... composed of starlark macros/rules/aspects + command line tools [C]Beeve's output is... A lockfile is a set of 3p deps, each pinned an application's reqs.txt, but autogenerated as needed We leveraged: Bāzel's... walk the graph, collect 3pd; pip-tools's... resolve deps [C]We take each app's lockfile and use it...
  4. Here's an example. toy BUILD Note we've annotated each py rule... new attr requires... dep spec a dep spec is just a constraint on py_venv - tells Beeve we want lockfile for the... When run Bāzel on py_venv rule, our Starlark code... [C] requirements.in... union [C]then calls pip-compile to generate... the lockfile contains pinned versions of... as well as their transitive deps... ie it's the complete set of Py pkgs that our py_binary... not hermetic; nor reproducible; current state PyPI if you run... different lockfiles guaranteed that any resulting lockfile conforms to constraints... !!!
  5. one downside is... devs must now be aware of py envs, switch between them Bāzel-test error prone Bāzel caching less effective 3p dep changes are not propagated automatically to downstream... [C]on plus side... most important goal... reliably know precise set of 3p deps... contains everything/nothing monorepo-friendly... go without saying nice side benefit... track dep changes over time, since... can easily produce wheels for our Py apps, has made deployments...
  6. I'm happy to talk more about Beeve or Python dependencies ...email me with any questions or just find me at the conference. Thank you!