4. # requirements.lock
boto3==1.10.2
botocore==1.13.2 # via boto3, s3transfer
certifi==2019.9.11 # via requests
chardet==3.0.4 # via requests
docutils==0.15.2 # via botocore
futures==3.3.0 # via s3transfer
idna==2.8 # via requests
jmespath==0.9.4 # via boto3, botocore
python-dateutil==2.8.0 # via botocore
requests==2.22.0
s3transfer==0.2.1 # via boto3
six==1.12.0 # via python-dateutil
urllib3==1.25.6 # via botocore,
requests
py_binary(
name = "my_app",
deps = ["lib_a"],
requires = ["requests>=2.21"],
)
py_library(
name = "lib_a",
deps = ["lib_b"],
requires = ["requests>=2.18"],
)
py_library(
name = "lib_b",
deps = ["lib_c"],
requires = ["boto3~=1.9"],
)
py_venv(
name = "my_app_env",
targets = [":my_app"],
)
# requirements.in
boto3~=1.9
requests>=2.18
requests>=2.21
5. Needs improvement:
● Dev/CI workflow more complicated than before
● Bazel cache misses
● Dependency changes are not propagated automatically
Working well:
● 100% correct (and minimal) dependencies!
● Monorepo-friendly
● History of dependency changes (lock files)
● Wheels are easy
We're 30 / monorepo / 100 / 300k
The problem... how to manage our Py 3p deps
By third-party...
[C]When we bāzel-test or bāzel-run some target...
also deps of every other target in its xtive clo
same in prod
[C]When had one or two apps, we solved... reqs.txt
...this didn't scale well once critical mass by multiple engs
rules_python - can't guarantee import correct version
pipenv
Which led us to roll...
Our solution, which we call Beeve... set of tools
pip-tools is...
composed of starlark macros/rules/aspects + command line tools
[C]Beeve's output is...
A lockfile is a set of 3p deps, each pinned
an application's reqs.txt, but autogenerated as needed
We leveraged: Bāzel's... walk the graph, collect 3pd; pip-tools's... resolve deps
[C]We take each app's lockfile and use it...
Here's an example. toy BUILD
Note we've annotated each py rule... new attr requires... dep spec
a dep spec is just a constraint on
py_venv - tells Beeve we want lockfile for the...
When run Bāzel on py_venv rule, our Starlark code... [C] requirements.in... union
[C]then calls pip-compile to generate...
the lockfile contains pinned versions of...
as well as their transitive deps...
ie it's the complete set of Py pkgs that our py_binary...
not hermetic; nor reproducible; current state PyPI
if you run... different lockfiles
guaranteed that any resulting lockfile conforms to constraints... !!!
one downside is... devs must now be aware of py envs, switch between them Bāzel-test
error prone
Bāzel caching less effective
3p dep changes are not propagated automatically to downstream...
[C]on plus side... most important goal... reliably know precise set of 3p deps...
contains everything/nothing
monorepo-friendly... go without saying
nice side benefit... track dep changes over time, since...
can easily produce wheels for our Py apps, has made deployments...
I'm happy to talk more about Beeve or Python dependencies
...email me with any questions or just find me at the conference.
Thank you!