Leveraging Gradle @ Netflix (Madrid GUG Feb 2, 2021)
Gradle is an open-source build automation tool focused on flexibility, build reproducibility and performance. Over the years, this tool has evolved and introduced new concepts and features around dependency management, publication and other aspects on build and release of artifacts for the Java platform. Keeping up to date with all these features across several projects can be challenging. How do you make sure that all your projects can be upgraded to the latest version of Gradle? What if you have thousands of projects and hundreds of engineers? How can you abstract common tasks for them and make sure that new releases work as expected? At Netflix, we built Nebula, a collection of Gradle plugins that helps engineers remove boilerplate in Gradle build files, and makes building software the Netflix way easy. This reduces the cognitive load on developers, allowing them to focus on writing code. In this talk, I’ll share with you our philosophy on how to build JVM artifacts and the pieces that help us boost the productivity of engineers at Netflix. I’ll talk about: - What is Nebula - What are the common problems we face and try to solve - How we distribute it to every JVM engineer - How we ensure that Nebula/Gradle changes do not break builds so we can ship new features with confidence at Netflix. --- About Roberto: Roberto Perez Alcolea is a Senior Software Engineer at Netflix. He is a member of the Java Platform team providing the core language and framework components that enable the Java community at Netflix. He's an active maintainer of Netflix Nebula Plugins (https://nebula-plugins.github.io/) and passionate about Gradle. Prior to that, he spent several years building high performant APIs with Ratpack and web applications using Grails.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Leveraging Gradle @ Netflix (Madrid GUG Feb 2, 2021)
Similar to Leveraging Gradle @ Netflix (Madrid GUG Feb 2, 2021) (20)
More from Roberto Pérez Alcolea
More from Roberto Pérez Alcolea (7)
Recently uploaded
Recently uploaded (20)
Leveraging Gradle @ Netflix (Madrid GUG Feb 2, 2021)
- 2. Who am I? Leveraging Gradle @ Net fl ix Roberto Pérez Alcole a Senior Software Enginee r Java Platform @ Netfli x rperezalcolea@netflix.com @rpalcolea
- 3. JVM Buil d @ Netflix Leveraging Gradl e @ Net fl ix
- 4. •Many repositories (~3.2k ) •Binary integration (JARs ) •Microservices with fat client s •Hundreds of engineer s •Ivy + Ant in the past. Now we use Gradle ! •Java, Groovy, Scala, Kotlin, Clojure Leveraging Gradle @ Net fl ix
- 5. Builds per wee k Leveraging Gradl e @ Net fl ix
- 6. Artifact Publishin g Leveraging Gradl e @ Net fl ix We publish ~9k artifacts per day !
- 7. Consumers rely on latest releas e Leveraging Gradl e @ Net fl ix
- 8. JVM Buil d Problems & Challenge s @ Netflix Leveraging Gradl e @ Net fl ix
- 10. Broken Build s •Add/upgrade dependencie s •Unaligned dependencie s •Dependencies with bugs or security vulnerabilitie s •Gradle plugin is not compatibl e Leveraging Gradl e @ Net fl ix
- 11. Slow Build s •Dependency resolutio n •Plugins and custom tasks impacting con fi guration tim e •Using excludes on dependencies declaration s •Lack of parallelization Leveraging Gradl e @ Net fl ix
- 12. Metadat a •Bad Ivy or Maven metadata can lead to a poisoned dependency graph . •Dynamic dependencie s •Bad scope s •Dependency declaration for shaded librarie s Leveraging Gradl e @ Net fl ix
- 13. Reproducible Build s •Same binary dependencie s •File orde r •Avoid preserving timestamp s Leveraging Gradl e @ Net fl ix
- 14. Packagin g •Make sure that a java application can be deployed with easiness in our infrastructure •VMs on EC 2 •Docker containers in Titu s Leveraging Gradl e @ Net fl ix
- 15. Plugin Ecosyste m •Third party plugin compatibility with new versions of Gradl e •Strong opinions from external and internal plugins (forcing dependencies ) •Upgrade well known plugins for existing build s Leveraging Gradl e @ Net fl ix
- 16. Fast evolution of Gradl e •Deprecated APIs and the need to migrate folks to use the new one s •It isn’t only about the DSL but also for plugin authors at Net fl i x •Introduce new features and keep up with Gradle release s Leveraging Gradl e @ Net fl ix
- 17. Leveraging Gradl e @ Net fl ix Publishers lack visibility • Who will I break ? • Who is consuming my library ? • Who is using this API that I want to change?
- 18. What ca n We do? Leveraging Gradl e @ Net fl ix
- 20. Nebula t o The rescue Leveraging Gradl e @ Net fl ix
- 21. Leveraging Gradl e @ Net fl ix What is Nebula ? •Gradle plugins and internal custom distribution built to eliminate boilerplate build logic and provide sane convention s •Dependency managemen t •Publishin g •OS Packagin g •Lintin g •Dependency lockin g •And many others !
- 22. This plugin provides general purpose rule types on top of Gradle resolution strategies and module metadata, allowing rules to be published, versioned, shared between projects, and optionally dependency locked. •Duplicate classes caused by changes to group or artifact ids, without renaming package s •Duplicate classes caused by bundle dependencies, which do not con fl ict resolve against the 'regular' dependencies for that library •Lack of version alignment between libraries, where version alignment is needed for compatibility •Ensuring a minimum version of a library Resolution Rules Plugin Leveraging Gradl e @ Net fl ix
- 28. How do we distribute Nebula? Leveraging Gradl e @ Net fl ix
- 29. Gradle custo m Distributions Leveraging Gradle @ Net fl ix • Apply internal and OSS plugin s • Custom init script s • Configure repositorie s • Configure status schemes (snapshot, candidate, release ) • Enable metrics collectio n • Enable/disable feature s • Default gradle.properties
- 30. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/custom-distribution-example Custom Gradle distribution
- 31. How do we validate new versions of Nebula? Leveraging Gradl e @ Net fl ix
- 32. •Leverage Niagara to clone every Nebula based project at Gradle on its own container and build it using the new version of Nebula . •Gradle nightly release s •Gradle candidate releases •Leverage Niagara to clone our OSS plugins and test them against nightly releases of Gradle. Acceptance Testin g Leveraging Gradl e @ Net fl ix
- 33. Leveraging Gradl e @ Net fl ix Nebula acceptance tests
- 34. How do we upgrade projects? Leveraging Gradl e @ Net fl ix
- 35. •Automatic Pull requests : •Dependency lock s •Nebula wrappe r •Lint rule s Managed Sourc e Evolving Continuous Integration at Netflix (Elise McCallum) | 2019 Leveraging Gradl e @ Net fl ix
- 37. •Build Cach e •Shared Read-only Dependency Cach e •Build Scan s •Gradle Enterpris e •Parallel execution Gradle Feature s Leveraging Gradl e @ Net fl ix
- 38. Leveraging Gradl e @ Net fl ix https://github.com/rpalcolea/leveraging-gradle-at-netflix-madrid-gug-2021/tree/main/shared-dependency-cache- example Shared Dependency Cache
- 39. Leveraging Gradl e @ Net fl ix Gradle Enterprise (values)
- 40. Leveraging Gradl e @ Net fl ix Gradle Enterprise (Tags)
- 42. •Collection of services and UIs that enable artifact observability and the ability to effect change in the Net fl ix ecosystem . •High level goal is to decrease time and effort needed to propagate change through the Net fl ix ecosyste m •Useful for Library deprecation and Security Vulnerabilitie s Astri d Leveraging Gradl e @ Net fl ix
- 43. Astrid - Artifact Managemen t Leveraging Gradl e @ Net fl ix
- 44. Astrid - Artifact Managemen t Leveraging Gradl e @ Net fl ix
- 45. Astrid - Campaign s Leveraging Gradl e @ Net fl ix
- 46. What’s next for us? Leveraging Gradl e @ Net fl ix
- 47. •Distributed Testin g •Java Toolchain s •Gradle Metadata Adoptio n •Con fi guration Cache Gradle Features in Our Horizo n Leveraging Gradl e @ Net fl ix