This talk will introduce the SUSE Product Security team, who handles the software security processes for openSUSE and also SUSE Linux Enterprise. The SUSE Product Security work is split into "reactive" and "proactive" areas and engineering groups these days. Reactive work refering to what is traditionally known as "security incident response", while proactive refers to security audits, design reviews and related areas of secure software development. The talk will focus on the reactive side, giving statistics, and talk about some highlights from the last year. Also bringing a small overview over how closing the leap gap changes affects the openSUSE Maintenance process.

1. Security Overview 2022
Marcus Meissner
Projectmanager Security / Distinguished Engineer
meissner@suse.de

2. 2
Agenda
●
Introduction to the SUSE Product Security Team
●
Retrospecting year 2022/2021
●
Closing the (openSUSE) Leap Gap
●
Supply Chain Security
●
Outlooks

3. 3
Introducing the team
Teamlead: Stoyan Manolov
Reactive:
• Projectmanager/DE: Marcus Meissner
• Security Engineers: Alexander Bergmann, Robert Frohl, Gianluca Gabrielli,
Carlos Lopez, Gabriele Sonnu, Thomas Leroy, Mæve Rey, Cathy Hu
• Trainee: Michael Muschner
Proactive:
• Lead: Johannes Segitz
• Security Engineers: Matthias Gerstner, Wolfgang Frisch, Paolo Perego, Filipo
Bonazzi

4. 4
Proactive vs Reactive
Reactive:
• Incident response and handling (PSIRT)
• Tracking CVEs, coordinating fixes, monitoring
• also working on proactive projects
Proactive:
• Audit programs and system services (systemd, dbus, network)
• Approving product releases
• All security work before shipment
• SELinux

5. 5
Reactive Security

6. 6
Reactive Security
• Reacting on reported security issues
• Roles: Incident and Update Managers
• Coordinating from begin to end, delegate
– Fixing to package maintainer
– Source review to OBS/IBS reviewers
– Testing to QA
• Release
• Documentation
– Human and machine readable

7. 7
Incident Management
Open bug:
• Summary line: VUL-x: CVE: package: summary
• In SUSE Security Incidents product
• Description with references / links
• Patches and reproducers attached
• Assign to packager / bugowner
SMASH Issue:
• Rating and CVSSv3 scoring
• Affected Software (for SLE)

8. 8
Opened bugs per week: 2006 - 2022

9. 11
Update Management
Packager submits updates
Security Team
• Writes metadata (summary, description, issues)
• Checks if building
• Pushes to QA
QA tests update
Security Team
• Releases update
• Additional documentation work

10. 14
Reactive work – what is delivered
The updates themselves
Notifications for every update
• E-Mail and web https://www.suse.com/support/update/
• E-Mail to security-announce@lists.opensuse.org
Autogenerated information:
• CVE webpages https://www.suse.com/security/cve/
• OVAL data https://www.suse.com/support/security/oval/
• Advisory CVRF data http://ftp.suse.com/pub/projects/security/cvrf/ (CVRF 1.1,
1.2)
• CVE CVRF 1.2 data: https://ftp.suse.com/pub/projects/security/cvrf-cve/

11. 16
High Impact Vulnerabilities
Usual incidents are handled in a pipeline, but some stand out:
HEARTBLEED, SHELLSHOCK … , MELTDOWN, SYSTEM DOWN, MDS
Issues that need:
• Quick reaction
• More communication
• More documentation
• Tricky fixes
Special category
Special process and special team to handle it

12. 24
Proactive Security

13. 25
Proactive Security
Making products secure before shipment
Secure Development Lifecycle
Automated checks during build
• setuid binaries
• DBUS services
• PolicyKit rules
• PAM modules
And:
• systemd and network services
• Whatever comes up

14. 26
Proactive Security
Security audits are “rolling”
• Based on Factory development
• Based on packages
• But also on final product
Most of our work for SUSE Linux Enterprise is done in openSUSE!

15. 27
Retrospecting 2021 and 2022

16. 28
Reactive Security Stats
Year 2019 2020 2021 2022
Total CVE 2645 2461 2559 1197 (30.5.)
CVE released 2139 2176 1911 1131 (30.5)
Updates
released
2032 2129 2364 965 (30.5.)
OpenSUSE
updates
862 729 880 211 (30.5.)

17. 29
High Impact Vulnerabilities 2021-2022
2021:
●
LOG4SHELL
●
Boothole2 – 2nd
iteration of grub2 secure boot issues
●
DHEATER – exploit Diffie Hellman key exchange (e.g. in TLS)
●
DNSPooq, FREAKOUT, Baron Samedit, FRAGATTACK, Sequoia,
OMIGOD
●
Various CPU side channel issues – similar to last years
2022:
●
Dirty Pipe, Pwnkit, Samba vfs_fruit, More CPU side channel issues
●
… and the year is not over yet ...

18. 30
Proactive highlights
●
RPMLINT 2 migration
●
More whitelisting methods (logrotate, rpm scripts, hashes of
monitored config files)
●
AUDITs happened on a lot of packages
●
Factory uses “-z now” linking (Full RELRO)
●
Factory supports crypto-policies (and SLES 15 SP4 too)
●
We are picking up more SELinux work

19. 31
Closing the Leap Gap

20. 32
Closing the Leap Gap
History:
OpenSUSE Leap 42.x, 15.0 - 15.2
●
Imported and rebuilt SLES sources after their release
●
Pushed through separate maintenance pipeline
●
SLE PackageHub imported and rebuilt Leap sources
Problems:
●
Always delays, due to build times and double QA
●
especially visible with Mozilla Firefox
●
Always hand-holding updates even though they passed SLES
●
Fragile origin-manager

21. 33
Closing the Leap Gap
Now:
OpenSUSE Leap 15.3 and newer:
●
Instead of rebuilding sources ship SLE binaries directly
●
PackageHub shared between SLE and openSUSE
●
No source rebuilds anymore at all
●
Rough start, but now working with way less effort

22. 34
Supply Chain Security

23. 35
Supply Chain Security
Code
review
Package
selection
Building Testing Signing
Delivery
Network
Community
Code

24. 36
Supply Chain Security
Trusting incoming Sources
We take sources from thousands of OSS developers, from a very
diverse ecosystem.
Ensuring Maintainer provided the sources:
Today:
●
Source URLs (largely https, but still http and ftp, old git protocol, or
none)
●
GPG signatures (rare)
●
GIT services
●
Storing all tarballs in OBS for history

25. 37
Supply Chain Security
Trusting incoming Sources
Tomorrow:
●
Bring up usage of https and GPG signatures!
●
More Git storage with implicit hash integrity and/or GPG signing?
●
Better signing methods? Sigstore.dev?
●
Help by centralized hosting like e.g. github?
CALL TO ACTION: Easy clean up work:
●
Adjust Source URLs to be https (spec-cleaner work by Johannes)
●
If we don’t reference GPG signature, check if its there and add it

26. 38
Supply Chain Security – Building with SLSA
What is SLSA? (https://slsa.dev/)
Google supplied standardization framework on build processes and
build services assurance to mitigate supply chain attacks.
●
Sources
●
Building
●
Provenance
●
Common

27. 39
SLSA
How does it match what SUSE and openSUSE do?
●
The open build service shines, it helps us meet most criteria
already.
●
Some process adjustments
●
Two person reviews
●
Strong user authentication: MFA finally!
●
Something new: Provenance
●
Proof it to the outside, via automation
●
In-toto attestation data
●
TODO: reproducible builds

28. 40
Supply chain security – cryptographic signatures
All deliverables must be signed and verifiable!
Today we sign:
●
RPMs, repos, ISOs with GPG
●
UEFI secure boot binaries (pkcs12/x509)
●
Containers with Notary v1 , Atomic and Cosign
Challenges:
●
Root of trust … how do you know it is the openSUSE key?
●
Web of trust for 3rd
party repos

29. 41
Supply chain security – cryptographic signatures
What is missing?
●
Container signature verification nearly non existent
●
Always optional and needs to be configured
●
Kubernetes however is doing Cosign from sigstore now
●
GPG checks often bypassed in automation
●
Seeing a LOT of zypper –no-gpg-checks or –gpg-auto-import-keys
●
Means no signature checking is done!

30. 42
Supply chain security – sigstore
What is sigstore? ( https://www.sigstore.dev/ )
Sigstore is an initiative to make signing of artefacts easier.
●
Yet another approach to container signing (cosign)
●
Keyless signing with short lived keys during automated builds
(cosign and fulcio)
●
Transparency Logfile to store signatures for later verification
(rekor)

31. 43
Supply chain security – sigstore
What is SUSE doing with sigstore?
●
Sign containers on registry.suse.com with cosign
●
Store SLE RPM-MD and container signatures in rekor log
Todo:
●
Implement also on OBS / registry.opensuse.org
●
Verification in system against rekor log

32. 44
Outlook and Call To Action

33. 45
Outlook – Call To Action
For Security:
●
Keeping up our existing work and improvements
●
Working on ALP (Adaptable Linux Platform)
●
More Supply Chain Hardening
Where you can help (easy to moderate tasks):
●
If you spot security issues without a CVE, tell us!
●
Add / Adjust Source URLs / GPG signatures of packages
●
Increase crypto-policies Adoption

34. 50
Landing page: https://www.suse.com/security/
E-Mail: security@suse.de
We are looking for:
• Security Engineers
• Students (for bachelor/master work)
Questions?

35. 51

36. 52

38. 54
Internal Use Slides

39. Presentation Title(36pt)
Subhead or Second Line (20pt)
Presenter Name (14pt)
Title
Company/Email
For Internal Use Only