We all know about the importance of automatic deployment of applications to production, but do we all know how to build an automated quality control system that continuously provides feedback on the application’s state and decides whether the application is ready for production rollout in terms of quality metrics? I have prepared a step-by-step algorithm for setting up such a system on your project, with specifics for web applications. I will talk about the best quality control practices at different levels, from linters to performance testing, from underwater problems to the ideal monitoring system.
2. ALEXPSHE ALEXPSHE ALE
● SDET at JetBrains
● 20+ fully automated
quality control systems
● Big fan of process
automation 💚
3. Get an understanding
of the high-level algorithm
for building a quality system
Main takeaway
*There will be no technologies and specific solutions for individual cases here.
4. THE PROBLEM IS ..
Typical view on CI/CD and testing
Automation
tests
Not part of
CI/CD OR Automation
tests
Part of
CI/CD
Green/Red
NOT BAD, BUT…
5. CI/CD -> Automated Quality Control System
New feature developed
Quality Metrics measured
Responsible people notified
Next process started
Quality Gates called
Pipeline artifacts collected
Time to respond increased 34%
Compressed stored image
Allowed +10% Automessage to
#performance channel
Save node settings
and benchmark log
Request developer`s update
6. DELIVERY
PROCESS
For feature:
● Requirements testing
● Functional testing
● Non-functional testing
For product version:
● UAT testing
● Alpha/Beta testing
● Functional testing
● Non-functional testing
For release:
● Configuration testing
● Compatibility checks
● Availability checks
WHAT PROCESSES DO YOU NEED?
8. FOLLOW STEPS: FOLLOW STEPS:
● Write down ALL processes in your team
● Choose the most crucial, expensive and long
● Implement automatic testing control system
9. A TESTER`S GUIDE TO CI/CD
Preparation steps:
● Business Metrics
● Quality Metrics
● Quality Gates
Pipeline steps:
● Pipeline triggers
● Pipeline stages
● Pipeline criteria
Feedback steps:
● Metric`s collecting
● System notifications
● Call to next process
FOR EACH DELIVERY PROCESS ->
Follow steps
15. PREPARATION
STEPS
1.BUSINESS
METRICS
WHY
Do we need
test automation?
Business goal Business metrics Product metrics
More money More paid users +25% of retention
QA Auto as a tool to achieve product metrics
Then achieve
Then achieve
17. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS
Product metrics
Business Metric: Revenue Growth
Goal:
20% Overall Revenue Growth
for the Company
Over the Quarter
Product Metric: User Retention
Goal: User retention after first use
increases from 25% to 40% following
the implementation of:
feature that recommends content based
on the longest videos watched from the
last three
Business metrics
18. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS Product metrics
Quality metrics
Quality Metric: Test Coverage
for New Recommendation Feature
Goal: Achieve 90%
test coverage for the new video
recommendation feature.
Product Metric: User Retention
Goal: User retention after first use
increases from 25% to 40% following
the implementation of:
feature that recommends content based
on the longest videos watched from the
last three
19. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS
Quality metrics
FROM NON FUNCTIONAL REQUIREMENTS
Quality Metric: Performance
Goal: Ensure TikTok can
maintain smooth performance
during peak times, handling
up to 1 million concurrent
users without degradation in
service quality.
21. PREPARATION
STEPS
LIST
OF
METRICS
LIST OF METRICS Quality metrics
Quality gates
https://pshe-academy.com/quality-metrics/en
Quality Metric: Test Coverage
for New Recommendation Feature
Goal: Achieve 90%
test coverage for the new video
recommendation feature.
Quality Gates:
>= 90% - allowed
< 90% - not allowed
22. PREPARATION
STEPS
CONCLUSION
STEPS TO DO:
1. Ask for business goals for the next quater/year
2. Clarify product metrics with product owner
3. Build qulity metrics based on product metrics
4. Setup quality gates based on quality metrics
24. PIPELINE
STEPS
1.
PIPELINE
TRIGGERS
Git events:
● New Merge Request
● New Commit
● New changes in folder
Manual actions:
● Manual start
● Person`s approve
● Message in Slack
Previous step ended:
● Triggered
by another pipeline
WHAT THE FLAG OF PROCESS START?
28. PIPELINE
STEPS
3.
PIPELINE
CRITERIA
“FAIL FIRST” approach:
Pipeline has to fail
when it's obvious that
the process is not met
requirements
Time of execution:
Pipeline has to be finished
in appropriate time
(time that does not affect
other processes)
Reliability:
Pipeline has to provide
only true-positive feedback
(low flakiness rate)
WHAT CRITERIA OF THE EFFECTIVE PIPELINE?
29. FEATURE ASSESSMENT
New feature
Triggered by commit:
1 commit = 1 MR
Build
Unit
tests
E2E
tests
Performance
Tests
Optimize
build chain
Set Up
parallel runs
Increase
stability
Prepare
Prod-like env
30. PIPELINE
STEPS
CONCLUSION
STEPS TO DO:
1. Find correct trigger/s for the process
2. Design pipeline stages based on metrics/checks
3. Prioritize stages following “fail first” approach
4. Measure time/flakiness rate
to confirm pipeline quality criteria
42. FEEDBACK
STEPS
CONCLUSION
STEPS TO DO:
1. Choose the best way to collect all metrics
2. Figure out responsibilities and call of actions
3. Setup a call to the next process based on
processes` map
44. AUTOMATED
QUALITY
CONTROL
SYSTEM
1. Prepared stage priority
2. Automated metrics calculation
3. Implemented instruction
4. Collecting data and call next step
CI/CD as tester`s tool
Continuous feedback about
system quality status