The million-dollar question for every QA lead, project manager, or even a tech lead of any #OEM: “How to reduce TCO and increase RoI? What are the best approaches to make my #testautomation more effective and efficient?” We covered the above questions in our focused #webinar on test automation titled “Managing Your RoI & TCO in Automation Testing” on 29th July 2021, 4:00 PM – 5:00 PM Indian Time. Utthunga’s very own #testautomationexperts Mr. Ravi Kumar N G and Mr. Nandan HA were the presenters and guided attendees with various ways to calculate and improve your RoI and #TCO. In the 60 minutes of the webinar, the speakers discussed on below points: 1. How to enhance RoI with a good test #automation strategy 2. Learn how to calculate and improve TCO 3. Best practices to improve #ROI and TCO You can view the webinar recording by clicking on the link https://youtu.be/qAHAFPmc3HM Please feel free to share these links with your colleagues who may be interested. If you have any queries or require more information regarding the topic or wish to know more about Utthunga you can mail us at contact@utthunga.com or visit our website https://utthunga.com/ © 2021 Utthunga

1. Managing Your ROI & TCO In Automation Testing

2. 2
© Utthunga 2021
OFFERINGS
Production Applications
Engineering
Operations Applications
Engineering
Security Engineering Verification &
Validation
Embedded Engineering
FOCUS
Power & Utilities
Motion & Mobility
Building Automation
& Control
Digital Engineering
Connectivity and
Data Solutions
Industrial Automation
& Control

3. 3
© Utthunga 2021
600+
Professionals
14+
Years
Headquarters: Bangalore, India
Offices In: Japan, Germany and USA
4
Locations
100+
Products
50+
Customers
Increase velocity of releases
up to 40%
Reduced TCO
Reduced vendor management
Co-creation partnership
Certifications

4. 4
© Utthunga 2021
ASSOCIATIONS
Part of various
Special Interest Groups
1
Technology Outsourcing
Partner
2
Involved in Reference
Application Architecture,
Design and Development
3
Part of Global Expert
Speakers Community
4 FDT Test & Certification Center
5 Compliance and Certification
6

5. 5
© Utthunga 2021
Speakers for Today
Nandan HA
 Nandan is an expert software test architect with over 12 years of experience in the testing
landscape
 Internally at Utthunga, Nandan is referred to as the sign-off gatekeeper for delivering zero
business impact and bug free production software
 He is proficient in performance testing, test automation framework development for web
applications, mobile and backend automation
Ravikumar NG
 Ravi specializes in the software test automation and manual testing domain
 He has over 15+ years of experience in different automation tools like Selenium, WinAppDriver,
Ranorex, TestComplete, UFT, AutoIT and other open source tools.
 His expert recommendations allows quality test execution of multiple process automation projects
like DCS Testing, DTM application automation, MS update patching in industrial automation domain
for desktop, mobile, web and cloud applications.

6. 6
© Utthunga 2021
Agenda
1
2
3
4
5
6
What is ROI and TCO of Automation Testing?
Challenges in Testing
Few Use cases
Best Approaches
How to calculate ROI
Best Approaches for Improving ROI and Reducing TCO

7. 7
© Utthunga 2021
Return On Investment & Total Cost of Ownership
What is ROI ?
 Savings related to cost
 Savings related to time
 Savings related to number of bugs detected
 Increase test coverage
Why do we need to calculate ROI?
 Improve efficiency
 Improve cost effectiveness
 Reduce repetition
 Improve product quality
 Improve release frequency

8. 8
© Utthunga 2021
Return On Investment & Total Cost of Ownership
When do we need to calculate ROI?
 After each release cycle
 After sprint cycle
 After product release
 After completing testing activity
How to calculate Testing ROI?
 % Change in execution duration
 Cost Analysis
 % Change in product quality
 Resources
CapEx
 Investment
 Procurement
OpEx
 Daily Operation
 Maintenance

9. 9
© Utthunga 2021
Testing Challenges
Dimensions
 Conformance
 Safety and Regulations
 Security and Privacy
 Network
 Interoperability
 Performance and scalability
 Usability
Challenges
 Lack of communication
 Overall system complexity
 Higher number of configurations to test due to hardware and software
architecture
 Dependency on 3rd party hardware and software
 Bugs in 3rd party software affecting testing activities
 Fast evolving architectures and protocols
 Incomplete documentation
 Co-existence of different applications (like antivirus, OS dependencies etc.)

10. 10
© Utthunga 2021
Problem Statement : Testing of HMI and DCS System
Problem Statement:
 DCS Systems operate in huge and complex settings
 Connects to multiple devices at a time
 Need cross platform testing to cover both Desktop and Cloud
scenarios
 Requires Functionality, Performance and Load testing
 Requires higher number of human resources
 Involves higher investment for testing infrastructure
Ineffective and inefficient testing leading to:
 Increased time consumption for validation of different parameters on
multiple devices
 Reduced test execution speed and unsatisfactory test coverage
 Test reporting and analysis consumes lot of time
 Painful infrastructure maintenance over extended period of time

11. 11
© Utthunga 2021
Stakeholders across the business experienced improved communication
and significant reduction in various aspects of their testing processes:
 Cost wise ROI within 6 build releases
 Increase in release velocity by > 120%
 Infrastructure reusability increased by > 100%
 Manpower dependency reduced by > 50%
 Maintenance cost reduced by > 80%
 Reusability of scripts > 35%
 Test coverage improvement by > 50%
 Improved bug detection by > 70%
Solution – Testing of HMI and DCS System
 Detailed feasibility study to analyze ROI and TCO, propose suitable
automation tools, and develop end-to-end automation framework
 Framework for test data management, ease of maintenance,
exception-handling and reusability
 Reliable, scalable and application-independent framework
 Identification and handling Flaky test and edge transitions
 Detailed test report for better readability and tracking
 Use of test simulators / device simulators to reduce costs
Proposed Solution
Benefits

12. 12
© Utthunga 2021
ROI Calculations
0.00%
50.00%
100.00%
150.00%
200.00%
250.00%
Build1 Build2 Build3 Build4 Build5 Build6 Build7 Build8 Build9 Build10 Build11 Build12
0
100000
200000
300000
400000
500000
600000
700000
Build1 Build2 Build3 Build4 Build5 Build6 Build7 Build8 Build9 Build10 Build11 Build12
Savings ($)

13. 13
© Utthunga 2021
Problem Statement: End-To-End AV and MS Security Updates
Qualification Process
Problem Statement:
A leading provider of Industrial Automation Solutions with a wide product
portfolio, recommends using 3 Anti-Virus software on Microsoft platform to its
end customers. The frequent Anti-Virus software, windows security and multiple
3rd party software updates to the system was hindering their overall efficiency
and productivity. Before recommending to the end customers, these updates
needed to be validated on daily basis or very frequently against the client
product’s functionality and performance.
 Manual test execution taking minimum 18 hours of effort per release
 Multiple new releases of AUT are added every year
 Support new operating systems and their versions as and when Microsoft
releases a new one
 Time consuming activities related to:
 Identifying network and other hardware related issues (before execution)
 Consolidation of test results and test report generation
 Average number of 504 hours of manual effort required per week
 Approximate cost per week is around $20,000
 Daily updates of antivirus software by vendors requires regular test
execution for correct validation of the software on a daily basis

14. 14
© Utthunga 2021
Test Solution: Approach I
Approach 1
 Automation of download process and pushing the updates to all the machines
 COTS automation tool used for easier scripting and maintenance
 Isolated execution of automation scripts for each product and release
 Framework for test data management, ease of maintenance, exception-handling
and reusability
 Reliable, scalable and application-independent framework
 Independent test scripts maintained for each release
 Separate logs and test reports generated for each release
Advantages
 Cost wise ROI within 63 build releases
 Increase in release velocity by > 11 %
 Cost per week reduced by approximately ~ $2050
 Manpower dependency reduced by > 30%
 Maintenance cost reduced by > 80%
 Reusability of Scripts > 25%
 Improved bug detection by > 60%
 Overall improvements of communication across all stakeholders
Product 2 - Release 2
Test Runner
Product 2 - Release 1
Test Runner
Product 1 - Release 2
Test Runner
Product 1 - Release 1
Test Runner

15. 15
© Utthunga 2021
Test Solution: Approach II
Approach 2
Develop and execute one common script that handles the End-To-End automation of
entire process
 Script handles all the test scenarios and test cases for all the releases/products
 Uses Open Source tools
 Support parallel script execution
 A centralized location for test execution on all releases
 Logs and Reports generated in one common location for all releases
 CI / CD implemented
 Use simulators and VM’s for improved test coverage
Advantages
 Cost wise ROI in < 50 build releases
 Increase in release velocity by > 15 %
 Cost per week reduced by approximately > $3000
 Manpower dependency reduced by > 90%
 Maintenance cost reduced by > 80%
 Reusability of scripts > 40%
 Improved bug detection by > 60%
 Overall improvements of Communication across all stakeholders
Product 1 - Release 2
Product 2 - Release 1
Product 2 - Release 2
Test Runner
Product 1 - Release 1
CICD

16. 16
© Utthunga 2021
ROI Calculations – Approach I & II
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
140.00%
160.00%
ROI (%) – Approach 1
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
140.00%
160.00%
180.00% ROI (%) – Approach 2
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
Savings – Approach 1
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
Savings

17. 17
© Utthunga 2021
Problem Statement : Cloud Testing
Problem Statement
A leading OEM faced issues with quality deliverable. They migrated
from an no QA setup to fully operational QA in 8 months. This
resulted in challenges for testing various scenarios and input
combinations and comparing the results to the expected behavior
and recording their observations.
 Manual test execution resulted in an increased testing efforts
with minimum 18 hours per release
 Encountered multiple browser and platform combinations
 Challenges in testing scalability
 On-premise testing infrastructure requires high testing effort
and ultimately rising overhead cost.
 Requires continuous maintenance & up-gradation to meet the
constant business needs

18. 18
© Utthunga 2021
Test Solution: Approach
Approach
 Implementing an effective test automation setup that can scale easily
 Adopting cloud automation platform for scalable testing
 90% reduction in regression testing time
 Dashboard service which gives historical data helps in tracking the progress

19. 19
© Utthunga 2021
ROI Calculation
ROI = Benefit / Cost
Item Cost Time
Develop manual tests for the site 1 tester @ $50 hr 4 days
Execute manual tests for the site 2 testers @ $50 hr 1 day
Develop automated tests for the site 1 tester @ $50 hr 10 days
Execute automated tests for the site 1 tester @ $50 hr 1 hour to review logs
Maintain manual test cases 1 tester @ $50 hr 1 day every two weeks
Maintain automated test scripts 1 tester @ $50 hr 1 day every week
Hardware for test execution 2 computers @ $1,000
Test software and licenses 2 licenses @ $2,000

20. 20
© Utthunga 2021
ROI Calculation
Cost of automation = price of hardware + price of software + time to develop scripts + (time to maintain scripts x number of times
scripts are executed) + (time to execute scripts x number of times scripts are executed)
Cost of automation = 2 computers @ $1,000 + 2 licenses @ $2,000 + 10 days @ $50 hr + (1 day @ $50 hr x 52 weeks) + (1 hr @
$50 x 52 weeks)
Cost of automation = $2,000 + $4,000 + $4,000 + $20,800 + $2,600
Cost of automation = $33,400
Cost of manual = time to develop test cases + (time to maintain test cases x number of times tests are executed) + (time to execute
manual testing x number of times tests are executed)
Cost of manual = 4 days @ $50 hr + (1 day @ $50 x 26 weeks) + (1 day @ $100 x 52 weeks) Cost of manual = $1,600 + $10,400 +
$41,600
Cost of manual = $53,600 ROI = benefit / cost ROI = (cost of manual – cost of automation) / cost of automation
ROI = ($53,600 - $33,400) / $33,400
ROI = about 61% over the course of the year

21. 21
© Utthunga 2021
Conclusions - Best Approaches for Improving ROI and
Reducing TCO
Detailed Feasibility study
 Technical : Tools, Complexities, Resource availability,
limitations, Advantages, approaches, Frameworks
 Economic : TCO, ROI
 Schedule : Based on the client requirement
 Operational : Direct and Indirect Costs, Maintenance,
Resource Availability, Training

22. 22
© Utthunga 2021
Conclusions - Best Approaches for Improving ROI and
Reducing TCO
 Automate as much as possible:
 Decide and prioritize what to automate (Test case selection)
 Do NOT try to automate everything
 Use standard test automation tools, frameworks & test techniques
 Utilize good quality test data
 Plan End-to-End user scenarios
 Divide testing efforts
 Smaller teams higher benefits
 Shift left approach
 Parallel testing wherever possible
 Risk based and combinatorial testing approaches
 Use simulators / emulators wherever possible
 Documenting each and every activity
 Proper maintenance of scripts and other artefacts
 CI/ CD
 DevOps

23. 23
© Utthunga 2021
Questions?

24. 24
© Utthunga 2021
Upcoming Webinar
 Smart QE testing with AI & ML – September 2021

25. www.utthunga.com
contact@utthunga.com
+91 80 6815 1900
India Germany Japan
USA
Follow us:
/utthunga-technologies
/utthunga
Thank you