SlideShare a Scribd company logo
1 of 21
Download to read offline
open source administration software for education
next generation student system
open source administration software for education 2
Agenda
• Introductions
• The Challenge
• Approach
• Performance Automaton Strategy
• Picking a tool
• Application Results
• Demo
• Next Steps
• Lessons Learned
• Practices that work for us
open source administration software for education 3
Introduction
• Kyle Campos: Performance Engineer for Kuali Student. 11
years experience testing and building automation frameworks,
both functional and performance, for web, unix and mobile
applications. QA Architect for several startups building processes,
tooling and test automation frameworks.
• Ben Clark, QA Manager for Kuali Student. 12 years of experience in
both the computer software and banking industries. Have lead
performance testing efforts using off-the-shelf and home grown
solutions. ASQ Certified Six Sigma Black Belt and Software Quality
Engineer. Earned a B.A. from the University of Montana.
open source administration software for education 4
The Challenge: Performance Testing KS
• Experience
• Management support
• Implementor User Group support
• Lacked common definitions of the
space
• Poorly defined requirements
• Lacked institutional system
environment information
• No Kuali Student profile information
(how many courses over a year,
cycle info, user info, etc)
• No test machines for perf testing
• Lacked tools for the job
• No qualified resources
+ Forces for achieving success - Forces against achieving success
open source administration software for education 5
The Approach
Action Plan
•Hire experienced performance engineer
•Utilize Functional Council and IUG to obtain
requirements and data
•Define the performance automation strategy
•Utilize cloud technologies for test infrastructure
•Pick a tool that meets the need
• Lacked common definitions of the space
• Poorly defined requirements
• Lacked institutional system environment
information
• No Kuali Student profile information (how
many courses over a year, cycle info, user
info, etc)
• No test machines for perf testing
• Lacked tools for the job
• No qualified resources
open source administration software for education 6
Performance Automation Strategy
• Develop test suites based on critical features list
• Benchmark tags & measure performance gain/loss
• Add suites as features are developed
• Use usage targets provided by universities
open source administration software for education 7
Open source tools –lots out there!
Tool Description Pros Cons
JMeter A java desktop app designed to load test
functional behavior and measure
performance
• HTTP protocol Support
• Documented
• List Support
• Didn’t work with GWT/Ajax
httperf A software tool which is able to simulate
load situations for web services, and
calculates performance metrics against the
results
• Familiarity• Robust• Extensible• HTTP protocol Support• Documented• List Support
No dynamic variable support
The Grinder Java load testing framework that can run a
distributed test using many load injector
machines
• Flexible
• Extensible
• Documented
• List Support
Requires Java expertise
Tsung Built for high transaction Env • Built for high trans env
• Extendable
• HTTP protocol Support
• Documented
• Dynamic variable support
Not being used across other KS
products
Homegrown Build a tool in-house to test performance Total control • Expensive
• Time consuming
• Proprietary
• Reinventing the wheel
open source administration software for education 8
Tool Choice - Tsung
• Tsung XML API is a homegrown Ruby framework that
allows for more rapid load test development, maintenance
and execution
• Load Framework is tagged and released as validated
against milestone releases
• Can easily configure custom test suites and load profiles
Tsung + Tsung XML API (Ruby) = Load Automation Framework
open source administration software for education 8
Tool Choice - Tsung
open source administration software for education 6
Test Design
• Test layer doesn’t generate
any XML, all XML
generated by API calls
• Test layer environment
agnostic
(standalone/embedded)
• Tests are added to suites
and given a probability to
run
• API docs dynamically
created through rdoc
• Test configured &
launched by homegrown
utility
open source administration software for education 9
EC2 Network Diagram
open source administration software for education 10
Application Results - Key Metrics
• Concurrent sessions
– driven by launched sessions/min
• Mean transaction times
– feature defined transactions
– highest 10sec mean time
• Iterative growth changes (linear?)
• System resources
– cpu, mem
open source administration software for education 10
Application Results
• Data collected into Numbers(excel) for
calculations
• Published on wiki
• Demo
open source administration software for education 11
Demo
• Driver Demo
open source administration software for education 12
What’s Next?
• Tuning guide documentation
• Configuration
open source administration software for education 13
Example problem walkthrough
• During Create Proposal load test found 4
concurrent sessions
– upped to 16-18 sessions by upping thread pool
– Way too many threads being consumed per
session
• Several rounds of debugging through
configuration changes
• Traced bottleneck to disabling of Rice
caching. Resolved after 5 months
open source administration software for education 14
Lessons Learned
• A proof of concept may not have really
proved anything
• Tsung is great for flexible load sessions but
difficult to manage concurrent sessions
• Your load tool must have runtime dynamic
variable support to work with GWT (Tsung
does)
open source administration software for education 15
Practices that work for us
• Know your goals and requirements
• When in doubt, benchmark
• Review the test approach with entire team
• A coding effort needs a coder; hire the best
• Pick a tool that’s within your resource capability
• Separate data, tests and tool
• Utilize the cloud – it’s cheap and easy
d
open source administration software for education 16
Resources
• Automation testing home
– https://wiki.kuali.org/display/KULSTG/Automation+Testing
• Load Test Framework tester’s guide
– https://wiki.kuali.org/display/KULSTG/Load+Testing+-
+Tester%27s+Guide
• Load testing results
– https://wiki.kuali.org/display/KULSTG/Automation+Testing+
Results
• Contact
– Kyle Campos / Skype: kylecampos_kuali /
kcampos@rsmart.com
– Ben Clark / Skype: benclark_kuali /
benclark@u.washington.edu
d
open source administration software for education 16
Questions?
d
open source administration software for education 17
Timeline
Perf.Bug
D
etected
Perf.B
ug
Fixed
12/09 01/10 02/10 03/10 04/10 05/10 06/10 07/10 08/10 12/1011/1010/1009/10
H
ire
C
om
pleted
Strategy
in
place
T
oolReview
C
ode
C
om
pleteT
oolT
esting
Benchm
ark
Benchm
ark
Benchm
ark
Benchm
ark
M4
M5
M6
M7
M8
M9
Release 1.1
Release 1.0
Benchm
ark
Benchm
arkM3
EC
2
Im
plem
ented
EC
2
U
se
A
pproved

More Related Content

What's hot

Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi
Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang PhiIntroduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi
Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang PhiHo Chi Minh City Software Testing Club
 
Testing Frameworks
Testing FrameworksTesting Frameworks
Testing FrameworksMoataz Nabil
 
Load Testing: See a Bigger Picture, ALM Forum, 2014
Load Testing: See a Bigger Picture, ALM Forum, 2014Load Testing: See a Bigger Picture, ALM Forum, 2014
Load Testing: See a Bigger Picture, ALM Forum, 2014Alexander Podelko
 
Leveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver ExampleLeveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver ExampleTechWell
 
Dev ops != Dev+Ops
Dev ops != Dev+OpsDev ops != Dev+Ops
Dev ops != Dev+OpsShalu Ahuja
 
Streamlining Testing with Visual Studio 2012
Streamlining Testing with Visual Studio 2012Streamlining Testing with Visual Studio 2012
Streamlining Testing with Visual Studio 2012Imaginet
 
Automated Testing with Agile
Automated Testing with AgileAutomated Testing with Agile
Automated Testing with AgileKen McCorkell
 
Automation Tools Overview
Automation Tools OverviewAutomation Tools Overview
Automation Tools OverviewMurageppa-QA
 
Performance Testing
Performance TestingPerformance Testing
Performance TestingSelin Gungor
 
Test automation methodologies
Test automation methodologiesTest automation methodologies
Test automation methodologiesMesut Günes
 
Agile testing - Principles and best practices
Agile testing  - Principles and best practicesAgile testing  - Principles and best practices
Agile testing - Principles and best practicesDr Ganesh Iyer
 
Framework for Web Automation Testing
Framework for Web Automation TestingFramework for Web Automation Testing
Framework for Web Automation TestingTaras Lytvyn
 
How To Transform the Manual Testing Process to Incorporate Test Automation
How To Transform the Manual Testing Process to Incorporate Test AutomationHow To Transform the Manual Testing Process to Incorporate Test Automation
How To Transform the Manual Testing Process to Incorporate Test AutomationRanorex
 
ISTQB Advanced Level Test Automation Engineering Training
ISTQB Advanced Level Test Automation Engineering TrainingISTQB Advanced Level Test Automation Engineering Training
ISTQB Advanced Level Test Automation Engineering TrainingHiraQureshi22
 
Continuous test automation
Continuous test automationContinuous test automation
Continuous test automationMacharla Pradeep
 
Combining Automated Functional And Load Testing
Combining Automated Functional And Load TestingCombining Automated Functional And Load Testing
Combining Automated Functional And Load TestingRanorex
 
How Manual Testers Can Break into Automation Without Programming Skills
How Manual Testers Can Break into Automation Without Programming SkillsHow Manual Testers Can Break into Automation Without Programming Skills
How Manual Testers Can Break into Automation Without Programming SkillsRanorex
 

What's hot (20)

Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi
Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang PhiIntroduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi
Introduction to Back End Automation Testing - Nguyen Vu Hoang, Hoang Phi
 
Testing Frameworks
Testing FrameworksTesting Frameworks
Testing Frameworks
 
Common Web UI Problems Transforming Manual to Automation
Common Web UI Problems Transforming Manual to Automation Common Web UI Problems Transforming Manual to Automation
Common Web UI Problems Transforming Manual to Automation
 
Test Automation
Test AutomationTest Automation
Test Automation
 
Load Testing: See a Bigger Picture, ALM Forum, 2014
Load Testing: See a Bigger Picture, ALM Forum, 2014Load Testing: See a Bigger Picture, ALM Forum, 2014
Load Testing: See a Bigger Picture, ALM Forum, 2014
 
Tashy's resumé
Tashy's resuméTashy's resumé
Tashy's resumé
 
Leveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver ExampleLeveraging Open Source Automation: A Selenium WebDriver Example
Leveraging Open Source Automation: A Selenium WebDriver Example
 
Dev ops != Dev+Ops
Dev ops != Dev+OpsDev ops != Dev+Ops
Dev ops != Dev+Ops
 
Streamlining Testing with Visual Studio 2012
Streamlining Testing with Visual Studio 2012Streamlining Testing with Visual Studio 2012
Streamlining Testing with Visual Studio 2012
 
Automated Testing with Agile
Automated Testing with AgileAutomated Testing with Agile
Automated Testing with Agile
 
Automation Tools Overview
Automation Tools OverviewAutomation Tools Overview
Automation Tools Overview
 
Performance Testing
Performance TestingPerformance Testing
Performance Testing
 
Test automation methodologies
Test automation methodologiesTest automation methodologies
Test automation methodologies
 
Agile testing - Principles and best practices
Agile testing  - Principles and best practicesAgile testing  - Principles and best practices
Agile testing - Principles and best practices
 
Framework for Web Automation Testing
Framework for Web Automation TestingFramework for Web Automation Testing
Framework for Web Automation Testing
 
How To Transform the Manual Testing Process to Incorporate Test Automation
How To Transform the Manual Testing Process to Incorporate Test AutomationHow To Transform the Manual Testing Process to Incorporate Test Automation
How To Transform the Manual Testing Process to Incorporate Test Automation
 
ISTQB Advanced Level Test Automation Engineering Training
ISTQB Advanced Level Test Automation Engineering TrainingISTQB Advanced Level Test Automation Engineering Training
ISTQB Advanced Level Test Automation Engineering Training
 
Continuous test automation
Continuous test automationContinuous test automation
Continuous test automation
 
Combining Automated Functional And Load Testing
Combining Automated Functional And Load TestingCombining Automated Functional And Load Testing
Combining Automated Functional And Load Testing
 
How Manual Testers Can Break into Automation Without Programming Skills
How Manual Testers Can Break into Automation Without Programming SkillsHow Manual Testers Can Break into Automation Without Programming Skills
How Manual Testers Can Break into Automation Without Programming Skills
 

Viewers also liked

NCURA PRA 2015 - Sponsored Program Management in the Cloud
NCURA PRA 2015 - Sponsored Program Management in the CloudNCURA PRA 2015 - Sponsored Program Management in the Cloud
NCURA PRA 2015 - Sponsored Program Management in the CloudMichael Wallgren
 
My First 90 Days - Strategies for Success
My First 90 Days - Strategies for SuccessMy First 90 Days - Strategies for Success
My First 90 Days - Strategies for SuccessSuresh Kodoor
 
A PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATION
A PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATIONA PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATION
A PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATIONSagar Anand
 
30 60 90 Day Sales Plan
30 60 90 Day Sales Plan30 60 90 Day Sales Plan
30 60 90 Day Sales Plannatevans65
 
21 Hidden LinkedIn Hacks Revealed
21 Hidden LinkedIn Hacks Revealed21 Hidden LinkedIn Hacks Revealed
21 Hidden LinkedIn Hacks RevealedEmma Brudner
 
Diabetes powerpoint
Diabetes powerpointDiabetes powerpoint
Diabetes powerpointmldanforth
 

Viewers also liked (11)

Viewing informationinkc ss
Viewing informationinkc ssViewing informationinkc ss
Viewing informationinkc ss
 
Kuali Ready Overview
Kuali Ready OverviewKuali Ready Overview
Kuali Ready Overview
 
NCURA PRA 2015 - Sponsored Program Management in the Cloud
NCURA PRA 2015 - Sponsored Program Management in the CloudNCURA PRA 2015 - Sponsored Program Management in the Cloud
NCURA PRA 2015 - Sponsored Program Management in the Cloud
 
Award Pre-Closeout, Closeout, Record Retention-screen shots
Award Pre-Closeout, Closeout, Record Retention-screen shotsAward Pre-Closeout, Closeout, Record Retention-screen shots
Award Pre-Closeout, Closeout, Record Retention-screen shots
 
Proposal Development in KC
Proposal Development in KCProposal Development in KC
Proposal Development in KC
 
My First 90 Days - Strategies for Success
My First 90 Days - Strategies for SuccessMy First 90 Days - Strategies for Success
My First 90 Days - Strategies for Success
 
Brainstem Lesions
Brainstem LesionsBrainstem Lesions
Brainstem Lesions
 
A PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATION
A PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATIONA PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATION
A PROJECT REPORT ON EXPORT PROCESS AND DOCUMENTATION
 
30 60 90 Day Sales Plan
30 60 90 Day Sales Plan30 60 90 Day Sales Plan
30 60 90 Day Sales Plan
 
21 Hidden LinkedIn Hacks Revealed
21 Hidden LinkedIn Hacks Revealed21 Hidden LinkedIn Hacks Revealed
21 Hidden LinkedIn Hacks Revealed
 
Diabetes powerpoint
Diabetes powerpointDiabetes powerpoint
Diabetes powerpoint
 

Similar to Performance Testing your Kuali Student Product

Response time difference analysis of performance testing tools
Response time difference analysis of performance testing toolsResponse time difference analysis of performance testing tools
Response time difference analysis of performance testing toolsSpoorthi Sham
 
5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test AutomationSauce Labs
 
Lecture #6. automation testing (andrey oleynik)
Lecture #6. automation testing (andrey oleynik)Lecture #6. automation testing (andrey oleynik)
Lecture #6. automation testing (andrey oleynik)Andrey Oleynik
 
7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with Automation7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with AutomationWorksoft
 
Test Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdfTest Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdfDiana Gray, MBA
 
Automation Culture: Essential to Agile Success
Automation Culture: Essential to Agile SuccessAutomation Culture: Essential to Agile Success
Automation Culture: Essential to Agile SuccessTechWell
 
_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt
_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt
_VoicePPT_QA_Testing_Training_4_Days_Schedule.pptAnilKumarARS
 
Meet Big Agile: Testing on Large-Scale Projects
Meet Big Agile: Testing on Large-Scale ProjectsMeet Big Agile: Testing on Large-Scale Projects
Meet Big Agile: Testing on Large-Scale ProjectsTechWell
 
The QA/Testing Process
The QA/Testing ProcessThe QA/Testing Process
The QA/Testing ProcessSynerzip
 
Test Automation using UiPath Test Suite - Developer Circle Part-4.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-4.pdfTest Automation using UiPath Test Suite - Developer Circle Part-4.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-4.pdfDiana Gray, MBA
 
Sucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QASucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QASucheta Kale
 
No Devops Without Continuous Testing
No Devops Without Continuous TestingNo Devops Without Continuous Testing
No Devops Without Continuous TestingParasoft
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Kari Kakkonen
 
#ATAGTR2020 Presentation - Universal Test Automation Framework
#ATAGTR2020 Presentation - Universal Test Automation Framework#ATAGTR2020 Presentation - Universal Test Automation Framework
#ATAGTR2020 Presentation - Universal Test Automation FrameworkAgile Testing Alliance
 
Agile Methodology - Software Engineering
Agile Methodology - Software EngineeringAgile Methodology - Software Engineering
Agile Methodology - Software EngineeringPurvik Rana
 
Multiple Dimensions of Load Testing
Multiple Dimensions of Load TestingMultiple Dimensions of Load Testing
Multiple Dimensions of Load TestingAlexander Podelko
 
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".Emerasoft, solutions to collaborate
 

Similar to Performance Testing your Kuali Student Product (20)

Response time difference analysis of performance testing tools
Response time difference analysis of performance testing toolsResponse time difference analysis of performance testing tools
Response time difference analysis of performance testing tools
 
5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation
 
Lecture #6. automation testing (andrey oleynik)
Lecture #6. automation testing (andrey oleynik)Lecture #6. automation testing (andrey oleynik)
Lecture #6. automation testing (andrey oleynik)
 
SachinKumar_Imp
SachinKumar_ImpSachinKumar_Imp
SachinKumar_Imp
 
7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with Automation7 Tips from Siemens Energy for Success with Automation
7 Tips from Siemens Energy for Success with Automation
 
Test Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdfTest Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-3 - 07262022.pdf
 
Automation Culture: Essential to Agile Success
Automation Culture: Essential to Agile SuccessAutomation Culture: Essential to Agile Success
Automation Culture: Essential to Agile Success
 
Performance Testing Overview
Performance Testing OverviewPerformance Testing Overview
Performance Testing Overview
 
_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt
_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt
_VoicePPT_QA_Testing_Training_4_Days_Schedule.ppt
 
Meet Big Agile: Testing on Large-Scale Projects
Meet Big Agile: Testing on Large-Scale ProjectsMeet Big Agile: Testing on Large-Scale Projects
Meet Big Agile: Testing on Large-Scale Projects
 
The QA/Testing Process
The QA/Testing ProcessThe QA/Testing Process
The QA/Testing Process
 
Test Automation using UiPath Test Suite - Developer Circle Part-4.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-4.pdfTest Automation using UiPath Test Suite - Developer Circle Part-4.pdf
Test Automation using UiPath Test Suite - Developer Circle Part-4.pdf
 
Sucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QASucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QA
 
No Devops Without Continuous Testing
No Devops Without Continuous TestingNo Devops Without Continuous Testing
No Devops Without Continuous Testing
 
Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle Agile Testing – embedding testing into agile software development lifecycle
Agile Testing – embedding testing into agile software development lifecycle
 
UNIT IV.ppt
UNIT IV.pptUNIT IV.ppt
UNIT IV.ppt
 
#ATAGTR2020 Presentation - Universal Test Automation Framework
#ATAGTR2020 Presentation - Universal Test Automation Framework#ATAGTR2020 Presentation - Universal Test Automation Framework
#ATAGTR2020 Presentation - Universal Test Automation Framework
 
Agile Methodology - Software Engineering
Agile Methodology - Software EngineeringAgile Methodology - Software Engineering
Agile Methodology - Software Engineering
 
Multiple Dimensions of Load Testing
Multiple Dimensions of Load TestingMultiple Dimensions of Load Testing
Multiple Dimensions of Load Testing
 
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
Webinar: "5 semplici passi per migliorare la Quality e i processi di Test".
 

Recently uploaded

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 

Recently uploaded (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 

Performance Testing your Kuali Student Product

  • 1. open source administration software for education next generation student system
  • 2. open source administration software for education 2 Agenda • Introductions • The Challenge • Approach • Performance Automaton Strategy • Picking a tool • Application Results • Demo • Next Steps • Lessons Learned • Practices that work for us
  • 3. open source administration software for education 3 Introduction • Kyle Campos: Performance Engineer for Kuali Student. 11 years experience testing and building automation frameworks, both functional and performance, for web, unix and mobile applications. QA Architect for several startups building processes, tooling and test automation frameworks. • Ben Clark, QA Manager for Kuali Student. 12 years of experience in both the computer software and banking industries. Have lead performance testing efforts using off-the-shelf and home grown solutions. ASQ Certified Six Sigma Black Belt and Software Quality Engineer. Earned a B.A. from the University of Montana.
  • 4. open source administration software for education 4 The Challenge: Performance Testing KS • Experience • Management support • Implementor User Group support • Lacked common definitions of the space • Poorly defined requirements • Lacked institutional system environment information • No Kuali Student profile information (how many courses over a year, cycle info, user info, etc) • No test machines for perf testing • Lacked tools for the job • No qualified resources + Forces for achieving success - Forces against achieving success
  • 5. open source administration software for education 5 The Approach Action Plan •Hire experienced performance engineer •Utilize Functional Council and IUG to obtain requirements and data •Define the performance automation strategy •Utilize cloud technologies for test infrastructure •Pick a tool that meets the need • Lacked common definitions of the space • Poorly defined requirements • Lacked institutional system environment information • No Kuali Student profile information (how many courses over a year, cycle info, user info, etc) • No test machines for perf testing • Lacked tools for the job • No qualified resources
  • 6. open source administration software for education 6 Performance Automation Strategy • Develop test suites based on critical features list • Benchmark tags & measure performance gain/loss • Add suites as features are developed • Use usage targets provided by universities
  • 7. open source administration software for education 7 Open source tools –lots out there! Tool Description Pros Cons JMeter A java desktop app designed to load test functional behavior and measure performance • HTTP protocol Support • Documented • List Support • Didn’t work with GWT/Ajax httperf A software tool which is able to simulate load situations for web services, and calculates performance metrics against the results • Familiarity• Robust• Extensible• HTTP protocol Support• Documented• List Support No dynamic variable support The Grinder Java load testing framework that can run a distributed test using many load injector machines • Flexible • Extensible • Documented • List Support Requires Java expertise Tsung Built for high transaction Env • Built for high trans env • Extendable • HTTP protocol Support • Documented • Dynamic variable support Not being used across other KS products Homegrown Build a tool in-house to test performance Total control • Expensive • Time consuming • Proprietary • Reinventing the wheel
  • 8. open source administration software for education 8 Tool Choice - Tsung • Tsung XML API is a homegrown Ruby framework that allows for more rapid load test development, maintenance and execution • Load Framework is tagged and released as validated against milestone releases • Can easily configure custom test suites and load profiles Tsung + Tsung XML API (Ruby) = Load Automation Framework
  • 9. open source administration software for education 8 Tool Choice - Tsung
  • 10. open source administration software for education 6 Test Design • Test layer doesn’t generate any XML, all XML generated by API calls • Test layer environment agnostic (standalone/embedded) • Tests are added to suites and given a probability to run • API docs dynamically created through rdoc • Test configured & launched by homegrown utility
  • 11. open source administration software for education 9 EC2 Network Diagram
  • 12. open source administration software for education 10 Application Results - Key Metrics • Concurrent sessions – driven by launched sessions/min • Mean transaction times – feature defined transactions – highest 10sec mean time • Iterative growth changes (linear?) • System resources – cpu, mem
  • 13. open source administration software for education 10 Application Results • Data collected into Numbers(excel) for calculations • Published on wiki • Demo
  • 14. open source administration software for education 11 Demo • Driver Demo
  • 15. open source administration software for education 12 What’s Next? • Tuning guide documentation • Configuration
  • 16. open source administration software for education 13 Example problem walkthrough • During Create Proposal load test found 4 concurrent sessions – upped to 16-18 sessions by upping thread pool – Way too many threads being consumed per session • Several rounds of debugging through configuration changes • Traced bottleneck to disabling of Rice caching. Resolved after 5 months
  • 17. open source administration software for education 14 Lessons Learned • A proof of concept may not have really proved anything • Tsung is great for flexible load sessions but difficult to manage concurrent sessions • Your load tool must have runtime dynamic variable support to work with GWT (Tsung does)
  • 18. open source administration software for education 15 Practices that work for us • Know your goals and requirements • When in doubt, benchmark • Review the test approach with entire team • A coding effort needs a coder; hire the best • Pick a tool that’s within your resource capability • Separate data, tests and tool • Utilize the cloud – it’s cheap and easy
  • 19. d open source administration software for education 16 Resources • Automation testing home – https://wiki.kuali.org/display/KULSTG/Automation+Testing • Load Test Framework tester’s guide – https://wiki.kuali.org/display/KULSTG/Load+Testing+- +Tester%27s+Guide • Load testing results – https://wiki.kuali.org/display/KULSTG/Automation+Testing+ Results • Contact – Kyle Campos / Skype: kylecampos_kuali / kcampos@rsmart.com – Ben Clark / Skype: benclark_kuali / benclark@u.washington.edu
  • 20. d open source administration software for education 16 Questions?
  • 21. d open source administration software for education 17 Timeline Perf.Bug D etected Perf.B ug Fixed 12/09 01/10 02/10 03/10 04/10 05/10 06/10 07/10 08/10 12/1011/1010/1009/10 H ire C om pleted Strategy in place T oolReview C ode C om pleteT oolT esting Benchm ark Benchm ark Benchm ark Benchm ark M4 M5 M6 M7 M8 M9 Release 1.1 Release 1.0 Benchm ark Benchm arkM3 EC 2 Im plem ented EC 2 U se A pproved

Editor's Notes

  1. Force Field Diagram is used to weigh up or lay out the points for and against a potential action or task