Faced with two critical goals and low resources, the Kuali Student Test Team developed and implemented a test automation framework based on an open source tool ? Tsung and Amazon?s Elastic Compute Cloud (EC2). This automation framework netted success right away as it allowed the team to identify and regress critical performance issues through out development.
Kuali developers and test engineers, and technical staff from implementing institutions will learn how industry best practices for performance testing were applied to Kuali Student and will walk away with guidance on how to setup and run an open source performance testing tool to support their needs (live demonstration included).
2. open source administration software for education 2
Agenda
• Introductions
• The Challenge
• Approach
• Performance Automaton Strategy
• Picking a tool
• Application Results
• Demo
• Next Steps
• Lessons Learned
• Practices that work for us
3. open source administration software for education 3
Introduction
• Kyle Campos: Performance Engineer for Kuali Student. 11
years experience testing and building automation frameworks,
both functional and performance, for web, unix and mobile
applications. QA Architect for several startups building processes,
tooling and test automation frameworks.
• Ben Clark, QA Manager for Kuali Student. 12 years of experience in
both the computer software and banking industries. Have lead
performance testing efforts using off-the-shelf and home grown
solutions. ASQ Certified Six Sigma Black Belt and Software Quality
Engineer. Earned a B.A. from the University of Montana.
4. open source administration software for education 4
The Challenge: Performance Testing KS
• Experience
• Management support
• Implementor User Group support
• Lacked common definitions of the
space
• Poorly defined requirements
• Lacked institutional system
environment information
• No Kuali Student profile information
(how many courses over a year,
cycle info, user info, etc)
• No test machines for perf testing
• Lacked tools for the job
• No qualified resources
+ Forces for achieving success - Forces against achieving success
5. open source administration software for education 5
The Approach
Action Plan
•Hire experienced performance engineer
•Utilize Functional Council and IUG to obtain
requirements and data
•Define the performance automation strategy
•Utilize cloud technologies for test infrastructure
•Pick a tool that meets the need
• Lacked common definitions of the space
• Poorly defined requirements
• Lacked institutional system environment
information
• No Kuali Student profile information (how
many courses over a year, cycle info, user
info, etc)
• No test machines for perf testing
• Lacked tools for the job
• No qualified resources
6. open source administration software for education 6
Performance Automation Strategy
• Develop test suites based on critical features list
• Benchmark tags & measure performance gain/loss
• Add suites as features are developed
• Use usage targets provided by universities
7. open source administration software for education 7
Open source tools –lots out there!
Tool Description Pros Cons
JMeter A java desktop app designed to load test
functional behavior and measure
performance
• HTTP protocol Support
• Documented
• List Support
• Didn’t work with GWT/Ajax
httperf A software tool which is able to simulate
load situations for web services, and
calculates performance metrics against the
results
• Familiarity• Robust• Extensible• HTTP protocol Support• Documented• List Support
No dynamic variable support
The Grinder Java load testing framework that can run a
distributed test using many load injector
machines
• Flexible
• Extensible
• Documented
• List Support
Requires Java expertise
Tsung Built for high transaction Env • Built for high trans env
• Extendable
• HTTP protocol Support
• Documented
• Dynamic variable support
Not being used across other KS
products
Homegrown Build a tool in-house to test performance Total control • Expensive
• Time consuming
• Proprietary
• Reinventing the wheel
8. open source administration software for education 8
Tool Choice - Tsung
• Tsung XML API is a homegrown Ruby framework that
allows for more rapid load test development, maintenance
and execution
• Load Framework is tagged and released as validated
against milestone releases
• Can easily configure custom test suites and load profiles
Tsung + Tsung XML API (Ruby) = Load Automation Framework
10. open source administration software for education 6
Test Design
• Test layer doesn’t generate
any XML, all XML
generated by API calls
• Test layer environment
agnostic
(standalone/embedded)
• Tests are added to suites
and given a probability to
run
• API docs dynamically
created through rdoc
• Test configured &
launched by homegrown
utility
12. open source administration software for education 10
Application Results - Key Metrics
• Concurrent sessions
– driven by launched sessions/min
• Mean transaction times
– feature defined transactions
– highest 10sec mean time
• Iterative growth changes (linear?)
• System resources
– cpu, mem
13. open source administration software for education 10
Application Results
• Data collected into Numbers(excel) for
calculations
• Published on wiki
• Demo
15. open source administration software for education 12
What’s Next?
• Tuning guide documentation
• Configuration
16. open source administration software for education 13
Example problem walkthrough
• During Create Proposal load test found 4
concurrent sessions
– upped to 16-18 sessions by upping thread pool
– Way too many threads being consumed per
session
• Several rounds of debugging through
configuration changes
• Traced bottleneck to disabling of Rice
caching. Resolved after 5 months
17. open source administration software for education 14
Lessons Learned
• A proof of concept may not have really
proved anything
• Tsung is great for flexible load sessions but
difficult to manage concurrent sessions
• Your load tool must have runtime dynamic
variable support to work with GWT (Tsung
does)
18. open source administration software for education 15
Practices that work for us
• Know your goals and requirements
• When in doubt, benchmark
• Review the test approach with entire team
• A coding effort needs a coder; hire the best
• Pick a tool that’s within your resource capability
• Separate data, tests and tool
• Utilize the cloud – it’s cheap and easy
19. d
open source administration software for education 16
Resources
• Automation testing home
– https://wiki.kuali.org/display/KULSTG/Automation+Testing
• Load Test Framework tester’s guide
– https://wiki.kuali.org/display/KULSTG/Load+Testing+-
+Tester%27s+Guide
• Load testing results
– https://wiki.kuali.org/display/KULSTG/Automation+Testing+
Results
• Contact
– Kyle Campos / Skype: kylecampos_kuali /
kcampos@rsmart.com
– Ben Clark / Skype: benclark_kuali /
benclark@u.washington.edu
21. d
open source administration software for education 17
Timeline
Perf.Bug
D
etected
Perf.B
ug
Fixed
12/09 01/10 02/10 03/10 04/10 05/10 06/10 07/10 08/10 12/1011/1010/1009/10
H
ire
C
om
pleted
Strategy
in
place
T
oolReview
C
ode
C
om
pleteT
oolT
esting
Benchm
ark
Benchm
ark
Benchm
ark
Benchm
ark
M4
M5
M6
M7
M8
M9
Release 1.1
Release 1.0
Benchm
ark
Benchm
arkM3
EC
2
Im
plem
ented
EC
2
U
se
A
pproved
Editor's Notes
Force Field Diagram is used to weigh up or lay out the points for and against a potential action or task