Software Test automation tools are available under several categories such as commercial, free software, open source software and etc. In this paper Open Source Software Testing Tools will be discussed. Open source software test automation tools may be practical alternatives to popular closed-source commercial applications and some open source tools offers features or performance benefits that exceed their commercial counterparts. The source code is openly published for use and/or modification from its original design, free of charge. And these are usually available under a license defined by the Open Source Initiative.

1. ICT 3003 – Software Quality Assurance
Assignment I
Open Source
Software Quality Assurance Testing Tools
University of Colombo School of Computing
Prepared by
Waruna Kodituwakku

2. Page | 1
Table of Contents
1. Introduction............................................................................................................................2
1.1 Open Source Software Testing Tools ..............................................................................2
1.2 Types of Software Testing Tools.....................................................................................2
2. Popular Open Source Software Testing Tools.......................................................................3
2.1 Selenium ......................................................................................................................3
2.2 FitNesse........................................................................................................................3
2.3 Watir ............................................................................................................................4
2.4 Apache JMeter .............................................................................................................4
2.5 Robotium......................................................................................................................4
2.6 Bugzilla........................................................................................................................5
2. Comparison of Selenium, Bugzilla and JMeter .....................................................................5
2.1 Analysis of Strengths and Weaknesses of Selected Tools...............................................6
2.1.1 Selenium ...................................................................................................................6
2.1.2 Bugzilla.....................................................................................................................6
2.1.3 JMeter .......................................................................................................................7
3. Future of Software Testing Tools ..........................................................................................7
4. Conclusion .............................................................................................................................7
5. References..............................................................................................................................8

3. Page | 2
1. Introduction
Software Quality Assurance (SQA) involves monitoring and improving the software testing
process, making sure that any agreed-upon standards and procedures are followed, and
ensuring that problems are found and treated. In today’s context SQA is essential to every
software development process. Software Testing is a costly and time consuming process in
software development. Therefore, software testing tools are often deployed to automate the
process in order to reduce cost and improve efficiency.
SQA now incorporates more sophisticated tools that enhance software testing capabilities.
Testing automation tools enable and help developers and testers to automate the entire
process of testing in software development which include; find and report defects and issues
faster; confirm tests meet customer specifications and requirement-traceability; have a fine
way for communication; archive the test plan and execute it manually or automatically;
maintain reusable tests; and document, verify and audit which tests were run for each project.
1.1 Open Source Software Testing Tools
Software Test automation tools are available under several categories such as commercial,
free software, open source software and etc. In this paper Open Source Software Testing
Tools will be discussed.
Open source software test automation tools may be practical alternatives to popular closed-
source commercial applications and some open source tools offers features or performance
benefits that exceed their commercial counterparts. The source code is openly published for
use and/or modification from its original design, free of charge. And these are usually
available under a license defined by the Open Source Initiative.
1.2 Types of Software Testing Tools
Software testing tools can be categorized under several bases. Following table is shows
software testing tools categorized under the testing activity or the process in which they are
used.
Type Description Open Source
Examples
Functional
Testing
Functional testing is the process of confirming the
functionality of the application. This type of testing
ignores the internal parts and focuses on the output and
determines that it is as per requirement or not.
Selenium, Watir,
WatiN, Canoo
WebTest, Cucumber
Source Code
Testing
Source Code Testing tools are designed to analyze source
code and/or compiled version to verify the source code
and help identify the security flaws.
SWAAT, Flawfinder,
PMD, Brakeman,
RIPS, RATS
Performance
Testing
To check whether system meets performance
requirements
Apache JMeter,
FunkLoad, OpenSTA
Database
Testing
Database testing is about checking exact values which
have been retrieved from the database by the web or
desktop application. Data should exactly match with
those in the records that are stored in the Database
Database Opensource
Test Suite,
DBMonster, DB
Sanity, DBFeeder
Bug Tracking
Tools
These software applications are designed to help keep
track of reported software bugs in software development
efforts.
Bugzilla, Mantis,
Trac, Redmine,
Request Tracker,

4. Page | 3
Test
Management
Tools
Test management tools are used to structure automated
tests and manual test processes, and to manage multiple
environments easily. Quality assurance teams use these
types of tools as a single application for managing test
cases, environments, automated tests, defects and project
tasks.
Test Manager,
TETware,
Requirements and
Testing Hub (RTH)
2. Popular Open Source Software Testing Tools
2.1 Selenium
 Category : Functional Test Tool, Web Application Testing
 License: Apache License 2.0
Selenium is composed of multiple software tools. Those are Selenium IDE, Selenium
WebDirver and Selenium Remote Control. Selenium is a portable software testing framework
for web applications.. It supports a number of accepted programming languages, including
C#, Java, Groovy, Perl, PHP, Python and Ruby for writing test cases. The tests can then be
run against many modern web browsers. Selenium is deployed on Windows, Linux, and
Macintosh platforms.
Features
 Record and playback
 Intelligent field selection
 Auto complete for all common Selenium commands
 Walk through tests, debug and set breakpoints
 Save tests as Selenese, Ruby scripts, or other formats
 Option to automatically assert the title of every page
 Easy customization through plug-ins
2.2 FitNesse
 Category : Test Management Tools
 License : Common Public License (CPL)
This tool allows users to specify and run acceptance tests for software systems (an agile style
of black-box testing acceptance and regression testing). FitNesse facilitates detailed readable
description of system function and it supports programming languages such as Java, C++,
Python, Ruby, Delphi, C#, etc.
Features
 Wiki engine
 Embedded web server
 Test engines (FIT, SLIM)
 Automated input generation and test case creation
 Additional libraries

5. Page | 4
2.3 Watir
 Category: Functional Test Tool, Web Application Testing
 License: BSD License
Watir is a free, open-source functional testing tool for automating browser-based tests of web
applications. It works with applications written in any language. Watir project consists of
several smaller projects. Some of them are Watir-classic, Watir-webdriver and Watirspec.
Features
 Handles multiple browser windows with attach method.
 Supports JavaScript events
 Supports frames and nested frames
 Supports visible and invisible running of test cases
 Automatic page loading waiting
 Supports hidden fields and saving images in a page and Screen capturing
2.4 Apache JMeter
 Categories: Performance Testing, Load Testing
 License: Apache License 2.0
Apache JMeter is a 100% pure Java desktop application designed to load test functional
behavior and measure performance. It was originally designed for testing Web Applications
but has since expanded to other test functions. JMeter architecture is based on plugins. Most
of its features are implemented with plugins. Off-site developers can easily extend JMeter
with custom plugins.
Features
 Can load and performance test many different server types (Web - HTTP, HTTPS,
SOAP, Database via JDBC, LDAP, JMS, Mail - SMTP(S), POP3(S) and IMAP(S),
Native commands or shell scripts)
 Complete portability and 100% Java purity.
 Full multithreading framework allows concurrent sampling by many threads and
simultaneous sampling of different functions by separate thread groups. Careful GUI
design allows faster operation and more precise timings.
 Caching and offline analysis/replaying of test results.
 Highly Extensible
2.5 Robotium
 Categories : Mobile Testing tool, Android Application Test Tool
 License: Apache License 2.0
Robotium is a test framework created to make it easy to write powerful and robust automatic
black-box test cases for Android applications. With the support of Robotium, test case
developers can write function, system and acceptance test scenarios, spanning multiple
Android activities.

6. Page | 5
Features
 Fast test case execution.
 The framework handles multiple Android activities automatically.
 Minimal time needed to write solid test cases.
 Readability of test cases is greatly improved, compared to standard instrumentation
tests.
 Test cases are more robust due to the run-time binding to GUI components.
 Integrates smoothly with Maven or Ant to run tests as part of continuous integration
2.6 Bugzilla
 Category: Bug Tracking Tool
 License: Mozilla Public License
Bugzilla is a web-based bug tracker and testing tool. This allows individuals or groups of
developers to keep track of outstanding bugs in their product effectively. Generally Bugzilla
tracks bugs and code changes, communicates with teammates, submits and reviews patches
and manages quality assurance. Bugzilla is written in Perl, and works on various databases
including MySQL and Oracle.
Features
 Advanced search capabilities
 Email notifications controlled by user preferences and integrated email capabilities
 Bug lists in multiple formats (Atom, iCal, etc.) and reports and charts
 Automatic Duplicate Bug Detection
 Time Tracking, Request System, Private Attachments and Comments
 Automatic Username Completion or Drop-Down User Lists and Patch Viewer
 Multiple Authentication Methods and Support for Multiple Database Engines

2. Comparison of Selenium, Bugzilla and JMeter
Selenium (IDE,
WebDriver and Remote
Control)
BugZilla Apache JMeter
Testing Type Functional Testing,
Regression Testing
Bug Tracking Performance Testing,
Load Testing
Developed by ThoughtWorks Inc. Mozilla Foundation Apache Software
Foundation
Written in Java (Programming
Language)
Perl (Programming
Language)
Java (Programming
Language)
Operating System Cross-platform (Windows,
Linux, OS X, Solaris)
Cross-platform Cross-platform
User Interface
Available
Integrated into ALM & IDE,
Standalone application,Web
based interface
Web based Interface Batch Mode, Stand
Alone application
Record and
Playback Option
Available No Available

7. Page | 6
Application Type It can be used to test only
web based applications.
It can be used to test
only web based
applications
It can test web-based
as well as desktop
applications.
Integration with
other tools
Allowed Allowed Allowed
Report Generation
Facility
Available Available Available
Multilingual
(Localization)
No Yes No
Plugins, Add-ons,
Customizations
Available Available Available
Installation Required Installation Required Installation Required Installation Required
Documentation and
Community Support
Available Available Available
2.1 Analysis of Strengths and Weaknesses of Selected Tools
2.1.1 Selenium
Strengths
Selenium is one of the most popular open-source automation tools. Selenium provides a
handy unified interface that works with a large number of browsers, and allows writing tests
in almost all the languages (E.g. Java, C#, Python, JavaScript, PHP, Erlang and etc).
Weaknesses
Selenium is written in JavaScript which causes a significant weakness: browsers impose a
pretty strict security model on any JavaScript that they execute in order to protect a user from
malicious scripts. And also it doesn't work directly with Flash.
2.1.2 Bugzilla
Strengths
Main strength of Bugzilla is that it is one of the well established and widely used software in
the industry. Other key strengths of BugZilla can be identified as; easily understood, easily
integrates with several test management tools (e.g. Test Link, Test Runner, Testopia), very
test friendly and contains almost all the features found in commercial products.
Weaknesses
Main weakness of Bugzilla is its complex installation. And even with improvements;
Bugzilla user interface hasn't changed much over the years – a fact that is not attractive for
end-users.

8. Page | 7
2.1.3 JMeter
Strengths
JMeter is an Open Source Tool and it also a Freeware. It supports the distributed testing and
has various target systems and further it is Cross-platform. It has very high extensibility too,
because pluggable samplers allow unlimited testing capabilities. Overall JMeter is a stable,
easy to comprehend and long standing desktop Java application.
Weaknesses
It’s always advisable not to run JMeter on the same machine running the application to be
tested. JMeter may use extensive resources that might affect the other application's
performance. JMeter chart representation looks a bit confusing to some users and
terminology appears to be not very clear. Getting "correct" response times from JMeter
process is very complex. Also remote machines must be declared in a property file before
starting application.
3. Future of Software Testing Tools
Today, software has changed from a standard, single, windows-based platform to multiple
platforms. When considering about growth of software test automation tools we can see that
there will be new areas which will enhance user friendliness, integration between tools,
combining different tools to fit user needs.
With script-less automation, new tools will be shaped so that users could setup automation
testing in a simple and logical way. It means that technical skills and programming
knowledge will not be required any more. Another aspect is that Visual Test Automation is
the future for test object recognition in the modern software application development and
testing world. It contains algorithms built on OCR (object character recognition), ICR (image
character recognition) and gesture recognition technology (interpreting human gestures).
Technology based on visual based object recognition will allow testers to execute scripts
cross-client and cross-platform. Automated testing tools will become less complex, easy to
use and adaptive. Users will not need weeks to learn how to use testing tools. This kind of
software testing will be even faster and will give exact results.
4. Conclusion
To be successful in the challenging business of software development, it is essential to
provide a quality product to the customers. Such process is known as ‘Software Quality
Assurance’ (SQA). Software Testing Tools play a vital role in SQA. Testing is more than just
debugging. Testing is used to locate defects and correct them and also in validation,
verification process, and reliability measurement. There are commercial and free testing tools
as well as open source testing tools. Among them, Open Source Testing Tools seems to the
most popular. Comparatively it provides more flexibility, more eases of usage and allows tool
customization also. In this paper several popular and notable open source software testing
tools have been categorized, analyzed, compared and their respective strengths and
weaknesses discussed. The future of the testing tools is indicative that they will become less
complex, more easy to use more adaptive and faster too.

9. Page | 8
5. References
1. Open Source Testing. 2013. Open source functional testing tools. [ONLINE]
Available at: http://www.opensourcetesting.org/functional.php. [Accessed 15 March
2013].
2. VTB. 2013. Article - Test Automation Tools. [ONLINE] Available at:
http://www.vietnamesetestingboard.org/zbxe/?document_srl=555503. [Accessed 25
March 2013].
3. Wikipedia, the free encyclopedia. 2013. Test automation - Wikipedia, the free
encyclopedia. [ONLINE] Available at: http://en.wikipedia.org/wiki/Test_automation.
[Accessed 25 March 2013].
4. SparkWiz. 2013. The Types and Benefits of Software Testing Tools - SparkWiz |
Spark Wiz. [ONLINE] Available at: http://sparkwiz.com/tips-n-tricks/the-types-and-
benefits-of-software-testing-tools/. [Accessed 25 March 2013].
5. OWASP. 2013. Source Code Analysis Tools - OWASP. [ONLINE] Available at:
https://www.owasp.org/index.php/Source_Code_Analysis_Tools. [Accessed 25 March
2013].
6. Google Project Hosting. 2013. robotium - The world's leading Android™ test
automation framework - Google Project Hosting . [ONLINE] Available at:
http://code.google.com/p/robotium/. [Accessed 25 March 2013].
7. Wikipedia, the free encyclopedia. 2013. Bugzilla - Wikipedia, the free encyclopedia.
[ONLINE] Available at: http://en.wikipedia.org/wiki/Bugzilla. [Accessed 25 March
2013].
8. TheServerSide.com. 2013. Open source software testing tools news, help and
research - TheServerSide.com. [ONLINE] Available at:
http://www.theserverside.com/resources/Open-source-software-testing-tools. [Accessed
25 March 2013].
9. IndianGNU. 2013. IndianGNU.org » Blog Archive » Top 5 Open Source Software
Testing QA tools. [ONLINE] Available at: http://www.indiangnu.org/2011/top-5-open-
source-software-testing-qa-tools/. [Accessed 25 March 2013].
10. Xoriant Software Product Engineering. 2013. Selenium Open Source Test Automation
Tool: An Overview | Xoriant Software Product Engineering. [ONLINE] Available
at: http://www.xoriant.com/blog/software-testing-and-qa/selenium-open-source-test-
automation-tool-an-overview.html. [Accessed 25 March 2013].
11. Apache JMeter™. 2013. Apache JMeter - Apache JMeter™. [ONLINE] Available at:
http://jmeter.apache.org/index.html. [Accessed 25 March 2013].

10. Page | 9
12. CSS Corp Labs. 2013. CSS Corp Labs: Testing Tools Comparison. [ONLINE]
Available at: http://cssinnovations.blogspot.com/2011/10/testing-tools-
comparison_27.html. [Accessed 25 March 2013].
13. ProgrammingZ.com. 2013. Open-Source Quality Assurance And Performance
Analysis Tools . [ONLINE] Available at:
http://www.programmingz.com/b/2435541/Open-Source-Quality-Assurance-And-
Performance-Analysis-Tools/#.UU-rkaAtg3I. [Accessed 25 March 2013].
14. Open Source Testing. 2013. Open source test management tools. [ONLINE]
Available at: http://www.opensourcetesting.org/testmgt.php. [Accessed 25 March 2013].
15. PHP Unit Manual. 2013. Chapter 8. Database Testing . [ONLINE] Available at:
http://www.phpunit.de/manual/3.6/en/database.html. [Accessed 25 March 2013].
16. Mobile QA Zone. 2013. Visual Test Automation Leads the Future of the Software
Testing Industry - Mobile QA Zone. [ONLINE] Available at:
http://www.mobileqazone.com/profiles/blogs/visual-test-automation-leads-the-future-of-
the-software-testing. [Accessed 25 March 2013].
17. Software QA and Testing Resource Center. 2013. Database Testing Tools. [ONLINE]
Available at: http://sqa.fyicenter.com/FAQ/Testing-Tools/Database_Testing_Tools.html.
[Accessed 25 March 2013].
18. Ryan Schuetzler. 2013. QA Tool Review: JMeter. [ONLINE] Available at:
http://www.schuetzler.net/2010/01/02/jmeter/. [Accessed 25 March 2013].
19. Wikipedia, the free encyclopedia. 2013. Comparison of issue-tracking systems -
Wikipedia, the free encyclopedia. [ONLINE] Available at:
http://en.wikipedia.org/wiki/Comparison_of_issue-tracking_systems. [Accessed 25
March 2013].
20. Jon's Technical Notes. 2013. Jon's Technical Notes: Pros and Cons of Selenium.
[ONLINE] Available at: http://jdrew33.blogspot.com/2012/02/pros-and-cons-of-
selenium.html. [Accessed 25 March 2013].