Automated System Testing by Steven Erat
•Download as KEY, PDF•
1 like•5,106 views
Learn how to build a robust, modular, automated test suite for your website's UI that models real users traversing the application. Built with the free ColdFusion tools CFSelenium and MXUnit, tests can pull in parameterized input data from external sources to make them more dynamic and flexible. Using the Page Object Design Pattern, test suites can be made more maintainable and less brittle while allowing for faster test development by abstracting how a user interacts with a website, thus minimizing the effort to fix tests when the website UI changes. Pull it all together with ANT and Jenkins for hands free, reliable test suite execution and test result notification on a daily schedule.
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Recently uploaded
Recently uploaded (20)
Automated System Testing by Steven Erat
- 1. Automated System Testing For Web Applications Using CFSelenium, MXUnit, & Jenkins CI Steven Erat FirstComp Insurance talkingtree@gmail.com CF.Objective() 2012
- 3. There’s no handout Downloads at talkingtree.com/cfo PDF of Slides The Full Demo Test Suite Readme for Instructions
- 4. About you
- 5. What to Test? Functional Testing Unit, Integration, System, Regression, Acceptance does a specific feature work? can the user do {this}? Non Functional Testing Security, Performance, Usability cuts across features
- 6. Unit Testing A type of Functional Testing Written & run by software developers Finds problems early in Development Tests components in isolation Dependencies usually mocked Can’t test legacy (non-CFC) code
- 7. System Testing A type of Functional Testing Tests the completed software product Often performed manually Can be automated with testing tools: QTP - The $6000 Gorilla Selenium - FOSS; Has many language APIs
- 8. Continuous Integration Continuous process of quality control Development - Creates “The Build” QA - Test Suite Execution Performed Frequently after each commit, or scheduled regularly (daily, hourly, etc) CI Software: Jenkins, Bamboo, CruiseControl
- 9. How to Test? System Testing mimics user interaction with GUI QTP - Expensive, complex, scriptable (VB) TestNG Java Test Framework with Selenium Free. Write tests in Java. MXUnit ColdFusion Framework with CFSelenium
- 10. The Landscape Test Suite is independent of AUT AUT 1 Test Suite AUT 2 Framework AUT 3 ColdFusion MXUnit CFSelenium Application(s) Under Test
- 11. What is Selenium? Selenium IDE record/playback user actions in a browser recorded in Selenese, a special test scripting language Selenium Client API alternative to Selenese, tests can also be written in various programming languages Selenium (RC) Server Java-based server that accepts commands for browser
- 12. Selenium IDE Firefox plugin Record in HTML, Java, Python, etc Playback in browser
- 13. What is CFSelenium? Extends the Selenium IDE Record as CFML in MXUnit TestCase
- 14. CFSelenium component extends="cfselenium.CFSeleniumTestCase" { public void function beforeTests() { Static input values browserUrl = "http://localhost:8500/"; } like ‘username’, public void function testSimple_blog_admin_login() { ‘password’ selenium.open("/blog/admin/index.cfm"); assertEquals("BlogCFC Administrator", selenium.getTitle()); Linear & non- selenium.click("css=input[type="submit"]"); selenium.waitForPageToLoad("30000"); iterative assertTrue(selenium.isTextPresent("Welcome")); assertEquals("Logout", selenium.getText("link=Logout")); Object identification selenium.click("link=Logout"); selenium.waitForPageToLoad("30000"); is mixed with assertEquals("BlogCFC Administrator: Logon", assertions selenium.getTitle()); } }
- 16. Selenese Commands Actions open a URL, click a link, wait for page to load Accessors store results of actions in a variable: storeTitle Assertions assert (failure stops further test method) verify (test method continues after failure)
- 17. Selenese HTML Format <tr> <td>open</td> <td>/blog/admin/index.cfm?</td> <td></td> </tr> <tr> Selenium <td>assertTitle</td> <td>BlogCFC Administrator: Logon</td> performs the <td></td> </tr> <tr> Assertion <td>click</td> <td>css=input[type=‘submit’;]</td> during playback <td></td> </tr> <tr> <td>waitForPageToLoad</td> <td>30000</td> <td></td> </tr>
- 18. Selenese CFML Format selenium.open("/blog/admin/index.cfm?"); assertEquals("BlogCFC Administrator: Logon", selenium.getTitle()); selenium.click("css=input[type='submit']"); selenium.waitForPageToLoad("30000"); MXUnit performs the Assertion during playback Verify() not supported by MXUnit, use Assertions
- 19. Object Locators Strategies for locating objects/elements on pageselenium.isElementPresent('id=username') selenium.isElementPresent('name=username') id selenium.select(‘document.forms['myForm'].cb1 name ’) dom selenium.click(“//a[contains(@href,'#id1')]”) xpath selenium.click(‘link=Add Comment”) link selenium.click('css=input[type="submit"]') css selenium.getText(‘regexpi:^[A-Z0-9+_.-] +@[A-Z0-9.-]+$’)
- 20. What is MXUnit? CFML-based Unit Test Framework ColdFusion Builder & Eclipse plugin AntTask
- 21. Anatomy of a TestCase beforeTests() afterTests() setup() tearDown() test functions
- 22. MXUnit Dataproviders Iterate over test methods using native CF data structures Test multiple logins with list of usernames &
- 23. MXUnit Plugin Test Runner Plugin for CFBuilder Run whole Test Suites Run single TestCases
- 24. What are the benefits so far? Quick testcase creation in familiar language Framework for test execution & reporting System Regression Testing
- 25. What are the problems? Object locators hard coded in each testcase Object locators repeated thru testcase/ suite Assertions interwoven with Selenium commands Static tests embedded with literal input
- 26. Solutions: POM The Page Object Model design pattern Models web page or page part independently navigation bar, sidebar, pods, footer Object identifiers exist in only one place Object oriented approach to testcase creation
- 27. Solutions: POM Dante Briones Must watch
- 28. What to Model? Banner NavBar Search Blog Entry Comments Add Comment Pods
- 29. Modeling a Part / Screen Add Comment pop-up One function for each form field, button, or widget Functions isolate the object locator Method chaining
- 30. Before using POM
- 31. Using Page Object Model
- 32. Method Chaining POM methods named intuitively Improved readability Faster to type with Code Insight **
- 33. The Landscape Logical divisions in AUT can be tested Test Suite Framework Applications Under Test
- 34. TestCase« » Page Test doesn’t know about object locators Page Object doesn’t know about Asserts Page can return instance of next page: loginAs returns Home
- 35. Page Inheritance HomePage, SearchPage, and ContactPage extend BaseNavItem
- 36. Parameterized Test Data External data read for use as MXUnit dataproviders Easy to modify data by non-technical users JExcel let’s you easily use multiple tables
- 37. Workflow Record user interaction in a browser Performed by QA or Business User Use Selenium IDE with CFML Format Produces linear recordings with static data Enhance static scripts Performed by QA Software Engineer Convert to Page Object Model Convert for Parameterized Data (Dataproviders) Commit to Source Control
- 38. Recording User Interaction Selenium IDE records object interactions Can change priority of which locators it uses id, name, link, XPath, DOM, etc During recording you may add Assertions manually
- 41. Running MXUnit via Ant
- 42. Ant Build Files Have it your way
- 43. Ant Properties Files Can use a global properties file shared aspects across entire test framework Can also use local properties files store config values unique to each AUT test suite Flexible configuration - It’s up to you
- 44. Global Properties Define a property to switch testing environments Dev or Staging? Tell Selenium which browser to launch
- 45. Local Properties Properties unique to each UAT being tested Define properties to initialize Selenium - browserURL $ENV swapped to match global env property (Dev or Staging)
- 46. MXUnit Entry Points MXUnit kicks off Test Suite via a Remote Method call Plugin test runner calls RemoteProxy.cfc Ant test runner calls HttpAntRunner.cfc
- 47. How are Tests Run So Far? Manually Locally per Individual Execute Plugin or Ant Test Runner
- 48. How to Improve? Automation Central Test Environment Reporting Notifications
- 50. Jenkins CI
- 51. Jenkins CI Create ‘jobs’ to do work Call Ant Tasks Run Source Control Schedule jobs using cron syntax Chain jobs to run in sequence Status / Trends / History Exceptions Stack Traces Email Notifications
- 52. Jobs Jobs for discrete Test Suites Status and Trend icons Last Run and Duration
- 53. Meta Jobs Jobs that trigger other jobs Jobs that restart servers Jobs that sync source control
- 54. Job Trends
- 55. TestCase History
- 58. Demo?
- 59. Lessons Learned Architect a meta framework to support multiple test suites with better code reuse
- 60. Lessons Learned Stub out Page Objects up front Organize test suite Page and Test directory structure by logical areas in the AUT Blog Home, Blog Entry, Comment, Contact, Search, RSS ... Blog Admin Login, Home, Edit Entry, Refresh, Stats ... Test writers can then work in parallel without duplication
- 61. Lessons Learned Convince AUT Developers Testing Works Develop AUTs with better object id/ naming unique IDs better than CSS, DOM, XPath locators Disambiguate significant data in AUT body
- 62. Lessons Learned Design Page Objects for method chaining LoginPage.enterUserID(‘me’).enterPass(‘pw’).submit(); Easier to read Takes advantage of code insight CF Builder’s code insight not quite there yet Other IDEs may provide better code insight (IntelliJ)
- 63. Lessons Learned Always perform LogOut() in beforeTests() method Run Login() in the setUp() method Run LogOut() in the tearDown() method Tests no longer need to perform login actions (you should already have tests just for Login)
- 64. Snippets Test Cases, Page Objects, Debugging
- 65. Troubleshooting Response of the Selenium RC is invalid: ERROR: Element css=p > input[name="search"] not found Response of the Selenium RC is invalid: Current window or frame is closed! Get the AUT’s HTML Source via Selenium Have Selenium grab a screenshot of AUT
- 66. Troubleshooting Response of the Selenium RC is invalid: ERROR: Element //input[name="search"] not found
- 67. Troubleshooting Use custom messages for Assertions Make them more meaningful Save more hair
- 68. Troubleshooting Jenkins will show your custom message for failures
- 69. Troubleshooting Slow the test execution down Rerun the test via MXUnit Plugin Watch the browser as test runs Add sleep(NNNN) statements at key points
- 70. Troubleshooting // Runs once before all test methods in file public void function beforeTests() { // dataproviders must be in the variables scope variables.TwoCommenters = application.dataproviders['TwoCommenters']; variables.selenium = application.selenium; variables.HomePage = new HomePage(selenium); variables.defaultSpeed = selenium.getSpeed(); selenium.setSpeed(1000); } // Runs once after all test methods in file public void function afterTests() { HomePage.clickNavButton_Home(); assertEquals(HomePage.getExpectedTitle(), selenium.getTitle()); selenium.setSpeed(variables.defaultSpeed); }
- 71. Troubleshooting Use Tasks view with keywords //TODO or // FIXME
- 72. Troubleshooting A TestCase method failed Did the TestCase change? Did a Page Object change? Check TestSuite Source Control for changes since last success Did the AUT change?
- 73. Troubleshooting A Job in Jenkins Failed to Build Check the Test Suite’s ColdFusion exception.log
- 74. Thank you! Steven Erat talkingtree@gmail.com Get Slides and Demo TestSuite code at: talkingtree.com/cfo
Editor's Notes
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n