Transcription
Software Test Automation Practices in Agile Development Environment: AnIndustry Experience ReportEliane Figueiredo CollinsVicente Ferreira de Lucena Jr.Nokia Institute of TechnologyManaus, AM, [email protected] University of AmazonasManaus, AM, [email protected] customer rather than following rigid plans and contractnegotiations. The Scrum, as agile methods, is an importantpractice in software development, with characteristics ofrelatively flexible organizational system, timely andinteractive feedback, objective-oriented flat management andeffective participation of members [4]. It can solve thedifficulties faced by traditional open methods, some to alarge extent and it can also be used to manage and controlsoftware and product development.Agile processes are iterative, so the test activities have tofollow the iterations which need to be executed fast andefficiently leading to the use of automated testing tools,quick responses to the requirement changes and interact withthe development team [5].Crispin and Gregory [6] identified the automationproblems in projects and according to their experiences inagile teams, the issues found that influence the success ofTest Automation are: Programmers’ attitude, some programmersdon’t bother to test much because they have theQA team as a safety net to catch bugs; The “Hump of Pain”, means the learningcurve for testers to learn about tools and code; Initial investment; Code that’s always in flux, interface changesturn the GUI automation not viable; Legacy systems, legacy code isn’t designed fortestability; Fear, testers that don’t have programingbackground and programmers that don’t haveexperience in testing; Old habits, manual regression tests instead ofautomating the regression suite;This paper presents experiences and results of theimplementation of Test Automation using open sourcetesting tools in projects which used the agile developmentprocess Scrum. For confidentiality reasons, the softwareprojects described in this paper will be called Project 1 andProject 2 respectively.In section 2, the agile development with Scrum and abrief on Agile Testing is explained. Section 3 shows detailsof case studies, the projects characteristics, testing processand common problems. Section 4 shows the results obtainedwith the experience collected and the lessons learned.Finally, section 5 presents the conclusions and further steps.Abstract— The increased importance of Test Automationin software engineering is very evident considering the numberof companies investing in automated testing tools nowadays,with the main aim of preventing defects during thedevelopment process. Test Automation is considered anessential activity for agile methodologies being the key to speedup the quality assurance process. This paper presentsempirical observations and the challenges of a test team new toagile practices and Test Automation using open source testingtools integrated in software projects that use the Scrummethodology. The results obtained showed some importantissues to be discussed and the Test Automation practicescollected based on the experiences and lessons learned.Keywords-test automation; agile testing; testing tools;software testingI.INTRODUCTIONIn recent years, Software Engineering has been adaptingto the evolutions occurring in the global softwaredevelopment scenario in order to alleviate high cost, highcomplexity, market time, maintenance difficulties and enduser satisfaction. In this context, two evolutions could behighlighted and they are the focus of this paper: the TestAutomation and the Agile Methods.Depending on the system phase, software testing can beexecuted manually or automatically. According to Dustin etal. [1] the definition for Software Test Automation means toautomate software testing activities including thedevelopment and execution of test scripts, verification oftesting requirements, and the use of automated testing tools.There are many reasons why software tests should beautomated. When compared, manual testing is timeconsuming while Test Automation increases efficiency forrepetitive steps to reproduce system functionality, especiallyin regression testing, where test cases are executediteratively and sometimes incrementally after makingchanges to the software [2]. Automate all test activitiesduring development phase can take more than 50% of theoverall project effort [3].The agile software development process stands outmainly because of its ability to rapidly accommodatechanges in the original requirements and prioritize thedevelopment of functionalities through executable codesinstead of extensively written documentation. It alsoprovides quick responses to changes and collaboration with978-1-4673-1822-8/12/ 31.00 c 2012 IEEE57AST 2012, Zurich, Switzerland
II.BACKGROUNDguide the testing in line with agile values working softwarewhich responding to change. According to the authors, for asuccessful agile testing approach the following are the keyfactors: Look at the Big Picture; Collaborate with the Customer; Build the Foundation for Agile Core Practices; Provide and Obtain Feedback; Automate Regression Testing; Adopt an Agile Testing Mindset; and Use the Whole Team Approach.Test Automation is considered the key of successfullyagile development and the core of agile testing. Agreeingthat [6], different tests have different purposes, they show adiagram of testing quadrants describing how each quadrantreflects different reasons to test, as presented in Figure 2.A. Agile Development ScrumThe Scrum is illustrated in Figure 1, the functionalitiesare to be implemented through a list called Product Backlog.Each sprint has two week duration (iteration). There is aSprint Planning Meeting where items on this list areprioritized by the customer representative (Product Owner).The team defines the features which may be accomplishedwithin the iteration. This list planned for the next iteration iscalled Sprint Backlog [3]. During the Sprint, the activitiesare broken down into tasks that are observed by the teamthrough a daily meeting which lasts a maximum of fifteenminutes where tasks impediments are identified.Figure 2. Agile Test Quadrants[6]Figure 1- Scrum methodology(Q2) Business Facing tests in Support ofProgramming (Business requirements testing –Does the code do what it should?) (Q3) Business Facing tests to Criticize the Product(Business defect testing – Does the code dosomething it shouldn’t? Are there missingrequirements?) (Q1) Technology Facing tests in Support ofProgramming (Technical requirement testing – Doesthis method do what the developer intended?) (Q4) Technology Facing tests to Criticize theProduct (Technical defect testing – Are there leaks?Can it handle a load? Is it fast enough?)Traditional software testing focuses almost exclusivelyon the right side, the functional tests are made to criticizethe product but not playing a productive part in supportingthe creation of the product. Moreover, it is involved late inthe development to detect failures but not to prevent them. The Scrum methodology has three key roles to run theprocess [7]: Product Owner (PO): This role is responsible forcommunicating the product vision to the scrumteam and must also represent the customer intereststhrough requirements, business rules and priorities; Scrum Master (SM): Link between the team andPO, but does not manage the team. Scrum masterworks to remove the obstacles that are blocking theteam tasks from achieving the sprint aim; Team Member: Team liable for completing thework, cross-functional and empowered to makeplan, effort estimates and take technical choices.B. Agile TestingAs Crispin and Gregory claim [6], agile testing does notjust mean testing on agile projects but testing an applicationwith a plan to learn about it, let the customer information58
In agile testing, the team is not only involved inidentifying, but also in preventing failures. Thus, agiletesting is a challenge for testers that used to work intraditional waterfall projects mainly because they do nothave to wait for system delivery at the end of the project tostart the test activities but they need to be proactive and startthe test tasks from the very beginning of the project togetherwith the developers as one team while bearing in mind thatsuccess in this challenge requires experience, overcomingbarriers and mind set.III.acceptance criteria, and use automated test tools to speed upthe execution activities of each sprint. The main toolsrecommended by the test team to be used to automate testtasks in the projects were: TestLink1: used to manage test plans, write test cases,and report test executions. This tool acted as an editorand organizer of test cases, storing all informationallowing updating and creating test builds easily. Itfacilitated the documentation of reports, and also thecontrolling of the test execution versions [8]; Mantis Bug Tracker2: through this tool, which wasalready used in the institute, the tester registered thedefects found, sent them to the developers andcontrolled the lifecycle of each defect. The friendlyinterface facilitated communication among teams. Subversion3: it was used to share and manage thesystem code, documentation and information amongthe project team’s members. Jmeter4: Open Source Tool for web performance andstress testing generating result reports.The tasks of test processing during a sprint wereincremental and iterative, according to the agile testing, theybegan with the development and did not wait for systemdelivery. The Test team was new to scrum and agile testing.They had experiences testing waterfall projects. They may beproactive doing verification tasks for the period of thedevelopment using automated tools. To avoid the Cascadeeffect, the sprint could not be delivered for tests at the end ofsprint. The Test Process flow is represented in Figure 3.ENVIRONMENTA. Software Projects ChacracteristicsDuring a period of one and a half year two softwareprojects had been conducted, all using Scrum as thedevelopment methodology for management. In the context ofa research institute in Manaus/Amazonas/Brazil they servedas study cases to apply Software Test Automation and collectthe practices and lessons learned.First, in Project 1, there was focus on usability. Thecustomer was able to use a web dynamic interface easily toregister campaigns, users and ads attaching images, videosand texts without difficulty and with good performance. Thisproject had a development team composed of one ScrumMaster and 3 full-time developers.The test team had one Test Leader half-time and twoTesters full time. Project 1 was web platform developedusing PHP programming language, MySQL database,Apache server and IDE Eclipse. The system was simple andcomposed of 8 screens (4 forms to register campaigns andusers and 4 screens to search and generate reports).Project 2 was a web project to attend the factory. Itautomates tasks according to procedures to control materialsand production costs. The main goal was the usability ofinterface for specific users, performance of system responsesand the accuracy of the data to be manipulated during thesystem calculation.This system had high complex rules in two webinterfaces: one counter user through mobile browser and anadministrator user through computer browser. Java languagewas used for development environment, JBoss server, Oracledatabase and Eclipse IDE. The scrum team for Project 2 wascomposed of 4 developers, one Scrum Master, one TestLeader half-time and two Testers full time.These projects initially followed the same test process,but at specific times during their execution, the TestAutomation structure must be modified to improve the testcoverage and the goal of the projects.B. Testing Process and Tools for the Scrum ProjectsThe Test team in accordance with agile principles wasintegrated to the project team. There wasn’t any formalseparation between the Testers and the Developers. TheTesters were participating in Scrum ceremonies (sprintreview, daily, retrospective and planning meetings). Theirresponsibilities basically was to plan test cases through thestories described in the Product Backlog, specify theFigure 3. Testing Process1TestLink URL: http://www.teamst.org/Mantis URL: http://www.mantisbt.org/3Subversion URL: http://subversion.tigris.org/4Jmeter URL: http://jmeter.apache.org/259
The dark rectangles represent tasks to be done by theTest Leader and the lighter rectangles were tasks to be doneby the Testers. The Test Leader was also responsible formonitoring all tasks in the process. The task IncrementalTest Execution included execution of manual exploratorytests and automated regression tests.For web projects 1 and 2 described, developers used forunit testing the frameworks: PHPUnit5 and JUnit6respectively. The tool of choice for functional GUI testautomation was Selenium RC and IDE7.Selenium uses the approach record-and-play supportweb applications for the browser Mozilla Firefox. SeleniumIDE8 can record the user actions in the browser, creating testscripts in several programming languages and executingthem later [9]. The Selenium Java API used in the projectallows running the test scripts in other browsers such as theversions of Internet Explorer.IV.In consequence, the greater part of the code needed to bemodified for the next sprint causing technical debt.Moreover, the task to automate functional GUI tests inthe current story of sprint presented problems because theGUI interface of system wasn’t stable enough, Testers hadno strong java programing background to improve theautomated suite, they were trying to automate using therecord-and-play approach for all test cases planned,including for interface issues, but the system interfacechanged during the sprint and the automated test scripts hadto be recorded and written again. The lack of time in thesprint to finish the automation tests and non-synchronizationtasks with development resulted in back to manualexecution of regression tests. Testers had technical debt too.After identifying problems, the project team startedrefining the strategy and changing the automation activitiesto the second stage of the project (Sprints 6 to 9). The Unittesting performed had to be reviewed by Testers and achieve100% coverage of business rules. It was an item for storyacceptance criteria in the backlog stories.To alleviate automation test problems, Testers decidedto perform the automation with the support of Developers,automatizing just the acceptance tests of past sprint whichinterface was considered stable. The exploratory manualtests should be run for current sprint stories. Developerswere helping with test activities programing the automatedtest suite. The team was working together supporting eachother in test activities and programing.Other improvements were implemented at the end of theproject. The automated regression test suite with Seleniumwas integrated with TestLink using the Testlink API Client.This API allows the automated test suite in java to accessthe Testlink and register the Selenium execution resultsautomatically. Testlink was integrared with Mantis also,allowing traceability of defects and test cases.Thus, when Testers were executing regression testsautomatically, results were registered in Testlink, facilitatingthe generation of test reports, providing quick feedback oftests and spending the time more efficiently to run manualexploratory tests to find usability and performance defects.STUDY CASES APPLYING TEST AUTOMATIONDifferent Test Automation practices were used duringeach project to achieve their goal. The scrum iterationsallow the project team to use different approaches until itfinds the right way to work. So, when some tool or methoddidn’t work well in the sprint, the team can try somethingdifferent in the next iteration to improve the process.Practices adopted for test automation and the commonproblems found for each project are outlined below.A. Project 1During the first stage of Project 1 (Sprints 1 to 3), theteam decided that developers should perform stories of newfunctionalities and unit testing using PHPUnit throughoutthe development process. Continuous integration (CI)process for infrastructure reasons and developers’ attitudecould not be implemented. Developers were focused on newfeatures and Testers should automatize functional teststhrough GUI for regression tests using Selenium RC andexplore the system through manual tests. The Stress testsshould be realized after all the functionalities wereimplemented at the end of the project. To apply agile testautomation in an environment without continuousintegration was a big challenge.This was the first attempt by the team to automate unittests and functional tests in an agile project. Internally theproject team was divided into Developers and Testers. TheDevelopers weren’t performing effective unit tests becausethey were focused on new features and confident that theTesters will find all the system defects. So, many codedefects as null pointer exceptions, screen crashes and illegalexceptions were found during functional system tests.Figure 4.5PHPUnit URL: www.phpunit.de6JUnit URL: www.junit.org7Selenium URL:www.seleniumhq.org60Project 1 Automation Test Coverage
The reliable system version could be available to customersat any time in the production environment.Figure 4 shows Selenium GUI test automation coverageof functional test cases. Initially Testers tried to automate alltest cases planned for the sprint and web interface achieving87% of coverage. During the sprints with interface changesand new test approach the automation cover decreased by50%. This happened because the team automated just theacceptance tests for stable interface. Compared to Figure 5, itis possible to realize that defects found along the sprintsincreased until down to the end of project. It proves that highautomation coverage for GUI didn’t mean finding manydefects, because Testers were very busy trying to automate alot of GUI unstable tests without value for the project. WhenTesters decided to automate just what was necessary forfeature acceptance, the automation coverage decreased, butwith time for exploratory tests, they found more defects anddid tests that added value to the project.Figure 6. Automation Structure with CIFigure 6 represents the automation structure using CI.Testers and Developers committed to sending the daily code,unit tests and automated acceptance tests (Selenium andJmeter) through Subversion. Hudson monitored theSubversion changes and started the CI process. Developerscould easily run acceptance tests ensuring a functionalsystem and Testers could execute manual exploratory tests,security tests and usability tests on Test environment.Test problems were found during sprints (sprint 5 to 10).The customer realized that interface, performance andusability were working well, but the data accuracy wasn’tright. This happened because the tests were performed forthe code layer and GUI layer, but functional tests forcomplex business roles may be executed for the layer behindthe GUI.Note that for this project the base of automation was unittests and acceptance tests, but to ensure data accuracy forcomplex business rules, it wasn’t enough, so integration testshad to be done to validate it. To solve this problem, Testersand Developers implemented tests using Fitnesse TestTool10.This tool can be used as wiki for input data andfixtures implemented in java. It was also integrated withHudson, team members easily implemented tests stubbingdatabase data, the test execution were behind GUIidentifying the integration problems earlier.Quick feedback in this project were observed becauseHudson displays real time reports of the tests and an e-mailwas sent to the project team to do the necessary correctionsin the project. The CI environment made it possible to takeautomation testing to another level of efficiency andendorsed the team’s integration.Besides, the regression tests suite of Selenium tests werelarge and were run in order of creation of every sprint, aftersometime it was observed that the first tests didn’t revealdefects. This happened because the first features werestressed with test cases several times, so they were stable.Figure 5. Project 1 Defects per SprintStress Tests using Jmeter tool and security tests asplanned were executed at the end of the project (Sprints 8and 9). The problems identified during this phase impactedthe entire system and many parts of the code and even testscripts had to be reworked. The functionalities and GUI wereaccording to backlog stories and acceptance criteria but thenon-functional system requirements and vulnerability testshad failed. So, non-scheduled tasks of reworking appearedclose to delivery of the project.B. Project 2The structure of Project 2 had some improvementscompared to Project 1. Continuous Integration (CI)environment was configured in the first sprints (1 to 3) usingthe open source tool Hudson 9. Automatically the CI shouldbe able to compile the code, execute unit tests, accept GUItests (Selenium) and deploy the versions of the system to theservers: Development, Test and Production.Due to the experience of Project 1, Testers hadadopted the strategy of startup system basic load tests andvulnerability tests as soon as possible using Jmeter testtool integrated in the CI environment. Thereby, it ensuredthat each release generated for Project 2 was covered by unittests, GUI acceptance tests, basic load tests and security testsaccording to the customer requirements specification in orderto minimize risks of problems and rework in the system atthe end of project as occurred in the previous experiment.910Hudson URL: http://hudson-ci.org/61Fitnesse URL: http://fitnesse.org/
The tools to be chosen in the agile environment should beflexible to be adequate in the project to facilitate the testprocess, scrum iterations, incremental addition of test scripts,foster collaboration, while supporting real programinglanguage allowing customizations [10].But the tests for new features reveal defects caused by codechanges. So, in conclusion, prioritizing the regression test fora large test suite reveals defects earlier.Project 2 had 20 sprints, were the automation for unittests, integration tests, non-functional tests (performance andsecurity) and GUI acceptance tests covered respectively 70%of code, 100% of business rules, 100% of non-functionalrequirements and 40% of GUI acceptance tests. The defectsfound per sprint before integration tests varied between 45and 48, after integration tests the defects found per sprintdecreased for 29.C. Automate Each Layer of Software When It Is Possibleand Just for Acceptance Tests in the SprintAs was observed in Project 1, when the team spent toomuch time on first sprints trying to automate all functionalGUI possible tests, they wasted time that should have beenspent on tests that could have added value to the project.It was noted in Project 2 that even after doing effectiveunit testing, good CI environment and GUI test automation,the business rules behind the interface couldn’t have beenignored because they detected problems of integration anddata accuracy important to the security of the system.V. RESULTS AND LESSONS LEARNEDIn this experience on conducting Test Automation inagile software projects, we could observe it can work verywell when the project team has the patience and persistenceto find the right way to work. However, some issues need tobe managed to avoid the risks introduced by the difficulty ofapply agile practices in an inexperienced project team. Thus,we have identified some key lessons that we have learned,minimizing the impact of automation risks:D. Test Automation in the Agile Project Should Be SimpleTest Automation cannot be a subject just for Testers. Allteam members should be able to use the automated test tools,run, write tests and see reports quickly. Concentrate theknowledge of tools in just one or two people, and adoptionof complex automation tools, represent risks for agileprojects and separate the team. The combination of skills tofinish a task achieves good and quick results.In Project 2 for example, the simplicity of Fitnesse toolallowed developers and testers to contribute together to thecreation and execution of integration tests.A. Collaboration is an Essential Factor for the Success ofTest Automation in Agile ProjectsAll main research works in the agile methods fieldindicate that team collaboration is an important aspect toachieve success in an agile project, and we could alsoconfirm these claims for Test Automation. Some practiceswere essential to reach this success: Involve Testers in backlog review, design interfaces,configuration environment and architecture systemdiscussions are very important to avoid separationbetween developers and testers. Maintaining theunity of the project team encourages cooperation. Ask for help when a task requires knowledge fromanother area. For example, in Project 1 Developershelped Testers to improve the Selenium automation testsuite and Testers supported Developers reviewing unittesting to achieve test coverage. Test Automation environment is not a responsibilityjust for Testers. All Team members have responsibilityin setting a test automation environment. Pairprogramming can be a good exercise to integrate theproject team. Pairs of Tester and Developers can beuseful for some tasks like setting environments, codeinspection, review and program scripts.E. Prioritize the Regression Test ExecutionAlong the sprints of Project 2, the first automated testscripts executed many times didn’t reveal defects because thefunctionality was stable, so they didn’t need to be executedfirst. It is important run tests that have more probability toreveal faults speeding up the defect detection.F. Automate and Run Security Tests and Stress Tests EarlyReduces Risks and ReworkNote in Project 1, security and stress tests were executedat the end of the project, it caused extra tasks on the systemand delay of project delivery chronogram. The defects couldbe detected before when intermediate versions were releasedavoiding risks and rework in the project.Spending time to do this kind of tests at the beginning ofthe project aggregate value, knowledge and confidence forthe agile team.B. Fit Testing Tools with Test Strategy and Agile MethodChoosing the right tools to be used to automate tests isvery important. Sophisticated tools are not always the bestchoices for Agile Test Automation and the project team hasto know the limitations of free test tools well and considerthe learning curve and time spent on configuring the testenvironment.It is important to keep in mind that if one tool ortechnique is not working or adding value to the project, inthe next iteration, the project team can starttryingsomething new or fit the tools with the project needs ashappened in Project 1.G. Use Test Automation for Documentation andInformation FeedbackTest Tools are also highlighted in the technical literatureas an important element for documenting software artifacts.In the context of these agile projects, some special toolsshould be cited: A test management tool controlled all informationregarding the software testing activities, test casesspecification, creation of new releases, testenvironment, test running, number of detected failures,and bug tracking (creation, correction, and validation).62
In our software projects we used the TestLink andMantis test tools. The integration of Selenium andTestlink allows the automatic execution resultsfeedback facilitating test reports. The Hudson CI has an interface available to all teamdisplaying release test status providing executionhistory and reports of all kinds of integrated tests. Itgave to Project 2 the general vision and metrics onhow the release status was and what problems wereoccurring in time to correct them. In Project 1, withoutCI, the quick feedback to team about the system statusand test execution wasn’t possible.ACKNOWLEDGMENTThe authors thank to Federal University of Amazonasand Nokia Institute of Technology for its contribution to thiswork.REFERENCES[1]E. Dustin, J. Rashka, and J. Paul, Automated software testing:introduction, management, and performance. Addison-Wesley,2008.J.[2] Karhu, K.; Repo, T.; Taipale, O.; Smolander, K.; , "EmpiricalObservations on Software Test Automation," Software TestingVerification and Validation, 2009. ICST '09. International Conferenceon ,vol.,no.,pp.201-209,1-4April2009doi: 10.1109/ICST.2009.1[3] E. Kit, Software Testing in the Real World: Improving the Process.Reading, MA: Addison-Wesley, 1995.J.[4] Hu Zhi-gen; Yuan Quan; Zhang Xi; , "Research on Agile ProjectManagement with Scrum Method," Services Science, Managementand Engineering, 2009. SSME '09. IITA International Conference on ,vol.,no.,pp.26-29,11-12July2009doi: 10.1109/SSME.2009.136[5] Collins Eliane, Lucena Vicente. 2010. Iterative Software TestingProcess for Scrum and Waterfall Projects with Open Source TestingTools Experience. In Proceedings of the 22nd IFIP InternationalConference on Testing Software and Systems2010( ICTSS’10).CRIM, 2010. 115-120 p. [ISBN : 978-2-89522-136-4] .[6] Crispin, L.; Gregory, J.; Agile Testing: A Practical Guide for Testersand Agile Teams, Addison-Wesley, 2009, ISBN 0-321-53446-8.[7] Beck, Kent; et al. (2001). "Manifesto for Agile SoftwareDevelopment". Agile Alliance. Retrieved February 2012.[8] Siebra, Clauirton A; Lino, Nancy L; Silva, Fabio Q B; Santos, AndreL M; , "On the specification of a test management solution forevaluation of handsets network operations," Automation Quality andTesting Robotics (AQTR), 2010 IEEE International Conference on ,vol.2,no.,pp.1-6,28-30May2010.DOI 10.1109/AQTR.2010.5520833.[9] Antawan Holmes and Marc Kellogg. 2006. Automating FunctionalTests Using Selenium. In Proceedings
B. Agile Testing As Crispin and Gregory claim [6], agile testing does not just mean testing on agile projects but testing an application with a plan to learn about it, let the customer information guide the testing in line with agile values working software wh
