Transcription
Continuous Performance TestingShopware Developer ConferenceKore Nordmann (@koredn)08. June 2013
About MeKore Nordmann@koredn
Co-founder ofHelping people to create high quality web applications.http://qafoo.comIExpert consultingIIndividual trainingGet a training on object oriented design for your team!
OutlineMotivationConclusion
MotivationIWhy should we do performance tests?IIILocate unknown bottlenecksMesaure behaviour of the full stackWhy should we do that continuously?IIFind performance regressionsEnsure optimizations are persistent
Often used toolsIOften misused toolsIIITesting for micro-optimizationsIIsiegeApacheBench (ab)Evaluating Hello-World-examples of FrameworksUseful tools, I won’t talk aboutIIIIxDebug ProfilingxhProfDatabase profiling toolsSystem metrics (Graphite, . . . )
The problem is more complexIYour task: Create a new webshopIIAssume it’s marchThe deadline is October this year, right before Christmas
A webshopAwesome Shop0 articles0.00 SmartphoneLorem ipsum dolor sit amet,consectetur adipisicing elit,sed do eiusmod temporincididunt ut labore et doloremagna aliqua.1337,-- 5 items in stockCommentsLorem ipsum dolor sit amet, consecteturadipisicing elit, sed do eiusmod tempor.
A real tool for performance testsIJMeterIIIIComplex user pathsConcurrend requestsRecord on proxyClustering
JMeter 101IThread GroupIIControllerIIConfigration, optionally from external sourcesTimerIIControls how samplers are executed (loop, random, . . . )Config ElementIIAn execution plan (like a user registration)Defining timing constraints for sampler executionSamplerIPerform the actual work (like HTTP requests)
Getting startedICreate a test planIIWhat do users actually do on your site?Example:IIIIIRandom browserUser registrationSign onShopping with checkoutCommenting products
AutomationIIAnt JMeter integrationAutomation of your environmentIIIIISetting up different software versionsTesting with different extensionsRunning different database setupsHandling multi-node environmentsPlain Ant or Puppet, Chef, Vagrant
Apache Ant 0221222223 t a r g e t name ” b u i l d apc bytecode cache user cache f i l e ”depends ” setup apc bytecode f i l e cache , b u i l d ”d e s c r i p t i o n ” Run w i t h APC opcode cache and f i l e based cache . ” / t a r g e t name ” b u i l d apc bytecode cache user cache apc ”depends ” setup apc bytecode apc cache , b u i l d ”d e s c r i p t i o n ” Run w i t h APC opcode cache and APC based cache . ” / t a r g e t name ” remote exec p a r a l l e l ” subant t a r g e t ” { t a r g e t } ” i n h e r i t a l l ” t r u e ” f i l e s e t d i r ” { p r o j e c t . d i r } ” i n c l u d e s ” s e r v e r . xml ” / / subant / t a r g e t t a r g e t name ” remote exec ” sshexec command ” { command } ”username ” { ssh . username } ”password ” { ssh . password } ”h o s t ” { hostname } ”t r u s t ” true ” / / t a r g e t
Apache Ant 68469470 t a r g e t name ” r e s t a r t h o s t ” a n t c a l l t a r g e t ” remote exec ” param name ” command ” v a l u e ” shutdown r now ” / / a n t c a l l echo taskname ” w a i t f o r ” message ” Wait f o r { hostname } t o s t o p . . . ” / w a i t f o r maxwait ” 5 ” m a x w a i t u n i t ” minute ” checkevery ” 100 ” not h t t p u r l ” h t t p : / / { hostname } ” / / not / w a i t f o r echo taskname ” w a i t f o r ” message ” Wait f o r { hostname } i s up again . . . ” / w a i t f o r maxwait ” 5 ” m a x w a i t u n i t ” minute ” checkevery ” 100 ” h t t p u r l ” h t t p : / / { hostname } ” / / w a i t f o r / t a r g e t
Apache Ant 35436 t a r g e t name ” j m e t e r ” depends ” s e t t i n g s i n i t , s t a r t j m e t e r ” / t a r g e t name ” s t a r t j m e t e r ” a n t c a l l t a r g e t ” s t a r t j m e t e r before hook ” / j m e t e r jmeterhome ” { l o c a l . j m e t e r . home . d i r } ”r e s u l t l o g ” { l o c a l . jmeter . log . f i l e } ”t e s t p l a n ” { l o c a l . j m e t e r . t e s t . d i r } / { j m e t e r . f i l e } ” p r o p e r t y name ” j m e t e r . data . d i r ” v a l u e ” { l o c a l . p r o j e c t . data . d i r } ” / p r o p e r t y name ” j m e t e r . rampup . t i m e ” v a l u e ” { j m e t e r . rampup . t i m e } ” / p r o p e r t y name ” j m e t e r . e x e c u t i o n . t i m e ” v a l u e ” { j m e t e r . e x e c u t i o n . t i m e } ” / / j m e t e r a n t c a l l t a r g e t ” s t a r t j m e t e r a f t e r hook ” / / t a r g e t
Apache Ant 65366367368 t a r g e t name ” backup data from h o s t ” scp r e m o t e f i l e ” { ssh . username } : { ssh . password }@ { hostname } : ’ { remote . webserver .error . log } ’ ”l o c a l t o f i l e ” { l o c a l . b u i l d d i r } / { hostname } e r r o r . l o g ”t r u s t ” true ” / scp r e m o t e f i l e ” { ssh . username } : { ssh . password }@ { hostname } : ’ { remote . webserver .access . l o g } ’ ”l o c a l t o f i l e ” { l o c a l . b u i l d d i r } / { hostname } access . l o g ”t r u s t ” true ” / scp r e m o t e f i l e ” { ssh . username } : { ssh . password }@ { hostname } : ’ { remote . php . e r r o r .log } ’ ”l o c a l t o f i l e ” { l o c a l . b u i l d d i r } / { hostname } p h p e r r o r s . l o g ”t r u s t ” true ” / scp r e m o t e f i l e ” { ssh . username } : { ssh . password }@ { hostname } : ’ { remote . b a s e d i r } /s c r i p t s / load . log ’ ”l o c a l t o f i l e ” { l o c a l . b u i l d d i r } / { hostname } l o a d . l o g ”t r u s t ” true ” / / t a r g e t
Test SetupWhat do we actually test?
Test Setupiftstat, iftopLoad BalancerjMeterResponse TimesVerify f, FrontEndResponse Timesvmstat,topSlaveMasterSlow Query Log, Services
HardwareITest in a realistic environmentIIIJMeter might have serious hardware requirementsIIIIIIf your software runs in the cloud test against virtualenvironmentIf you use real hardware, also test against real hardwareUse real hardwareUse the biggest VM availableEnsure that not the JMeter hardware is the bottleneckBe sure that the network is not the bottleneckI See ifstat, iftopMeasure several system metricsI See vmstat, top
Continuous PerformanceIPlugins available for:IIIJenkinsSonarMaintaining all those servers can be expensive
Continuous testing with Jenkins
QuestionsICommon questions:IIICan we survive christmas?What is the maximum we can accomplish with the currentsetup?Does (APC XCache MemCache) really help us?
Getting realistic settingsIAsk for real access logs:IIYour customer usually only knows very broad values, like:IIIIExtract exact request model from those1.000.000 PIs per month30.000 sold articles per month45.000 registrations per monthVery seldom, that you get more:IIIAsk for access statistics before Christmas and the ratiocompared with regular monthsAsk for hours with the highest conversion ratesMaybe get the aggregated access logs from existing similarshops
Example calculationICustomer provided values, for a classic webshop:IIIIIIIIII1.000.000 PIs per month30.000 sold articles per month45.000 registrations per monthPer day: 1.000.000/26 38.500 (non-business)Per hour: 38.500/12 3.200 (national shop)Peak hour: 3.200 8 25.500 (18:00 to 19:00)Per second: 25.500/3600 7PI/sAdd Christmas / Easter bonusAdd launch bonusSo . . . 50 PI/s should be safe?IIISpare resources for scaling are always a business decisionProvide with trade-off: Costs vs. downtime / slownessFail gracefully
VerificationIDo your requests actually model customer requirements?IICompare generated accesss logs with real access logsCompare user registrations / checkouts per hour withrequested values
OutlineMotivationConclusion
ConclusionIPlan your test scenarioIUse realisitic thresholdsIChoose the right toolwith care
Continuous Performance Testing Shopware D
