InternationalLabourOrganizationGuide onMeasuring Decent Jobs for YouthMonitoring, evaluation and learning in labourmarket programmes

InternationalLabourOrganizationGuide on Measuring Decent Jobs for YouthMonitoring, evaluation and learning in labour market programmesOverview

Guide on Measuring Decent Jobs for YouthMonitoring, evaluation and learning in labour market programmesOverview

Copyright International Labour Organization 2018First published 2018Publications of the International Labour Office enjoy copyright under Protocol 2 of the Universal Copyright Convention.Nevertheless, short excerpts from them may be reproduced without authorization, on condition that the source isindicated. For rights of reproduction or translation, application should be made to ILO Publications (Rights andLicensing), International Labour Office, CH-1211 Geneva 22, Switzerland, or by email: [email protected] The InternationalLabour Office welcomes such applications.Libraries, institutions and other users registered with a reproduction rights organization may make copies in accordancewith the licences issued to them for this purpose. Visit to find the reproduction rights organization in yourcountry.Guide on Measuring Decent Jobs for Youth - Monitoring, evaluation and learning in labour market programmes, Overview/International Labour Office. Geneva, 2018.ISBN :978-92-2-131670-1 (print)978-92-2-131671-8 (web pdf)Also available in Arabic: نظرة عامة - الرصد والتقييم والتعلّم يف برامج سوق العمل النشطة . دليل قياس الوظائف الالئقة للشباب ISBN 978-92-2-630801-5 (print), 978-92-2-630802-2 (web pdf), Geneva, 2018.The designations employed in ILO publications, which are in conformity with United Nations practice, and the presentationof material therein do not imply the expression of any opinion whatsoever on the part of the International Labour Officeconcerning the legal status of any country, area or territory or of its authorities, or concerning the delimitation of itsfrontiers.The responsibility for opinions expressed in signed articles, studies and other contributions rests solely with their authors,and publication does not constitute an endorsement by the International Labour Office of the opinions expressed in them.Reference to names of firms and commercial products and processes does not imply their endorsement by theInternational Labour Office, and any failure to mention a particular firm, commercial product or process is not a sign ofdisapproval.Information on ILO publications and digital products can be found at: and layout by the International Training Centre of the ILO, Turin - ItalyPrinted in Switzerland

AcknowledgementsThe Guide on Measuring Decent Jobs forYouth represents a collaborative effort involving expertise from the International LabourOrganization (ILO) and its main constituentsas well as from external partners. The Guideis a product of the ILO’s Youth EmploymentProgramme Unit (YEP) and is developed inclose collaboration with the other branchesand units of the Employment Policy Department including the Country Employment Policy Unit of the Employment and Labour MarketAnalysis Branch, the Skills and EmployabilityBranch and the Development and InvestmentBranch. Other main contributors include theEvaluation Office, the Enterprise Department,the Research Department, Better Work, theRegional Office for the Arab States, the Decent Work team for North Africa and CountryOffice for Egypt and Eritrea, the Country Office for Algeria, Libya, Morocco, Tunisia andMauritania, the Bureau for Workers’ Activities, the Bureau for Employers’ Activities andthe International Training Centre of the ILO(ITCILO).The drafting of the guide was made possiblein-part through generous support from theInternational Fund for Agricultural Development (IFAD), under an IFAD-ILO project entitled “Strengthening gender monitoring andevaluation in rural employment in the NearEast and North Africa.” Through rigorous impact research, this capacity developmentand learning grant project aims to understand “what works” in the promotion of gender mainstreaming, with the ultimate goal ofreaching gender equality in rural employmentoutcomes across the region.The drafting process included review at an experts meeting in March 2017 where excellentcomments were received. The expert meeting included: Samuel Asfaha, Naomi Asukai, Nathalie Bavitch, Romulo Cabeza, JuanChacaltana, Veronica Escudero, Sergio IriarteQuezada, Jean-François Klein, Tobias Lechtenfeld, Pedro Martins, Nawel Marzouki, RuteMendes, Mohammed Mwamadzingo, NiallO’Higgins, Aurelio Parisotto, Miquel Pellicer,Arianna Rossi, Roland Sarton, Olga StrietskaIlina, Steven Tobin, Sanchir Tugschimeg, Peter Van Rooij and Peter Wichmand.Drew Gardiner coordinated the drafting of theguide and authored Notes 1, 3 and 7. JonasBausch authored Notes 4 and 5. Paul Berbée,Verena Bruer, Paul Dyer, Sonja Kovacevic, Susana Puerto Gonzalez and Felix Weidenkaffprovided substantial technical inputs throughout. Matt Ripley authored several of the casestudies and acted as penultimate reader.Copy-editing was done by Book-Now andtypesetting by ITCILO.Note 6, “A step-by-step guide to impact evaluation” is an adaptation of a chapter of theGlobal Partnership for Youth Employment’s“Measuring success of youth livelihood interventions: A practical guide to monitoring andevaluation”, authored by Kevin Hempel andNathan Fiala.Sangheon Lee, Director, Employment PolicyDepartment, Sukti Dasgupta, Chief, Employment and Labour Markets Analysis Branchand Valter Nebuloni, Head, YEP, providedoverall guidance to the production of thispublication.Questions or feedback on this guide can besubmitted to YEP at [email protected]


ContentsOverviewNote 1: Diagnosing, planning and designing youth employmentinterventionsNote 2: Concepts and definitions of employment indicators relevant foryoung peopleNote 3: Establishing a monitoring systemNote 4: Enhancing youth programme learning through evaluationNote 5: Impact evaluation methods for youth employment interventionsNote 6: A step-by-step guide to impact evaluationNote 7: Evidence uptake in policy formulationAppendix: Answer key for case studies (available separately)OVERVIEWv

PrefaceIn June 2012, the International Labour Conference of the ILO resolved to take urgent actionto tackle the unprecedented youth employment crisis through a multipronged approachgeared towards pro-employment growth anddecent job creation. The resolution, entitled“The youth employment crisis: A call for action” (ILO, 2012a), contains a set of conclusions that constitute a blueprint for shapingnational strategies for youth employment. Theassociated background report (ILO, 2012b)warns: “(M)ajor gaps in knowledge (on “whatworks” on youth employment) persist. Therehave been relatively few rigorous evaluations of youth employment policies and programmes, of their impact in the short and longterm, and of their relative cost-benefit, including in developed countries. This needs to beremedied since lessons learned from evaluations can lead to greater programme effectiveness and better targeting of scarce resources.Continuous building of the knowledge baseon country policies and programmes and theimpact evaluations of the range of measuresis a paramount priority.”The ILO has responded to this call by makinggreater investments in efforts to develop theevidence base on youth employment. “Whatworks in youth employment” is the ILO’s offer to constituents to assist them in the rigorous monitoring and evaluation of their youthemployment programmes and policies. Theobjective is twofold: (1) to ensure accuratemeasurement of youth employment outcomesand (2) to promote evidence-based youthemployment interventions and programmesthrough policy dialogue. By building capacities to measure results, the ILO contributesto tracking progress on Sustainable Development Goal (SDG) 8 on the promotion of sustained, inclusive and sustainable economicgrowth, full and productive employment anddecent work for all.The ILO’s work on results measurement inyouth employment began in 2010 with theFund for Evaluation in Youth Employment.This action was complemented by the global“Youth employment crisis: A call for action” and its corresponding “Follow-up plan”(2012–2019), which appealed for improvedassessment of interventions to support betteryouth employment outcomes. Then in 2013,the “Area of Critical Importance: What Worksin Skills and Youth Employment” was set-upto provide financial and technical assistancefor the rigorous assessment of youth employment. Regional approaches have since beenestablished, including the Taqeem (meaning“evaluation” in Arabic) Initiative. Taqeem is apartnership between the ILO and the International Fund for Agricultural Development(IFAD) as part of an IFAD-financed project titled “Strengthening gender monitoring andevaluation in rural employment in the NearEast and North Africa”. Through rigorous research, this capacity development and learning project aims to understand “what works”in the promotion of gender mainstreaming,with the ultimate goal of achieving genderequality in rural employment outcomes acrossthe region.It is in this context that we have developedthe present guide. It offers a comprehensiveand accessible introduction to the topics ofresults measurement and impact assessment, their practical application in the youthemployment field and how evidence createdvia results measurement strategies can leadto improved programming. It is our sincerehope that this guide will help social partnersand practitioners to make informed decisionsOVERVIEW1

GUIDE ON RESULTS MEASUREMENT OF DECENT JOBS FOR YOUNG PEOPLE2in choosing the evaluation frameworks thatbenefit their organizations and youth-centredprogrammes and will contribute to enhancingthe youth employment sector in general. If, inthe future, we are able to draw more robustevidence from all the good work being undertaken to support youth throughout the worldand the wide experience of the various actors,we will also have a stronger voice for convincing policy-makers to scale up interventionsthat have proven successful. We look forwardto the continued collaborative work of policymakers, development practitioners and otherstakeholders in providing tomorrow’s leaderswith the economic opportunities they deserve.Valter NebuloniHeadYouth Employment Programme UnitEmployment and Labour MarketAnalysis BranchSukti DasguptaChiefEmployment and Labour MarketAnalysis BranchEmployment Policy DepartmentOVERVIEW

IntroductionInvesting today in the employment ofyoung people means investing in thepresent and future of our societies.Guy Ryder, ILO Director-GeneralYoung people are estimated to account forover 35 per cent of the unemployed population worldwide (ILO, 2017a). While the globalyouth unemployment rate stabilized at 13 percent in 2016, it rose slightly to 13.1 per centin 2017. The estimated figure of 70.9 millionunemployed youth in 2017 is an important improvement from the crisis peak of 76.7 million in 2009, but this number is expected torise by a further 200,000 in 2018, reaching atotal of 71.1 million. More importantly, 39 percent of young workers in the emerging anddeveloping world – 160.8 million youth – areliving close to or in extreme poverty, i.e. onless than US 3.10 a day (at 2011 purchasingpower parity (PPP)). More than two in everyfive young people in today’s workforce areunemployed or are working but remain poor,a striking reality that is impacting societiesacross the world.Despite this challenging situation, the adoption of the Sustainable Development Goals(SDGs) in 2015, and their explicit focus on decent work, offers hope for young people making the transition to the world of work. The ILOand its constituents – governments, workers’and employers’ organizations – are workingtowards global targets on decent work and inclusive growth. The interests of young peopleare strongly represented in the SDGs, specifically in Goal 1 on fighting poverty, Goal 4on providing quality education and Goal 8 ondecent work and economic growth. The wellbeing of the most vulnerable young peopleis explicitly addressed in the Goal 8.6 target,“By 2020, substantially reduce the proportion of youth not in employment, educationor training”. For youth employment practitioners, this group of excluded and marginalizedyoung people is a focus of primary concern.The trajectory of this growing young generation will depend on their social and economicintegration, their ability to live fulfilling livesand their participation in and contribution tosociety.Taking effective action towards realizingthese SDGs requires a systematic approachto measurement and evaluation. The UnitedNations (UN) community has laid out a results-based reporting plan, designed to assess and monitor the realization of the SDGsand the contributions of specific UN agenciesand member States. In monitoring and evaluating SDG implementation, the Agenda 2030document provides guidance for systematicand rigorous follow-up processes (UN, 2015).The Agenda 2030 document recognizes thatthe review process for the SDGs requires“enhanced capacity-building support for developing countries, including the strengthening of national data systems and evaluationprogrammes”.This guide contributes to our collective efforts in moving towards fulfilling the SDGsand securing better youth employment prospects by promoting the implementation ofeffective results measurement systems inOVERVIEW3

GUIDE ON RESULTS MEASUREMENT OF DECENT JOBS FOR YOUNG PEOPLEyouth employment programmes and interventions. With an extensive section on attributingchanges to youth employment interventions,this guide fosters evidence-based policy andprogramming choices by equipping ILO partners and stakeholders with the tools to attribute their actions to positive youth relatedoutcomes. Guidance on results measurementof decent jobs for youth is provided with thepremise that, for jobs to be created, employment conditions to be improved and SDGs tobe realized, we need more evidence on “whatworks”. And, more importantly, can we say“how” and “why” interventions work in orderto inform future programme design?Although knowledge gaps remain, we cansafely say that the evidence base is strongerthan it was 10 years ago and it continues togrow at an impressive rate. The recently released systematic review on “Interventions toimprove the labour market outcomes of youth”(Kluve et al., 2017) identified 113 rigorousevaluation reports to include in its meta-analysis, 74 of which have been released since2010. The main findings of the review werealso encouraging: Overall, youth employmentinterventions can produce positive job andincome effects for young people. As globalevidence increases, so will the effectivenessof youth employment interventions, resultingin the creation of more decent jobs for youngpeople.This guide proposes a framework that linkspractical, implementation-focused measurement and monitoring to research-oriented impact evaluation. This is achieved in a seriesof seven “notes”, which lead readers throughthe key steps in diagnosing and formulatingyouth employment interventions, setting up aresults measurement system and defining corresponding key youth employment indicators.It then moves on to providing an overview ofthe types of evaluation approaches availableto measure youth employment outcomes, including non-experimental, quasi-experimental and experimental methods that encouragereflection on the challenge of attribution. Themenu of evaluation methods available is thendiscussed and, finally, guidance is providedon how to ensure that the findings from evidence-based evaluations are taken up indialogue and policy formulation processes related to employment and young people. Thetechnical notes are supplemented by interactive case studies supplied at the end of eachnote, to be used as a complement to classroom based learning on this topic.ABOUT THIS GUIDEObjectiveWith this guide, we aim to equip readers withthe basic set of concepts and tools neededto make informed decisions about how bestto measure and evaluate the results of youthemployment interventions. We review the entire life cycle of a youth employment intervention, starting with diagnostics and programmedesign, then moving on to setting up a monitoring system and measuring results. We seekto provide a clear understanding of the variety4OVERVIEWof evaluation options available and the issuesto consider in choosing the most appropriatemethod, given the learning objectives and operational context. Additionally, the guide describes how to manage an impact evaluation,if that is the assessment method of choice.Our overarching goal is to strengthen thefoundation of sound programming and policy-making by increasing the number of quality evaluations in the youth employment field,thereby facilitating the process of scaling up

and replicating successful interventions. Wewant to promote a perception of evaluationas a tool for both internal and external learning rather than as simply an accountabilityexercise. Using evaluation to learn about theperformance of an intervention can facilitateadaptive management where problems canbe recognized early and projects adjusted accordingly. Learning that is shared publicly canhelp others to select the most effective intervention models, ensuring that resources spenton youth employment are optimally used.Learning toolThe guide can also be used to complementlearners taking training courses on resultsmeasurement and what works in youth employment. From 2010 to 2017, the ILO hastrained over 2,000 constituents on resultsmeasurement in youth employment using anexperiential learning approach called “Evaluation Clinics”. This guide closely follows thelearning modules and curriculum of the ILOEvaluation Clinics but provides learners withmore in-depth information and tools to buildon the knowledge gained in the Clinics.AudienceWhile this guide is primarily a reference toolfor ILO constituents, it can also be used by allthose involved in the implementation of youthemployment programmes.The guide has therefore been drafted with thefollowing organizations in mind.Primary audiencesGovernments: Particularly ministries andagencies which focus on employment, youth,labour market services and training. The information may be more relevant to agenciesand institutions which are directly responsible for delivering and monitoring differentprogrammes, such as youth employmentagencies, public employment service agencies and technical and vocational training institutes. It can be used to assist in designingnew public employment programmes.Employers’ organizations: Organizationsrepresenting the interests of private employers who offer youth vocational education,employment and entrepreneurship opportunities. They will be able to use the guide bothto inform their advocacy on important legislative and regulatory issues and to offer valueadded services and advice to their members,based on the latest research and evidence.Companies have an increasingly large stakein issues involving youth and making surethey are prepared with the requisite skills andattitudes to become the employers and employees of the future.Workers’ organizations: Workers’ councils and trade unionists who are interested inthe potential of youth employment interventions for realizing decent work for youth. Theguide will be particularly useful in formulatingprogrammes and interventions for evidencebased advocacy around key policy issues.Workers’ organizations have an interest inlearning about the conditions under whichemployment, and in particular decent employment, can be increased.Youth-led organizations: not-for-profit organizations working in the youth employmentfield whose staff and members are predominantly made up of young people. Most organizations use the UN’s definition of youth whichis a person between the ages of 15 to 24.Secondary audiencesCivil society: NGOs are one of the main implementers of youth employment programmesOVERVIEW5

GUIDE ON RESULTS MEASUREMENT OF DECENT JOBS FOR YOUNG PEOPLEin many countries. As the guide covers the fulllife cycle of a youth employment intervention,it can be used at every stage of a project. Thefocus on evaluation and evaluation methodsallows intervention results to be translated intolessons to inform both follow-up projects andother practitioners.Donors: Multilateral and bilateral development agencies provide a considerable shareof the funding for youth employment programmes. Donors, who have a fiduciary interest in maximizing the impact of their grantfunding and ensuring accountability in theuse of resources, will be able to use the guideto set appropriate evaluation schedules, aswell as to design and commission evidencebased youth programmes.Researchers: Researchers conducting fieldresearch and liaising with programme managers and implementing organizations willfind the information presented in this guiderelevant for adapting standard research methods to local programme conditions. A context-sensitive approach to impact evaluationis presented that facilitates research design.Key referencesThis guide complements existing materialson results measurement, applying them to thespecific area of youth employment. It buildson the following works:XX Measuring success of youth livelihoodinterventions: A practical guide to monitoring and evaluation (Hempel and Fiala, 2011).6OVERVIEWAuthored as a contribution to the GlobalPartnership for Youth Employment, this workwas used as the key reference document.XX ILO Policy Guidelines for Evaluation (ILO,2017b) and ILO Development CooperationManual (ILO, 2015), which set out the overall framework for project monitoring, reporting and evaluation in an ILO context.XX Monitoring and evaluation of youth employment programmes: A learning package(ILO, 2013). This guidebook provides adviceon monitoring the performance of youthemployment programmes and measuringboth short- and long-term outcomes.XX Practical toolkits that emphasize generalmonitoring and evaluation (e.g. Gospariniet al., 2003; Kellogg Foundation, 2004;Kellogg Foundation, 2017, 2004) and otherpublications that focus specifically on impactevaluation (e.g. Baker, 2000; Duflo et al.,2006; Gertler et al., 2016; Khandker et al.,2010; Ravallion, 2008).Considering similar resources on the topicof monitoring and evaluation, our Guideis unique in that it is the only guidance toolthat orients all elements of the results measurement life cycle to the topic of youth employment. These unique elements include achapter advising practitioners on the appropriate youth employment indicators to select(Note 2) ; a list of challenges, and their solutions, that are specific to youth employmentfocused impact evaluations (Note 5); and areflection on the existing body of youth employment evidence and advice on how thisevidence can lead to policy change (Note 7).

OVERVIEW OF THE GUIDE AND HOW TO USE ITWhile the guide leads the reader through allstages involved in formulating youth employment programmes, starting with the diagnostic phase and closing with evidence uptake inyouth employment policy formulation, the mainthrust of the guide is on monitoring, resultsmeasurement and evaluation. Figure 0.1 setsout a simple results measurement cycle andthe relating key elements of the seven notes.measurement. The second note covers labourmarket indicators with particular relevance foryouth employment interventions, while Note 3concentrates on setting up the monitoring system. Note 4 introduces readers to the importance of evaluation before presenting differentapproaches, including performance and impact evaluations. In order to address the challenge of attribution, Note 5 presents a numberof different methods which can help decisionmakers weigh the desired level of rigour withthe feasibility of conducting the research. Note6 guides readers through the step-by-stepThe first note focuses on designing a youth employment intervention and establishing a solidtheory of change as a basis for quality resultsFIGURE 0.1 RESULTS MEASUREMENT CYCLENote 1:Diagnose thechallengeNote 7:Uptake ofevidenceNote 2:SelectindicatorsNote 3:Set-up themonitoringsystemPlanEvaluateand learnNote 4 & 5:Plan theevaluationNote ntNote: 6AnalyzeresultsNote: 6CollectdataOVERVIEW7

GUIDE ON RESULTS MEASUREMENT OF DECENT JOBS FOR YOUNG PEOPLEprocess of implementing a youth employmentfocused impact evaluation. The guide ends byoffering practical advice on ensuring that evaluation evidence is taken up in policy formulation processes.Although it is important to be familiar with allparts of the measurement process, it is notnecessary to read the guide from beginning toend. Instead, each note is conceived as a standalone entity that can be read independently ofthe others, according to each reader’s needs.Table 0.1 indicates which notes are most relevant to different types of readers.Case studiesAt the end of each of the seven notes, we present case studies detailing how different youthemployment interventions apply results measurement strategies. The case studies illustratethe main points in each note and ask readersto apply the concepts they have learned to“real world” situations. The case studies aredesigned to complement classroom-basedlearning, as pedagogic exercises to be discussed in small groups with the assistance ofan expert or facilitator in results measurementin youth employment.While all of the case studies relate to youth labour market interventions, some of them arederived from experiences of the ILO in supporting organizations which are seeking toimprove their results measurement systemsand implement impact evaluation projects.Several of these case studies are drawn fromorganizations that were offered support under ILO’s Fund for Evaluation in Employment,a technical and financial support programmefor youth employment researchers and organizations in the Arab States and Africa regions.The case studies are accompanied by an appendix “Answer key for case studies”, whichis available separately from the seven notevolume of Guide. This answer key appendix8OVERVIEWis intended to be used by facilitators to assist in small group discussions about the casestudies.Overview of key termsThe guide addresses the effective monitoringand measurement of outcomes of youth employment interventions with a specific focuson impact evaluation. It is important here tomake reference to four key terms which areused extensively throughout the guide: resultsmeasurement system, monitoring, evaluationand impact evaluation.Results measurement system refers to theoverall processes, plans, tools and resourcesthat are used to determine whether a programme has been implemented accordingto the plan (monitoring) and is having the desired result (evaluation).1 A results measurement system specifies:XX indicators to be trackedXX milestones (mid-stream) and end targetsXX data collection toolsXX the personnel who will gather, record andanalyse the data andXX the types of reports that will be prepared,including for whom, why and how often.The key activities of a results measurementsystem are monitoring and evaluation:Monitoring tracks the implementation andprogress of an intervention in order to supportprogramme management. Monitoring:XX involves the collection of data on specificimplementation and results indicatorsXX assesses compliance with work plans andbudgetsXX uses information for project managementand decision-makingXX is ongoing1Also known as a monitoring and evaluation (M&E)system.

Table 0.1 Reader’s guideNoteTitle1Diagnosing,planning anddesigning youthemploymentinterventions2Concepts anddefinitions ofemploymentindicatorsrelevant for youngpeople3Establishing amonitoringSystem4Enhancing youthemploymentlearning throughevaluation5Impact evaluationmethods foryouthemploymentinterventions6A step-by-stepguide to impactevaluation7Evidence uptakein policyformulationDescriptionGuides readers throughan employment diagnosticanalysis as the basis fordeveloping the theory ofchange then proceeds toprogramme designReviews concepts anddefinitions of employmentindicators relevant foryoung people in the areasof employment opportunities, employment quality,employment access andemployment skillsPresents the main steps indeveloping a resultsmeasurement system,including how to collectand analyse data.Asks which type of evaluation best suits an individual programme. The answerdepends on evaluationquestions, the context andcharacteristics of the project, and availableresourcesPresents the main featuresof an impact evaluationand focuses on finding agood comparison group toreliably demonstrateimpact. Presents different(quasi-)experimentalmethods for conductingan impact evaluationMoves from the conceptual to the practical level,describing the major stepsinvolved in carrying out animpact evaluation and providing practical resources.These steps cover theentire process, from initialpreparations to the dissemination of resultsHelps readers to plan howevaluation results can beused to influence policyand improve programming. Communication ofresults and stakeholderengagement strategiesare discussedCase studyGovern- Employers/ Civil ResearDonorsments workers society chersLabour market diagnostics for the promotion of rural youthlivelihoods in Zambia Selecting indicatorsfor the NorthernUganda YouthEntrepreneurshipProgramme Establishing a monitoring system for theJordan EconomicGrowth andEmployment Project Developing Terms ofReference for a midterm evaluation of ayouth employmentproject in Egypt Assessing ruralmicroenterprisegrowth through different evaluationmethods Survey design andimplementation forNeqdar Nesharek inEgypt Uptake of evidenceon the effects of skillstraining on youngpeople’s financialbehaviou

The Guide on Measuring Decent Jobs for Youth represents a collaborative effort involv-ing expertise from the International Labour Organization (ILO) and its main constituents as well as from external partners. The Guide is a product of the ILO's Youth Employment Programme Unit (YEP) and is developed in close collaboration with the other branches