Transcription
MEASURING SOFT SKILLS & LIFE SKILLS ININTERNATIONAL YOUTH DEVELOPMENTPROGRAMSA REVIEW AND INVENTORY OF TOOLSMAY 2017This report is made possible by the support of the American People through the United States Agency forInternational Development (USAID) under YouthPower Action, Contract number AID-OAA-TO-15-00003/AIDOAA-I-15-00009. The contents of this report are the sole responsibility of FHI 360 do not necessarily reflectthe views of the United States Agency for International Development or the United States Government.
This page is intentionally left blank
YOUTHPOWER ACTIONMeasuring Soft Skills & Life Skills inInternational Youth Development Programs:A Review and Inventory of ToolsMay 2017Disclaimer:The author’s views expressed in this publication do not necessarily reflect the views of theUnited States Agency for International Development or the United States Government.
This report is made possible by the support of the American People through the United StatesAgency for International Development (USAID) under task order contract number AID- OAA-TO15-00003, YouthPower Action under IDIQ contract number AID-OAA-I-15-00009, YouthPower:Implementation.Recommended format for citation: Galloway, T, Lippman, L., Burke, H., Diener, O., andGates, S. (2017). Measuring Soft Skills & Life Skills in International Youth DevelopmentPrograms: A Review and Inventory of Tools. Washington, DC: USAID’s YouthPowerImplementation IDIQ- Task Order 1, YouthPower Action.Photo Credit (bottom right, cover page): FHI 360/USAID El Salvador, Education for Childrenand Youth Program
TABLE OF CONTENTS1.Executive Summary . 12.Acknowledgements . 93.Introduction and Purpose . 104.Research Landscape . 135.Methodology . 206.Analysis of Findings . 307.Conclusion and Recommendations . 528.References . 579.Appendices . 60
FIGURESFigure 1. Cross-Sectoral Youth Development: Top Supported Skills Across Fields . 11Figure 2: Steps in Identification of Measurement Tools . 20Figure 3: Number of Tools Measuring Top 9 Cross-Cutting Skills, by Score Position . 31Figure 4: Types of Tools for Each Key Skill . 32Figure 5: Percent of Tools Categorized by Number of Skills Measured . 34Figure 7: Percent of Tools by Evidence of Reliability . 36Figure 8: Number of All Tools that Predict Relevant Outcomes . 37Figure 9: Number of Tools by Ease of Use Based on Criteria . 38Figure 10: Percent of Tools by Age Ranges Assessed . 40TABLESTable 1. Most Supported Skills in the Literature within the Domains of Workforce Success,Violence Prevention, and Sexual and Reproductive Health . 11Table 2: First Screen Characteristics and Definitions . 23Table 3: Inventory Characteristics and Definitions . 25Table 4: Score Criteria Definitions and Range . 27Table 5: Top Scoring Tools that Measure Key Skills of Interest . 44Table 6. High Scoring Tools: Scoring Criteria Results . 47
ACRONYMS AND ABBREVIATIONSBARSBehaviorally anchored rating scaleCAWSChild and Adolescent Wellness ScaleCHKSCalifornia Healthy Kids SurveyCPYDSChinese Positive Youth Development ScalePISAProgram for International Student AssessmentRSQResponses to Stress QuestionnaireSEHMSocial and Emotional Health ModuleSENNASocial and Emotional or Non-Cognitive National AssessmentSJTSituational judgment testSRHSexual and reproductive healthUSAIDUnited States Agency for International DevelopmentWfDWorkforce development
1. Executive SummaryPURPOSEIn recent years, as the evidence base on the importance of soft and life skills for fosteringpositive youth outcomes has grown, international youth development programs haveincreasingly focused on interventions that develop those skills (also referred to as socioemotional skills, transferrable skills, non-cognitive skills, and developmental assets, amongother terms). For the sake of clarity, this paper hereafter uses the term “soft skills” to refer to thisbody of skills -- as this term is widely understood among youth, employers, programimplementers, and researchers – while acknowledging that the term “life skills” is preferred bymany in the sexual and reproductive health field, and that other terms may be preferred in othercontexts. The growth in soft skills-focused interventions has resulted in an urgent need amongyouth development programs for soft skill measures that can be used for programimplementation and evaluation.Soft skills measurement is still an emerging area of research, however, and the landscape ofsoft skills measures is varied and fragmented across disciplines. This report attempts to bringclarity to this field by identifying existing instruments that can be used or adapted for use acrossyouth programs in developing country contexts. USAID’s YouthPower Action project hascompleted a review of soft skill measurement tools and created an inventory describingcharacteristics that can be useful to international youth development programs that seek toassess participants’ soft skills. This summary report describes general findings about theuniverse of tools reviewed, as well as specific findings about tools that measure a select set ofkey soft skills, and suggests recommendations for improving those resources.Prior work by YouthPower Action identified key soft skills that foster positive workforce, violenceprevention, and sexual and reproductive health (SRH) outcomes (see the papers “Key SoftSkills for Cross-Sectoral Youth Development” by Gates et al. (2016) and “Key ‘Soft Skills’ thatFoster Youth Workforce Success: Toward a Consensus Across Fields” by Lippman et al.(2015)). From those systematic reviews of the literature, three skills emerged with the highestdegree of research support across all three outcome areas: self-control, positive self-concept,and higher order thinking skills. Four additional skills rose to the top for certain outcome areas,but not all: social skills, communication, goal orientation, and empathy. These cross-cuttingseven skills were the focus of the measurement tool review. In addition, the skills of hard workand dependability, responsibility, and positive attitude were also noted in the search formeasures since they received strong support in the workforce literature, and the latter tworeceived support at a lower level across all three outcome areas.State of the Field of Soft Skills MeasuresMeasurement tools may be used for a number of different purposes for international youthdevelopment programs, including: 1) formative assessments, to inform program participants oftheir progress; 2) implementation, to provide programs with information for the purpose of betterMeasuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 1 of 57
implementing their programs; 3) summative or descriptive, to describe or monitor the progressof youth at the group-level within a program; and 4) evaluative, to evaluate the effectiveness ofa program in developing skills or having an impact on specific outcomes through skillsdevelopment. Measures may differ by their use in form, content, the nature in which scores arereported, and the level of standards applied (Stecher and Hamilton, 2014). Multiple types ofmeasurement tools exist, and there are a number of different ways to organize them. Theinventory of measures includes: self-reports and self-ratings, and ratings and observations byothers; performance assessments, direct assessments from tests, and simulations, includinggames; and mixed methods measures.The field of soft skills measures faces a number of methodological challenges, including: Balancing technical considerations such as reliability, validity, and measurementinvariance Using tools to reliably measure change in skills over time, when measuring a soft skill ata single point in time is itself challenging The prevalence of self-report methods that are known to suffer from biases Developing or adapting tools for use across cultures and contexts with limited resources Lack of implementer inclusion in tool designBalancing these challenges can be difficult and there are trade-offs for every method. Thisreport reviews these challenges and discusses potential solutions.MethodologyThe YouthPower Action team conducted a review of close to 300 instruments to inform the field.Instruments were screened out that did not address the key soft skills, were not developed foryouth between the ages 12 and 29, or which had a cost associated with their purchase andadministration. Free access to instruments was considered necessary for programs around theworld to use them and to build the state of the evidence for the field. Seventy-four instrumentsmet those three criteria. An inventory of those measures was then created, which addressedcharacteristics of each instrument, which are described in detail in the Methodology section ofthe paper.The team then reviewed each tool based upon a set of criteria that was developed with inputfrom soft skill measurement experts and implementers. Each of these criteria are described inmore detail in the Methodology section. Each tool was then scored tool according to the degreeto which it met a set of seven criteria. The criteria include: Evidence of use by international youth development programsEvidence of validityRelevant validation sampleUsed with youth development outcomes of interestEvidence of reliabilityEvidence of international usageMeasuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 2 of 57
Ease of administration (points were granted for not needing trained personnel foradministration, short length, and availability in other languages)The tools were then divided into three groups based upon the degree to which they met all ofthe criteria: high (meeting five to seven criteria), medium (meeting from three to fewer than fivecriteria), and low (meeting fewer than three criteria).Limitations of Methodological ApproachIt is important to note a few limitations of the methodology. First, although this project hasidentified many key soft skills measurement tools, it should not be considered a comprehensivelist. Second, this identification and screening of tools represents those tools that were availableas of 2016. Tools may have been excluded due to their incomplete nature, or because they arestill undergoing validity and reliability testing, or they fell outside the scope of work for thisproject.Third, comparisons of tools are difficult, which underlines the importance of using or adaptingtools for specific purposes and contexts. Although the focus of this project is on tools that wouldbe appropriate to contexts in which USAID and other international youth development efforts areworking, many tools identified target U.S. or Western educational contexts, reflecting theburgeoning interest in soft skill measurement.Finally, given the breadth of contexts in which soft skills are measured, it should be apparentthat no one tool is capable of meeting all the measurement needs of youth developmentprograms. The purpose of the list of tools that have been categorized in the inventory is todescribe the breadth and depth that current tools reach both in the soft skills they measure andin the potential uses they may serve. The findings should not be used to definitively mark onetool as more useful or “better” than others; instead the scores are meant to describe differencesamong the tools with respect to measuring key cross-cutting soft skills for youth. Theirusefulness will vary according to the needs of each program, including the skills that are thefocus of the programs, the age group participating, and the purpose for which the tool will beused.Overall Findings from the InventoryHigh-scoring tools exist for each of the key soft skills previously identified. The evidencecompiled suggests that the field of measurement is generally well-aligned with literature on thekey soft skills that are most supported by evidence as promoting positive cross-sectoraloutcomes. For example, self-control has the most measures in the inventory (43), whereaspositive attitude has the least (19). Overall, the field of measurement remains largely dominatedby self-report measures for most of the key skills, and the availability of other types ofmeasurement (e.g., report by others) is limited and uneven.Although evidence of acceptable levels of reliability were found among the majority of tools (63percent), evidence of acceptable levels of validity was found among a minority of tools (44percent). The age group that enjoyed the most tools was 15–19 years, followed by 12–14.Measuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 3 of 57
There were fewer tools available in older age groups. Generally, the majority of tools met criteriafor ease of administration. A number of tools have been tested in various regions of the world.Selected High-Scoring Tools Measuring Top Three SkillsThe final step was to highlight the tools that earned high scores and measured the top threeskills that are linked to all three outcomes areas: higher order thinking skills, positive selfconcept, and self-control. These tools were selected from the larger inventory as potentially ofgreatest interest to international youth development programs working to promote positiveworkforce and sexual and reproductive outcomes, and preventing violence. Some of the toolshave been used in conjunction with other outcome areas as well, such as education,psychological and emotional health, substance abuse, and health (see the inventory for moredetails).There are 10 such tools (see Table 5 on page 42 for a breakdown of skills measured by eachtool): California Healthy Kids Survey: Social and Emotional Health ModuleChinese Positive Youth Development Scale (CPYDS)SENNA 1.0SENNA 2.0Child and Adolescent Wellness ScaleThe Anchored BFI ToolThe Big Five InventoryKnackJamaica Youth SurveyResponses to Stress Questionnaire (RSQ)In two cases, this group includes different versions of the same tool (SENNA 1.0 and 2.0; theBig Five Inventory and the Anchored BFI).In the this report, each of these tools is described in depth from the perspective of their utility forinternational youth development programs.The tools generally fall into the following three categories of usage:Program evaluation: The Chinese Positive Youth Development Scale and the Jamaica YouthSurvey meet the above-mentioned criteria and have been used to evaluate international youthdevelopment programs. The Chinese PYD Scale has the advantage of assessing eight of thetop nine skills, whereas the Jamaica Youth Survey assesses five.Group performance monitoring: The California Healthy Kids Survey, Social and EmotionalHealth Module, and the Brazilian SENNA surveys, are instruments of excellent quality that areuseful for monitoring group performance for summative, descriptive purposes, and which havebeen used in schools and school districts. They could be used for evaluations where group-leveldata are needed, but they are not validated for use by evaluations that seek to measureindividual improvements in soft skills over a program’s duration.Measuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 4 of 57
Individual assessments: The rest of the tools can be used for individual psychological or skillassessments and have been shown to be correlated with outcomes of interest. They can beused in formative assessments in which program staff give feedback and coaching to youthparticipating in the programs, and when grouped, may be informative for improving the targetingof skills within a program and for program implementation purposes. They are useful fordetecting differences among individuals in a program at one point in time, but they may not besensitive enough or validated for evaluation designs that need to detect improvements inindividuals’ skills over the duration of a youth development program.Programs will need to evaluate the tools in this inventory and this extracted list of tools for theirown purposes. A measurement instrument needs to align with the program it is being used for,as well as the design of an evaluation. Considerations may include whether the skills beingaddressed by the program match the skills that are measured in the assessment underconsideration, whether the tool has been validated for use with youth of the same age as are inthe program, whether it enjoys acceptable levels of validity and reliability, whether the tool hasbeen used for the same purpose as is envisioned by the program or program evaluation, andwhether it has been used to measure an impact on outcomes of interest to the program. Not allcriteria will be of equal importance to every program.Challenges for the FieldMany excellent tools measure soft skills and new ones are being developed. In general, the fieldcurrently exhibits some weaknesses and limitations that obstruct their usefulness for programmonitoring and evaluation. In addition, some challenges affect the ability to build evidence in thefield across programs, which is essential in order to learn what is working and which programsneed to be scaled up. Several challenges need to be addressed by the field.Terminology: The lack of a common terminology and skill definitions across measurementinstruments hampers the ability of program implementers and evaluators to choose instrumentsthat match the set of skills addressed by programs, and to compare results across programs. Italso hampers the ability to build the evidence across countries, cultures, research disciplines,policymakers, funders, and practitioners. Proposed common terminology and skill definitionsthat would bring coherence to the field were suggested in “Key ‘Soft Skills’ that Foster YouthWorkforce Success: Toward a Consensus Across Fields” (Lippman et al., 2015) which wasdrawn from the research terminology across fields and studies, but also with attention to theterms used by youth, practitioners, and employers.Evidence of reliability and validity: As noted in the analysis, many tools lacked evidence ofreliability and validity, as well as differential item functioning and measurement invariance,which are essential to provide confidence in the tools. Developers need to be encouraged topublish the results of their tests with their validation samples, and those who have used the toolfor assessing youth along with outcomes need to be encouraged to report their reliability andvalidity.Prevalence of self-report methods: All of the 10 tools highlighted above—except the Knackgame—and most of the tools in the inventory use youth self-rating scales, which suffer fromMeasuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 5 of 57
reference and social desirability biases. It is known that there is a tendency in most cultures torate oneself at the high end of a scale on a socially desirable quality, as well as to rate oneselfin reference to one’s own group. These tendencies not only bias results, but obstruct accuratecomparisons across participants in a program, or across programs and cultures, and acrosstime. Using reports by others along with self-reports, and focusing items on actual observablebehaviors rather than endorsements of statements, may produce more objective results (Bladeset al., 2012; Center for the Economics of Human Development, 2015). For example, theFlourishing Children Project’s Goal Orientation scale includes the following behavioral item,“How often do you make plans to achieve your goals?” on a frequency scale from “none of thetime” to “all of the time.”In addition, anchoring vignettes and situational judgment tests have been successful in reducingthese biases and increasing validity and reliability, but require a more sophisticated and costlyadministration and analysis process, and situational judgment tests require a high level ofliteracy of respondents. They have not yet been validated to detect change over time.Response scales: Response scales are often overlooked in reviews of instruments, but theyare critical in determining the sensitivity of items to detect differences between programparticipants and within participants over time. Most of the instruments reviewed use simpleLikert scales, which are good for identifying differences in general tendencies betweenindividuals, but finer grain response scales are needed. Specifically, improved response scalescould address the tendency toward an upward bias in self-report, by capturing variation at theupper end of scales to differentiate between youth who excel at a skill and those who are justabove average (Lippman et al., 2014). Making such distinctions could establish thresholds thatcould help answer the question of how much of a skill is enough to affect an outcome. Finergrained responses at the upper end also allow for the detection of growth over time within anindividual, due to a “ceiling” effect. If a youth rates highly at the start of a program, there is noroom on the scale to detect growth. Measuring frequencies of behaviors, when possible, is moreobjective than the degree of endorsement by the youth of a skill, and can be used in reports byothers as well (Lippman et al., 2014). When youth reports are triangulated with measures byothers for more objectivity, it raises the additional challenge of making sure that both youth andadults or “other” reporters share the same concept/understanding of the skill, which is, ofcourse, essential to model and develop the skill among youth.Developmental appropriateness: There are differences in how skills manifest as youth age.The age span from 12–29 is large and encompasses huge differences in development,including cognitive processing, identity formation, emotional regulation and executive function,social contexts, life experiences, and academic, technical, physical, and practical skills, to namea few. Items need to be used, adapted, or developed that are appropriate for specific agegroups and that reflect the youth’s understanding of a skill and how it is demonstrated acrosscontexts and relationships, such as school, work, with peers, or family members. Mostmeasures found were for adolescents ages 15–19 rather than early adolescents or youngadults, and so will need to be adapted or developed to suit all age groups of interest.Measuring change over time: Research is needed on how to reliably measure change in softskills at the individual level over time. This is needed specifically for program evaluations thatMeasuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 6 of 57
seek to determine whether a program has been successful in improving individual skills, but fewmeasures have been validated for that use. Some assessment developers warn against usingtheir measure for such purposes. Programs can succeed in educating youth and raisingawareness about what is involved in a skill, and in giving youth practice using a skill, yet scorescan decline in the program as a result of youth developing a more accurate self-perception oftheir skills in relation to others and to their own potential. The use of frequencies of behaviorsalong with reports by others may help to more accurately measure improvement.Validation of instruments for program evaluation purposes: Many current tools in theinventory can be used for formative assessment—to inform youth so they can improve; and forprogram implementation purposes—to improve a program, but few were found that have beenvalidated for program evaluation purposes. Specifically, the field needs tools that are sensitiveto program interventions of short duration and that will detect change over time either at theindividual or group level, depending on the evaluation design, and link performance on each skillto youth outcomes in order to discern how best to improve skills and improve youth outcomes.RecommendationsAn investment in tool development is recommended to provide the field with an improvedmeasure of youth soft skills that is tailored to the needs of diverse international youthdevelopment programs. A soft skill assessment should be developed that draws from the universe of existingtools, is designed specifically for program use, and is appropriate for the age groups ofinterest. Adaptation might focus first on the high-scoring tools, supplementing asnecessary with other relevant items or scales to adequately measure each skillindependently, and include age and culturally appropriate language that can beascertained through cognitive interviews.Such a tool should measure at least the three key cross-cutting skills (positive selfconcept, self-control, and higher order thinking skills), using common terminology anddefinitions developed for this project that enjoy the strongest evidence across the fieldsof workforce development, violence prevention, and sexual and reproductive health.Preferably, a tool should also include additional skills that enjoy strong support for one ormultiple outcome areas: communication, social skills, empathy, goal orientation, positiveattitude, and responsibility (see the report, “Key Soft Skills for Cross-Sectoral YouthOutcomes” (Gates et al., 2016)).The instrument should be short and easy to administer, translated into languagesneeded for programs in Latin America, Africa, Middle East, and Asia, and the dataresulting from assessments should be easy to analyze and report out.The measure should incorporate multiple methods to mitigate the shortcomings of selfreport. This might include accompanying self-report scales with an observer reportmethod such as program checklists and/or performance tasks, or at least a report fromanother person, preferably a program staff member. The items should measurefrequencies of behaviors that can be reported on by the youth as well as others, which ismore objective than endorsing statements. This will involve developing and testing newMeasuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 7 of 57
response scales that accurately report upon and discriminate frequencies of behaviors,particularly at the upper end of the scale.Given the need for international adaptation, the instrument should be developed andpilot tested in multiple international program contexts and should preferably be validatedfor measuring change over time before being used to evaluate program contributions tosoft skill development.This investment would build upon investments in research on common skills and measures todate, enabling consistency in skill definition and measurement, and, once used by programsthroughout the world, comparability across programs and evaluations, building the evidence inthe field. An immediate benefit to programs would be provided by helping them targetassessment and measurement efforts on the most important skills in a cost-effective manner.The long-term benefit is learning what works to improve youth skills in different contextsthroughout the world and how that relates to youth outcomes across sectors.Measuring Soft Skills & Life Skills in International Youth Development Programs:A Review and Inventory of ToolsPage 8 of 57
2. AcknowledgementsWe would like to thank the numerous experts involved in this study for their insightfulcontributions. In particular, the skill measurement experts and practitioners whom weinterviewed were instrumental in contributing technical insights from their own experience, andinput on and access to soft skills measurement tools. The full list of experts interviewed can befound in Appendix B. In addition, the report and inventory were greatly strengthened by thethorough review and thoughtful comments provided by external experts. Those included KoffiAssouan, MasterCard Foundation; Laurence Dessein, USAID; Clare Ignatowski, IndependentConsultant; Richard Lerner, Tufts University; Karen Moore, MasterCard Foundation; LeeNordstrum, RTI; Rebecca Pagel, EDC; and Rich Roberts, ProExam. The authorswould also especially like to thank Elizabeth Berard, Cate Lane an
Measuring Soft Skills & Life Skills in International Youth Development Programs: A Review and Inventory of Tools Page 1 of 57 1. Executive Summary PURPOSE In recent years, as the evidence base on the importance of soft and life skills for fostering positive youth outcomes
