Transcription

CHAPTER 3Monitoring at IOM

3MONITORING AT IOMList of abbreviations and acronyms. 433.1. An overview of how to monitor. 443.2. Programme theory. 443.3. Theory of Change. 453.3.1. What is the Theory of Change?. 453.3.2. When to use a ToC. 463.3.3. How to develop a ToC. 463.3.4. How to develop a Theory of Change using the if-then-because formula. 513.3.5. How to review a Theory of Change that applies the if-then-because formula. 533.4. IOM Results Matrix. 563.4.1. The difference between the Theory of Change and a logical framework. 573.4.2. Developing an IOM Results Matrix . 593.4.3. Results Matrix terminology. 613.4.4. Results Monitoring Framework . 713.5. Types of monitoring: An overview. 753.5.1. Types of monitoring. 763.5.2. Strategy- and policy-level monitoring . 783.6. Remote management, monitoring and third-party monitoring. 803.6.1. Remote management. 803.6.2. Remote monitoring. 823.6.3. Third-party monitoring. 863.7. Pulling it all together: Monitoring and evaluation plan. 893.8. Monitoring and reporting on results. 953.8.1. Reporting on results. 953.8.2. Reporting in a narrative donor report.1003.8.3. Reporting and learning.102The following chapter contains links to resources relevant to the content presented. Someresources presented are internal to IOM staff only and can be accessed only by those withIOM login credentials. These resources will be updated on a regular basis. To see the updatedresources, kindly follow this link.42CHAPTER 3Monitoring at IOM

List of abbreviations and AIDaccountability to affected populationsCenters for Disease Control and ProtectionCanadian International Development Agencychief of missionForeign, Commonwealth and Development Office, United Kingdom (formerlyDepartment for International Development (DFID)International Federation of Red Cross and Red Crescent SocietiesUS Bureau of International Narcotics and Law Enforcement AffairsInternational Organization for MigrationInternational Rescue CommitteeJustice and Security Research ProgrammeOffice of Legal Affairsmonitoring and evaluationMigration Governance Frameworkmanagement information systemmeans of verificationnon-governmental organizationOrganisation for Economic Co-operation and DevelopmentOrganisation for Economic Co-operation and Development/Development Assistance CommitteeOffice of the Inspector General’s Central Evaluation functionproject performance reviewProject Information and Management ApplicationProcess and Resource Integrated Systems Managementquantity, quality and timerights-based approachResults Monitoring FrameworkSustainable Development Goalspecific, measurable, achievable, relevant and time-boundsource of verificationTheory of Changeterms of referencethird-party monitoringTransition and Recovery DivisionUnited Nations Development Assistance FrameworkUnited Nations Statistics DivisionUnited Nations Sustainable Development Cooperation FrameworkUnited States Agency for International DevelopmentIOM MONITORING AND EVALUATION GUIDELINES43

Chapter 3 Monitoring at IOMThis chapter looks at the essentials for monitoring an intervention (a project/programme and/or a strategyor policy). It introduces the Theory of Change (ToC) and the IOM Results Matrix, and describes the basictypes of monitoring, including activity, results, financial and risk monitoring, as well as other types ofmonitoring. This chapter also focuses on remote management and monitoring, third-party monitoring(TPM), and explains how different monitoring elements come together to form an overall M&E plan, andfinally, looks at monitoring and reporting on results.3.1. An overview of how to monitorA strong project design is the foundation for successful monitoring. The proposal development stageclearly articulates the desired results an intervention aims to achieve, how it achieves them and stipulateshow progress towards these results will be measured. Modules 1 (Conceptualization), 2 (Proposal design)and 4 (Project management and monitoring) of the IOM Project Handbook provide an overview of thisprocess and show how the foundations for successful monitoring are laid.This chapter builds on the main points mentioned in the IOM Project Handbook and provides furthertechnical guidance, as well as expands on new concepts such as ToC. It primarily focuses on monitoringa project or programme and also shows that the principles of monitoring a project or programme arealso applicable to monitoring a strategy and/or policy. While many concepts covered in this chapter areimportant for both monitoring and evaluation (M&E) – such as the development of a ToC, the IOMResults Matrix, IOM Cross-Cutting Themes, remote management and the development of an M&E plan –guidance for conducting evaluation in IOM is covered in detail in chapter 5 of the IOM Monitoring andEvaluation Guidelines. IOM focuses on four key areas for monitoring: (a) activities; (b) results; (c) budgetand expenditure; and (d) risk.3.2. Programme theoryWhile various definitions exist of the programme theory, this section focuses on approaches most suitedfor the IOM operational context.1 Programme theory is a key aspect of implementation design andexplains how an intervention (project/programme, strategy and policy) is expected to contribute to achain of results.2 It is a representation of all the building blocks that are required to bring about a higherlevel change or results. It is usually formulated at the proposal development stage.3Programme theory is a logical thinking process on how to address a situation and respond to it throughan intervention. It can therefore be useful in providing a conceptual framework for monitoring, as wellas for evaluation.Various labels for programme theory exist, including logic model, intervention logic, causal model, resultschain and ToC. Two complimentary approaches, which are pertinent for IOM interventions, are furtherelaborated in this chapter: (a) ToC; and (b) logical framework, which is represented by the Results Matrixat IOM.While both approaches map out how an intervention leads to results, each has a slightly different purpose.12344This chapter, inter alia, draws on definitions and concepts as shared by BetterEvaluation, n.d., which are considered to be most suited forthe IOM context.Adapted from Rogers, n.d.Ibid.CHAPTER 3Monitoring at IOM

RSOURCEESBetterEvaluationn.d. Home page.Rogers, P.n.d. Develop programme theory/theory of change. BetterEvaluation.3.3. Theory of Change3.3.1. What is the Theory of Change?Clearly articulating the expected results or desired change of an intervention is the foundation for M&E.It is necessary to identify what requires change, what expected change looks like and, finally, how suchchange can be achieved through IOM interventions. This is where ToC comes in handy. While there aremany different definitions of ToC, this section focuses on approaches relevant to the IOM context.4A ToC may be viewed as a tool or methodology to map out the logical sequence of an intervention fromactivities to results, showing multiple pathways that may lead to change, including pathways not relatedto the planned intervention. It may also be viewed as a deeper reflective process and dialogue amongstaff and stakeholders, reflecting the values and philosophy of change that make more explicit theunderlying assumptions of how and why change may occur as a result of an intervention. At itsbest, a ToC is a combination of these two views.It is most often defined as illustrating a link between activities, outputs, outcomes and objectives, creatinga chain of results, referred to as the pathway of change or the causal pathway.5 It is essentially acomprehensive articulation of how and why desired change will occur within a specific context.6 Somebasic components of a ToC often include a big picture analysis of how change is expected to occurin relation to a specific thematic area, an articulation of a specific pathway in relation to this and anassessment framework that is designed to test both the pathway and the assumptions made about howchange happens.7What is a Theory of Change?8Theory of Change is a comprehensive description and explanation of howand why a desired change is expected to happen in a particular context.Specifically, it focuses on mapping out what an intervention or change initiativedoes (its activities) and how these lead to achievement of the objectivethrough results (outputs, outcomes, objectives).In this way, a ToC articulates hypotheses about how change happens byexplaining the connection between an intervention and its effect. It does soby surfacing the logic and rationale for an intervention and articulating theassumptions inherent in the approach (multiple pathways).45678See examples of different shapes of ToCs at Center for Theory of Change, n.d.a.A causal pathway is the process of step-by-step mapping through which all the required preconditions necessary to reach a desired changeare determined.Center for Theory of Change, n.d.b.Stein and Valters, 2012.This definition is adapted by IOM from Center for Theory of Change, n.d.c.IOM MONITORING AND EVALUATION GUIDELINES45

OURCEESSRA ToC can be viewed as a product of a series of critical thinking exercises that provide a comprehensivepicture of the different levels of change expected to occur due to an intervention, at the stage of itsdevelopment, during its implementation, as well as following its completion.Center for Theory of Changen.d.a Theory of Change examples.n.d.bWhat is theory of change?n.d.cHome page.Stein, D. and C. Valters2012 Understanding theory of change in international development. Justice and Security ResearchProgramme (JSRP) Paper 1. JSRP and The Asia Foundation.3.3.2. When to use a Theory of ChangeThe use of a ToC is more and more common and developed for all types of interventions. It can beapplied to design, monitor, as well as evaluate different types of interventions and is best used to measurethe complexity of transformation and change. Because a ToC acknowledges that change is not linear,but dynamic and complex, it often seeks to articulate social, political and community-based change(s)or empowerment initiatives. A ToC is a process-oriented approach that can be used to analyse theinterrelations and/or interactions in complex systems in which IOM, partners and allies work. Such aprocess helps navigate in unpredictable and complex environments and helps track and assess change inthe system to which an intervention may contribute.3.3.3. How to develop a Theory of ChangeIt is important to note that different terminologies may be applicable when defining the multiple pathwaysof change, such as: (a) objectives, outcomes and outputs; (b) long-term, intermediate and short-termoutcomes; or (c) outcomes and pre-conditions. This section uses the terms objectives, outcomes,outputs and activities to align with IOM’s M&E terminology and either develop or supplement IOMresults matrices.ToC is a guiding framework for all stages of thinking, action and sense-making for interventions involvedwith social and/or political change processes.Graphic depictionWhen graphically depicting a ToC, diagrams can be generally flexible in format and may be simple orcomplex. They can be vertical, horizontal or circular.46CHAPTER 3Monitoring at IOM

Figure 3.1. Graphic depiction of a Theory of Change diagramThe graphic depiction of a ToC can help with mapping out multiple causal pathways to identify the mostfeasible one for a given intervention. Another advantage of graphically depicting a ToC is that it makespossible causal links more understandable and immediately visible. It enables comparisons betweendifferent pathways and can help identify implicit assumptions.9Participatory approachThe process of developing a ToC should be participatory and collaborative, and include key stakeholders,as well as the beneficiaries, or people that the Organization seeks to assist, and/or affected populations,who can offer their different perspectives to define what an expected change within a specific thematicfield may look like.10 Their participation can also help identify underlying assumptions that are inherent toexplaining why a particular change is expected to occur.Multiple pathways of changeA ToC acknowledges that change is dynamic and complex and can show different possible pathways thatmight lead to change (see Figure 3.2 where each colour of the arrows represent a different pathway). Theprocess of developing a ToC helps discover these multiple pathways of change.910See examples of different shapes of ToCs in Figure 3.1, as well as at Center for Theory of Change, n.d.a.For the purpose of the IOM Monitoring and Evaluation Guidelines, IOM uses the OECD/DAC definition of beneficiary/ies or people that theOrganization seeks to assist as “the individuals, groups, or organisations, whether targeted or not, that benefit directly or indirectly, fromthe development intervention. Other terms, such as rights holders or affected people, may also be used.” See OECD, 2019, p. 7. The termbeneficiary/ies or people that IOM seeks to assist, will intermittently be used throughout the IOM Monitoring and Evaluation Guidelines, andrefers to the definition given above, including when discussing humanitarian context.IOM MONITORING AND EVALUATION GUIDELINES47

VO updatedModule 3: Monitoring at IOMTheory of Change (ToC)How to develop a ToC?Figure 3.2. Chain of results/Causal pathwaysChain of results / Causal yIllustrating a link between activities and results, creating a chain of results, referred to as the pathway of change, or the causal pathway.A common challenge when using the ToC is the lack of a “theory” and/or using a weak theory. Forinstance, weak theories do not explain how change is expected to occur or do not state/establishassumptions clearly. It is important to ensure that the ToC actually articulates a logical theory, whichmakes the implicit causal mechanisms explicit and supplements the graphical representation.By developing a valid and relevant ToC, implementers can ensure that their interventions will be deliveringthe appropriate activities for desired and realistic results. It ensures that interventions are easier tomonitor and evaluate, bring to scale, as well as sustain, as each step – from the ideas and assumptionsbehind it, to the results it hopes to achieve and resources required – are clearly articulated within thetheory. A well-articulated ToC can also promote a common understanding of the intervention for allactors involved in implementation, thereby facilitating a cohesive and common approach.The process of developing a ToC can help identify whether, and at which stage or level, assumptions,logical jumps or missing key steps in the change process are taking place. Developing a ToC is a good wayto raise further questions such as the following:(a)(b)(c)(d)Why is a particular change expected to happen?What evidence is available to support that expected change will/has occur/red?What logical jumps are made?What assumptions are made?When developing a ToC, it is important to understand its purpose. ToCs can be applied at different levels,ranging from world views, strategies and policies, to the project or programme level and all the waydown to activity level. For instance, world views can help clarify social and political theories that informone’s thinking. Organizational ToCs can help inform the vision, mission and values that the organizationrequires to contribute to social change. For policy ToCs, it can help identify how an organization expectschange to evolve in a specific sector and how it contributes to it.Scholars have not reached agreement on an overall definition of, and methodology for, developing a ToC,and donors may follow different approaches to drafting a ToC. For instance, the approach by the UnitedKingdom Foreign, Commonwealth and Development Office (FCDO) to drafting a ToC largely differsfrom that of the United States Agency for International Development (USAID) or that of the EuropeanUnion.11 While FCDO’s approach still contains strong elements of a logical-framework approach (lookingat inputs, outputs, outcomes and impact), USAID emphasizes the possibility of multiple pathways thatmay lead to change, while highlighting the importance of underlying assumptions throughout the process.The following depict these two different approaches:1148FCDO is formerly the Department for International Development (DFID).CHAPTER 3Monitoring at IOM

Figure 3.3. FCDO’s Theory of Change approachAnnex3InputsGBP 6m (included in theoverall financialcomponent)Management AgencySIDBI:PMU at New Delhi withGender specialistFCDO:Task Team Leader (0.5FTE);Economists (0.1 FTE);SD Advisor (0.5 FTE);Governance Advisor (02FTE);Programme Officer-A2L(0.5 FTE)Research/studies:FINSCOPE Survey;Cost benefit data/ Sectoralstudies; ProductdevelopmentOutputs 4: Women’s capacities to tackle genderconstraints relating to business and the householdenhancedActivitiesParticipationinsurance, pension - developed,piloted and rolled out based onwomen’s needs; capacity ofpartners enhancedGrant support to MFIs/SHPIs /NGOs/ training institutions throughNABARD, SFMC/similarinstitutionsProviding access to financialproducts and servicesIntegrating gender issues intheir microfinanceprogrammes,facilitate structured monthlydiscussions among the clientson social, gender and healthissuesFinancial literacy--Low cost approaches to building capacities ofwomen on financial and gender issues will beexplored during the operational phasePartner MFIs/SHPIs show commitment andwillingness to take forward the issue; gobeyond the assumption that simply organizingwomen into client groups addresses genderissuesWomen are able/willing to determine andprioritize social/ gender issues and needsOutcomesLong70% of theclients willbe poor orborderlinepoor (belowUSD2 a day)especiallywomenWomen gainself-esteemand selfconfidence0.3 mn women clientstrained/ made aware oftheir rights, including asconsumers of financialservices ona) Financial literacy andb) Social, health and genderissuesImpactPoor and vulnerablepeople, especiallywomen, benefitfrom economicgrowth in pooreststates, IndiaFinancial servicestargeted at women andsix specific productsrolled outFinancial products - savings,Assumptions and External Factors-ShortIncreaseddecisionmaking bywomen clientsWomen able totravel outside placeof residence withoutmale escort(50% increase overcontrol groups)35% decreasemalnourishmentrate amongstchildren below 5years over controlgroupsEVIDENCE Linking:Activities to short/ medium Outcome: Medium ‘Littlefield, Elizabeth, Jonathan Murdoch, and SyedHashemi, “Is microfinance an effectivestrategy to reach the millennium development goals?” Focus Note 24, Washington, D.C.:CGAP, 2003; Vaill, S., 2003, ‘More than Money: Strategies to Build Women’s EconomicPower, Impact Report No. 1: Economic Opportunity Initiative, The Global Fund for Women,San FranciscoPitt, M., Khandker, S. and Cartwright, J., 2006, ‘Empowering Women with Micro Finance:Evidence from Bangladesh’, Economic Development and Cultural Change, University ofChicago; Microfinance Programme Impact Assessment 2003, United Nations CapitalDevelopment Fund Based on Case Studies in Haiti, Kenya, Malawi and Nigeria:Outcome to Long-term Outcome/ Impact: Medium Kim, J. C. et al., 2007, ‘Understanding the Impact of a Microfinance-Based Intervention onWomen’s Empowerment and the Reduction of Intimate Partner Violence in South Africa’,American Journal of Public Health, vol. 97, no. 10, pp. 1794-1802 ; O'Rourke K, et.al.,“Impact of community organization of women on perinatal outcomes in rural Bolivia.” RevPanam Salud Publica 1998 3(1): 9-14.Manandhar DS, Osrin D, Shrestha BP, et al. “Eff ect of a participatory intervention withwomen’s groups on birth outcomes in Nepal: cluster-randomised controlled trial.”Lancet2004; 364: 970–79. Prasanta Tripathy, et al., “Effect of a participatoryintervention withwomen's groups on birth outcomesand maternal depression in Jharkhand and Orissa,India: a cluster-randomised controlled trial”; Lancet 2010: 375; Syed Hashemi, SidneySchuler, and Ann Riley, “Rural Credit Programs and Women’s Empowerment inBangladesh,” World Development24, no. 4 (1996): 635-53.Source: Vogel and Stephenson, 2012.Note: FCDO is formerly the Department for International Development (DFID).13IOM MONITORING AND EVALUATION GUIDELINES49

Figure 3.4. USAID’s Theory of Change approachSource: Kedzia, 2018.OURCEESSRIrrespective of how different stakeholders approach a ToC, they all have one commonality: they enablethe articulation of how, why and under what conditions a change is expected to occur within a specificcontext. While there is no one standard approach to developing a ToC, the following section illustrates aformula that can be applicable in most contexts and is commonly used by USAID to measure social andbehavioural change.Center for Theory of Changen.d.a. TOC examples.Kedzia, K.2018 Theory of change: It’s easier than you think. USAID Learning Lab, 13 March.Organisation for Economic Co-operation and Development (OECD)2019 Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use.OECD/Development Assistance Committee (DAC) Network on Development Evaluation.Vogel, I. and Z. Stephenson2012 Appendix 3: Examples of theories of change. FCDO, London.50CHAPTER 3Monitoring at IOM

3.3.4. How to develop a Theory of Change using the if-then-because formulaWhile ToCs can be illustrated in different ways, the logic of the chain of results, or causal pathway, canbe tested using if-then-because statements. In other words, it helps reveal assumptions that are “tested”through actions/activities, while assumptions play a central role in developing the ToC.Every step taken, from the overall objective of the intervention to each of its activities, has a ToC behindit that can explain and articulate the logical connections (or the different pathways) between the lowerlevel results, such as between outputs and outcomes, as well as between the higher-level results, such asthe outcomes and objectives.A common challenge when designing an intervention are logical leaps and gaps. There may be a disconnectbetween strong problem analysis and seemingly unrelated activities, with weak links and/or assumptionsbetween objectives, outcomes, outputs and activities. Through surfacing underlying assumptions, the ToCmay provide a bridge between analysis and programming.Generally, a ToC can be articulated using the “if X, then Y, because of Z” formula. That is, “if X action/activity occurs, then Y result will occur, because of Z assumption(s)”. The process of surfacing suchunderlying assumptions can help identify where logical jumps are made or helps identify missing key stepsin the change process.Figure 3.5. If-then-because formulaThe following section will focus on one of the many possible pathways illustrating the application ofthe if-then-because formula, noting that this exercise can be repeated for many different pathways fordifferent levels.IOM MONITORING AND EVALUATION GUIDELINES51

AEXM PL EFigure 3.6. Example of the if-then-because formulaIF relationships between the local authorities and conflict-pronecommunities in area Y of country X are strengthened, THENstability and security, and building foundations for political andsocial development in conflict-prone communities of country X willbe supported, because (assumptions):IF access to livelihoods opportunities for conflict-pronecommunities of country X are increased, THEN relationshipsbetween the local authorities and communities in area Y ofcountry X are strengthened, because (assumptions):IF vocational skills among beneficiaries in conflict-pronecommunities of country X are developed/enhanced, THEN theiraccess to livelihood opportunities will increase, becauseThe following elaborates on the example, identifying assumptions potentially surfaced through this processfor this one particular pathway down to the output level. As many different pathways can exist for eachlevel, this exercise can be done for each possible pathway.Ä Multiple pathways: During the process of identifying multiple pathways, it is important to note thatnot all pathways may be implemented by IOM, and that some of them can be implemented by actorsother than IOM, and out of IOM’s control.Objectives: Contribute to stability and security, and build a foundation for political and social developmentin conflict-prone communities of country X.Objective-level Theory of ChangeIf relationships between the local authorities and conflict-prone communities in area Y of country X arestrengthened, then stability and security, and building foundations for political and social development inconflict-prone communities of country X will be supported, because The relationship between the local authorities and conflict-prone communities is weak;The lack of attention from the local authorities towards the needs of conflict-prone communities hasaffected the relationship between them;The weakened relationship between the local authorities and the conflict-prone communities may beone of the causes for conflict;The lack of attention from the local authorities to the needs of conflict-prone communities is oneof the causes of instability and poor security, which impedes sociopolitical development in the area;Others.Ä Multiple pathways: With each result articulated at the outcome level, an if-then-because statement isarticulated, and the assumptions surfaced at the objective-level ToC.52CHAPTER 3Monitoring at IOM

Outcome-level Theory of ChangeIf access to livelihood opportunities for conflict-prone communities of country X is increased, thenrelationships between the local authorities and communities in area Y of country X are strengthened,because People consider the government responsive for meeting their needs;Targeted areas lack quality services from government;Target areas have historically been neglected;If people are grateful and appreciate the local, then they are more likely to perceive the authorities’service delivery more positively, which may lead to improving their mutual relationship;Others.Ä Multiple pathways: With each result articulated at the output level, an if-then-because statement isarticulated, and the assumptions surfaced at the outcome-level ToC.Output-level Theory of ChangeIf vocational skills among beneficiaries in conflict-prone communities of country X are developed/enhanced,then their access to livelihood opportunities will increase, because Improved vocational skills may increase the chance for beneficiaries to find a job;Capacities are relevant to the opportunities in the target areas;Opportunities exist in the area;Local government and line departments continue to provide support;Local government and line departments take ownership of the activity and provide follow-up supportto beneficiaries;Others.Ä Multiple pathways: With each result articulated at the activity level, an if-then-because statement isarticulated, and the assumptions surfaced at the output-level ToC.Additional examples of ToCs surfacing assumptions for multiple pathways can be found in the resourceslisted at the end of th

This chapter also focuses on remote management and monitoring, third-party monitoring (TPM), and explains how different monitoring elements come together to form an overall M&E plan, and finally, looks at monitoring and reporting on results. 3.1. An overview of how to monitor A strong project design is t