Evaluating Foreign Policy Measures in Fragile Contexts

Executive Summary

Modern foreign policy increasingly pursues its goals by using specific financial resources for concrete projects. In Germany, the budget lines for peacebuilding and stabilization, which includes humanitarian action, have grown threefold over the past decade. The design, tendering, supervision, monitoring and evaluation (M&E) of projects are thus becoming important elements of modern diplomatic craftsmanship. This study analyzes the international experience with the monitoring and evaluation of foreign policy projects in fragile contexts. It focuses on the fields of stabilization, humanitarian action, civilian crisis prevention and democratic transformation, by way of the experience of the Netherlands, Switzerland, the United Kingdom and the United States. 

Are There Specific Challenges for Monitoring and Evaluation of Foreign Policy Projects in Fragile Contexts?

A close look at the recent practice of M&E for foreign policy projects offers numerous opportunities for learning and for the improvement of project work in fragile contexts. However, these opportunities cannot be realized by using classical approaches to evaluation that follow a strictly positivistic concept of science. This is partly because such approaches fail to provide adequate methods and criteria to match the specific challenges of foreign policy projects in fragile contexts. One key challenge is posed by the combination of political-procedural goals (eg, increasing trust in state institutions) with sectoral ones (e.g., expanding education), with the latter often being subordinate. Another challenge is the dynamic nature of fragile contexts, which most causal-analytical approaches to M&E cannot properly account for. Where these traditional approaches fail, widespread disappointment with evaluation findings is the result. As a reaction, new and innovative approaches to M&E are increasingly emerging. These newer approaches are better tailored to the learning and accountability requirements of foreign policy projects; appropriate criteria and methods are consciously chosen for their specific utility to the challenges at hand.

Leveraging M&E for Learning in Foreign Policy Projects 

The study identified a disparate set of approaches to continuously improve foreign policy projects in fragile contexts. This disparity reflects that foreign ministries are still experimenting with how to integrate M&E in the processes and organizational traditions of foreign policy. Among the most promising examples are: participatory evaluation procedures used in the Netherlands, which aim to effectively combine learning and accountability requirements; informal peer review mechanisms used for program design in Switzerland; innovative approaches to monitoring in the United States; as well as the constructively challenging culture of learning and the strong emphasis on research in the United Kingdom. Overall, the diversity of widely practiced approaches, criteria and methods shows that there is no strict requirement to comply with any one narrowly defined approach.

The OECD Development Assistance Committee’s evaluation criteria (ie, relevance, effectiveness, efficiency, impact and sustainability) are widely used but not always useful for the evaluation of foreign policy projects in fragile contexts. It has internationally become common practice to adapt them or to develop individual criteria that capture the specific challenges of fragile contexts. On the methodological level, more recent perceptive or survey-based methods of data collection are thus taking on ever greater importance.

In the four countries under examination, we observed a tension between the two central goals of M&E: learning and accountability. Accountability-oriented evaluation usually is designed in an investigative, confrontational style and thus leads to considerable friction. At the same time, successful learning tools are often too informal in terms of methodology and process to be of any use for the purpose of accountability, because, for example, they fail to produce publishable outcomes. However, effective combinations of learning-focused and accountability-focused approaches are possible, if the emphasis is clearly placed on one or the other.

How to Design M&E Systems for Foreign Policy Projects in Fragile Contexts?

Developing a specific M&E system for foreign policy projects in fragile contexts encounters several challenges. Beyond the need to adapt criteria, procedures and methods, evaluation projects face substantial risks to achieve high-quality results. These quality issues are rooted, in part, in the structure of markets and incentives for evaluation consultants. However, another crucial reason for the success or failure of any evaluation lies in the quality of the terms of reference: For evaluation consultants to be able to achieve fair and useful results, a substantial amount of effort needs to be put into the formulation of evaluation questions, the application of success or evaluation criteria to the subject of evaluation and the management of the evaluation process. Therefore, the role of the evaluation management is essential. Furthermore, an effective M&E system benefits from professional project management and from a constructive culture of dealing with mistakes and conflicts.

All this requires time and money. Appropriate M&E instruments need to be developed, piloted and carefully introduced. After all, institutionalizing M&E is only going to have a chance of success if it comes with the possibility of adding personnel and investing resources in order to improve projects and their implementation.

The full policy paper (in German) is available for download.

A German version of this report’s executive summary is available.