Rachel Hinton,1 Rona Bronwin2 and Laura Savage3
This article sets out a perspective from the United Kingdom’s Department for International Development (DFID) on the challenges of achieving research uptake. Two country case studies are presented from India and Uganda, which explore research projects under the Economic and Social Research Council ESRC-DFID-funded Raising Learning Outcomes programme. These case studies focus on relationships between the key stakeholders that enable policy debate relevant to the funded research. They are not a direct assessment of the impact that this research has had. Rather, this article explores the messy and iterative processes that DFID advisers are engaged in within the networks that they are embedded (and those that they are not), the way that they use partnerships to access evidence and promote it into policy debate, and the other drivers that matter. This article is important as a contribution to ongoing efforts to improve the quality and usage of education evidence in low-income contexts.
Keywords: education research, impact, partnership, policy, donors, international development, learning outcomes, DFID.
Demand for evidence in education is growing. Countries across the globe recognise that education has the potential to unleash talent and support wider development. In January 2019, the Education World Forum (EWF) in London saw education ministers gather to debate ‘what we should do with what we know: developing education policy for implementation impact and exponential success’ (Education World Forum 2019). The 93 ministers at the EWF were on a quest to identify policies and programmes that could be applied in their country contexts to improve learning outcomes for all children. Rigorous evidence on how to do this is limited. Yet even the evidence that we do have is not routinely informing education policy and programming activity (RISE 2015). While the relationship between evidence and policy is not simple or linear, we believe it could be stronger. We, the three authors of this article, aim to show here the ‘knowledge space’ that exists between research, policy, and practice in two countries: Uganda and India. Case studies have been selected on the basis that research grants under the Raising Learning Outcomes programme are live. As such, we set up the context and themes that the rest of this IDS Bulletin will revisit.
The authors are education advisers and members of the Education Research Team at the Department for International Development (DFID).4 Our team has three key objectives: to commission rigorous qualitative and quantitative research; to focus research calls on ‘what works’ rather than problem identification; and thirdly, to build a culture of evidence (generation and use) in education in low-income contexts. High-quality evidence underpins effective policy engagement and dialogue with partner countries, and robust sector planning. We are part of a wider Research and Evidence Division (RED), which works to make DFID more systematic in the use of evidence and thereby have greater development impact.
RED contributes new knowledge and evidence to DFID and the broader global development community, as a global public good. It does this through active engagement with policymakers, commissioning research on key questions in development, and by ensuring robust evaluation of DFID’s programmes. We work with the global research community to help produce guidance to support this mission through the Building Evidence in Education (BEE) global group that includes the World Bank, USAID, and United Nations agencies and foundations (see Hinton 2015; and Patrinos and Cross 2015; BE2 forthcoming, 2019). DFID also recognised the need for accelerated action to ensure that research and evaluation evidence informs policy and programming choices. With this mission, in 2013, a dedicated Evidence into Action team was established within DFID.
One significant programme that DFID initiated, to help build the body of evidence in education, was developed in partnership with the Economic and Social Research Council (ESRC). Started in 2014, this eight-year programme, Raising Learning Outcomes (RLO), aims to ‘improve the efficiency, effectiveness and equity of education systems to deliver learning outcomes at scale in developing countries’ (DFID 2018). DFID tracks the extent to which ‘evidence generated through the programme contributes to debates amongst policymakers and practitioners’ (ibid.). In order to achieve this, RLO research commissioning calls demanded (1) policy-relevant questions, (2) focus on large-scale interventions – ideally embedded within government, (3) mixed methods approaches to consider ‘why’ and ‘how’ alongside ‘what works’, and (4) a focus on gender and equity. Every applicant had to outline their intended ‘pathway to impact’, which have since been supported by an additional programme function, the Impact Initiative (organisers of this IDS Bulletin) to enhance the potential for research impact.
The global context has shifted. Donors, once significant players in terms of financing and sector budget support, may be on the margins of government sector planning. With this relative reduction in donor funding, governments are seeking evidence as a valued contribution from development assistance, a new ‘currency’ of development. DFID-funded research aims to support the highest standards in delivering this evidence and requires the use of rigorous and robust research methodologies, including adherence to internationally recognised ethical standards, with outputs disseminated openly following a robust process of quality assurance and/or peer review. ‘Research outreach’ refers to activities by researchers, their teams and implementers, or DFID staff to share findings of this research. A key rationale for working with the global community through BEE is to ensure both efficiency by jointly funding research, but critically to also provide greater coherence in the messages that are delivered to policymakers at a country level. By ‘research uptake’ and ‘research impact’, we mean that this research has contributed to debate about policy and practice (uptake) with the ultimate objective of improving development outcomes (impact). These definitions make clear some of the assumptions that underpin our analysis. We recognise that achieving impact is not a linear or inevitable pathway.
The remainder of this article is in three parts. First, we will present the theoretical framework which guided our interview approach and analysis of case studies. This explains that we believe that these knowledge systems rest on relationships. Second, we will present findings of two case studies before, third, drawing conclusions about actions that we and those we work with can take, to engage more meaningfully in the ‘knowledge space’. It is important to note that we are not judging or assessing the policy impact of specific RLO grants here. Nor are we attempting to analyse academic incentives to achieve impact on policy and/or practice, or trends within education research on low-income contexts (though some observations from our wider work may implicitly emerge). We are not empirically documenting country reality either: this is not a full-blown political economy or behavioural insights analysis of the two countries.
We draw upon our wider experience and key informant interviews with three sets of people – DFID country advisers, RLO researchers, and government officials – to present a snapshot of how research can inform thinking and action in-country. The quotes represent individual opinions rather than being representative of the stakeholder group. In so doing, we challenge our own thinking about the relationships that enable research uptake and a culture of evidence in education.
2 Theoretical framework
We have developed a conceptual framework based on our wider experience of international development both from within and outside of DFID, and work with the RLO programme. We draw on the body of literature that explores knowledge systems in low-income contexts (which is relatively small – most of this literature focuses on OECD contexts). This includes research that shows decisions made in partnership between politicians, researchers, and civil society are more likely to lead to positive outcomes (Oliver, Lorenc and Innvaer 2014). This has particularly been evidenced in relation to health and agriculture (Datta 2018), which has a longer history, spend, and supply of more robust research in developing contexts than education research. We also draw upon literature that shows that ‘the seemingly straightforward story of information supply, demand and use is complicated by user norms (how they prefer to make decisions), relationships (who they know and trust) and capacities (their confidence and capability to turn data into actionable insights)’ (Custer et al. 2018).
In every results framework for a research programme funded by DFID, you will likely find an indicator related to ‘policy uptake’ or ‘policy impact’. The results framework will set out what seems to be a linear theory of change: that commissioned research will, through activities organised by the research programme, reach education policymakers and practitioners. This new knowledge will convince these actors to change their decisions, thus resulting in real-world change and positive impact. We aim to show, through our conceptual framework and underpinning literature, that we do not believe this to be a straightforward linear process, but do believe that through better understanding and support to local knowledge systems, we can enable a greater impact of research on practice.
The increased funding of education research and demand coming from governments can shift debate and inform the development of policy. In other words, the process of research matters alongside the findings. Research is often viewed as the professional activity of generating knowledge and not enough value is placed on peer networking, communication, and policy relevance throughout the process (Gévaudan 2017). Policymakers can be sceptical of researchers advocating their own findings; rather, they value the rigorous assessment of the global body of evidence. We observe that the role of a resident ‘neutral’ adviser (a role which in some countries is or could be taken up by a DFID adviser, for example) can be an important driver to facilitate the inclusion of evidence as part of a conversation in a non-linear policymaking process over time. Part of the importance of building a ‘knowledge system’ is to create an environment that enables evidence use, informs debate, and leads to more questioning of how policies and programmes should be developed and implemented.
Our framework makes explicit both the informal and formal relationships between three key stakeholder groups (see Figure 1) that we consider important to the achievement of large-scale policy impact: donors, researchers, and government actors.5 We are not ignoring civil society, non-government organisers, the private sector, teachers, parents, and wider community members; rather, we are focusing on the existing dominant pillars and the relationships within and between these three groups. We reflect on the limitations of this approach in the conclusion. The metaphor of pillars as siloes aims to highlight the powerful incentives that inadvertently operate to privilege building intragroup relationships, rather than developing intergroup relations.6 Each stakeholder pillar has a unique incentive set that drives the use of time and resources towards intragroup communication and flows of knowledge. For members of each pillar, success primarily relates to their status and esteem within their respective communities.
In over-simplified terms, governments need to identify ‘what works’ to secure buy-in to deliver manifesto commitments and implement reforms in an electoral cycle. Ministers are incentivised to maintain their popularity and voting base of citizens by delivering results. Academics need to develop ideas, deliver their research activity, speak at global conferences, and publish in top journals.7 For bilateral donors, the drivers are to deliver the global goals; this demands that they secure resources for programmes. Success is measured by indicators that are often preassigned in results frameworks. With responsibility to show value for money to taxpayers, attention is on delivery of results. DFID country advisers may not have available time to deliver to centrally managed research programmes, where the ‘senior responsible officer’ (SRO)8 is not in their chain of management. However, DFID’s focus on professional development, and the logging of activity has historically been an important driver of research engagement. Increasingly, advisers are responding to government demands for policy advice; they recognise that evidence is their new currency of engagement.
Relationships between, and the user norms and incentives of these actors, are the subject of a much wider set of literature which raises bigger issues that go beyond the scope of this article. For instance, the donor–government relationship is addressed by an extensive literature on the politics, geopolitics, and economics of aid. However, little of this literature considers the role of aid in global knowledge production, or the role of evidence in the aid relationship. It is easy to apply common theories on two sides of the aid debate to evidence. On the one hand, donors can be seen as brokers and translators of knowledge (Lewis and Mosse 2006), enabling national (government, civil society, local actors’) ownership of the response to the evidence. Donors fund and present evidence on a particular policy-relevant question to governments or practitioners; governments choose to adopt this and amend policy accordingly or not. There is also literature that considers incentives for governments to demand and use research. This covers political and cultural reasons for demand/lack thereof, as well as capacity issues (capacity of individual policymakers, as well as the system they work within – such as limited internet connections) (Carden 2009; Newman, Fisher and Shaxson 2012; Carter et al. 2018).
Relationships between academics and government or donors are not covered by an extensive literature (Georgalakis et al. 2017), but there are again big issues here that we do not tackle in this article: the relative lack of funding to education research; a smaller field of education researchers; the extent to which researcher incentives to get tenure track and be published in top journals can jar with efforts to achieve research impact). We are not assessing whether research generated through co-creation between governments and researchers will be more likely to achieve impact (Boateng 2018); nor whether local researchers are essential members of the team for this pathway; we are taking the importance of this relationship as a grounding assumption.
We use a series of interviews to chart the nature of the intergroup relationships between government, donors, and researchers. Our informants, from each of these groups, highlight the messy reality of a pathway to impact. The interviews do not reveal a straightforward process to be achieved through an equation of having a local research partner, meeting a government official, publishing a blog, and presenting at a conference. Rather, they reveal the power of partnerships, in different forms, that enable the development of trust and potential for impact. Do governments feel research produced is relevant to their needs, or have the financial or human capacity to engage with the research findings? Given the proliferation of actors, are ‘evidence-based’ messages similarly proliferating? Is the donors’ focus on their ‘national interests’ and their ‘own’ programmes constraining capacity to act as knowledge brokers for wider evidence? Is the need for an academic to publish in a high-ranking journal a hindrance or a help in terms of research uptake? Do institutional incentives of each of the stakeholder groups work against collective evidence-based action? Does this matter: is there a sense that the knowledge space in these contexts is alive, and prompting evidence-based decisions?
The case studies were selected to illustrate a range of the challenges noted above. They are not in-depth studies that can provide wider generalisations; rather, they represent individual insights from across the three stakeholder groups. Many of the interviews were conducted by phone, not in the country context; thus statements were not confirmed by observations. These personal testimonies provide insights that indicate areas for further exploration, including the assumption embedded in the programme design that strong relationships between researchers and policymakers will be needed to drive future research uptake.
Uganda and India were both chosen as case studies due to the authors’ familiarity and experiences in these countries. The case studies provide insights into critical factors that both restrict and enable research uptake. For example, the Uganda case study highlights the tension between the time required to produce high-quality research, and the time to engage with policymakers, who were often difficult to access. Meanwhile, the case study from India shows the shifting nature of the donor–government relationship, and reduced DFID education spend.
The case studies prompt a broader question. As the balance of resources shift away from donors being a powerful player in terms of their financial aid, will their ‘seat at the table’ depend in future on their ability to bring evidence to support policy debate? The EWF with 93 ministers present showed the increasing demand for knowledge of ‘what works’? Is this an opportunity for more equal debate on how to shift learning outcomes and meet the global goals? Is evidence indeed the new currency of development?
3 Case study 1: pathways to impact in Uganda
3.1 Background to Uganda case study
The goal of Uganda’s National Development Plan II (2015/16–2019/20) is to reach middle-income status by 2020. Education is regarded as a government priority; however, this is not reflected in the commitment to education spend,9 which in Uganda is around half the global recommendation and half that of its East African neighbours. The global ‘learning crisis’ is apparent in Uganda, with only one out of ten children assessed in Primary 3 in Uganda able to read and comprehend a Primary 2-level story and correctly solve Primary 2-level arithmetical division (Uwezo 2015). The most recent Education Sector Review seems to suggest that national education priorities are still focused on increasing school infrastructure, rather than a commitment to increasing learning outcomes. The challenge of raising learning outcomes is further compounded by the school-aged population (6–18 years) expected to almost double between 2010 and 2025 to 20.6 million (World Bank Group 2015: 52).
DFID and ESRC’s RLO research programme has funded three projects in Uganda, two of which contributed to this case study to explore the role of partnerships for impact. The first research programme, led by Edward Seidman at New York University, sought to develop and validate an innovative, affordable, scalable, and practical tool for assessing teacher practices and classroom processes. It tested the tool, Teacher Instructional Practices and Processes System (TIPPS), in the context of Uganda, with the additional aim of identifying its potential for providing feedback to teachers. The second research programme, led by Rebecca Thornton from the University of Illinois, uses longitudinal data to identify whether, with the right combination of training, teaching, and learning materials, teachers can be supported to effectively teach literacy – even in rural, under-resourced, overcrowded classrooms. The study also explores economic approaches to implementation at scale to determine value-for-money impacts on pupil learning and teacher performance in African schools. Researchers from both projects based in Uganda shared their experiences in relation to pathways to impact, in addition to representatives from DFID Uganda and a member of the National Examination Board working closely with the Ugandan government.10
3.2 Uganda government–donor intergroup relations
The appetite and use for education evidence in the Government of Uganda is still emerging. For example, the 2017/2018–2019/2020 Education Sector Strategic Plan, developed by the Ministry of Education and Sport has little to no reference to evidence or education sector analysis in Uganda. According to an adviser in DFID Uganda, up to 90 per cent of the education budget is already allocated to recurrent expenditure, which leaves limited room for creative thinking and decision-making and perhaps limits the perceived relevance of research. The DFID adviser added that there have been a number of examples of push back based on political decisions being made in the face of strong evidence; for example, in response to evidence on public–private partnerships. In addition, there is a general impatience from the government over research pace and timelines, and therefore a resistance to spending the time needed to test what works. The DFID adviser stated:
There is limited appetite from government on using research and evidence to inform prioritisation and decision-making… As a way of being strategic, government list their priorities and respond opportunistically to offers that come.11
This opportunistic approach from the government suggests the importance that partnerships can play in being able to respond to government demand effectively.
There are, however, examples of evidence uptake. For example, in 2017, the Ministry of Finance held a Growth Research Conference, which led to the current narrative around the importance of human capital on meeting the growth agenda; a significant shift considering the priorities stated above around infrastructure. The Government of Uganda and the Local Development Partner Group has recently developed a new National Partnership Framework to illustrate areas of shared priority and to present thematic and key action areas to drive progress, which includes human capital development.12
In education, research commissioned through DFID’s regional research hub on early years repetition, demonstrated how evidence can be successfully used for advocacy. The commissioner and his policy team were engaged in all parts of the research from the tools analysis and dissemination, which has since informed their thinking on early years. Ongoing research, led from UC Berkley on teacher attendance and teacher transfers, are good examples of government collaboration in evidence generation and application. A Southern-based co-investigator researcher commented that the interest in this particular research may be due to a number of reasons, including: the issues aligning directly with government challenges and priorities; strong partnerships between Northern and Ugandan researchers; Ugandan researchers having close links to the governments – for example, through having worked at the Ugandan National Examinations board; and involved collaboration with the government from the design phase. This again seems to suggest the value of having strong research expertise and partnership embedded in-country as well as the value of utilising those who have existing experience of collaborating with government.
3.3 Uganda researcher–government intergroup relations
The Prime Minister’s Delivery Unit has identified four priority sectors, of which education is one. With support from a DFID Uganda-funded programme (Strengthening Evidenced Based Decision Making in Uganda), the Office of the Prime Minister both commissions research and requires that all departments and ministries have monitoring and evaluation working groups. Some working groups are more active than others but according to researchers in Uganda, these working groups are hard to access, navigate, and negotiate. They are often reserved for the bigger multilateral partners and not necessarily those researchers who are not in a partnership or partnered with a small non-governmental organisation (NGO). However, the power of partnership with NGOs as part of the pathway to respond to evidence or influence policy is exemplified by Thornton’s research project partnered with the Ugandan NGO Mango Tree. The NGO, started in 2009, was incubated in context and already had informal partnerships with national and local government. The NGO has made small financial contributions to government initiatives to actively show reciprocal support, as well as to use relationships to build awareness. This has led to more formal integration of subsequent research and thinking, as well as enabling research to better respond and adapt to policy windows.
The classroom observation research, on the other hand, explored partnering with an NGO; however, they found this was difficult and ultimately not possible due to the mismatched incentives, timelines, and ways of working of researchers and the NGO, especially due to the conditions required for rigorous randomised controlled trials (RCTs). Again, this seems to come back to the value of relationships, where relationships are built over time, informally and formally, and viewed as two-way, not only between the NGO and researchers but also between the NGO–research partnership and government. The Mango Tree researcher concluded that ‘the researcher–NGO partnership can support national interventions and engage with government public activities… which goes a long way in building partnership’.13
The opportunities utilised by the Mango Tree research partnership with government does seem to support the researcher’s observation that there is a growing capacity and capability within those in government to recognise rigorous evidence and understand the role evidence and research can play. However, there are also still strong incentives and benefits for the government to engage with consultants over rigorous research. Therefore, the value of building and fostering partnerships and relationships over time seems to play a significant role in building the demand and use of evidence by government.
3.4 Uganda researcher–donor intergroup relations
Researchers funded by DFID central programmes can have stronger links to DFID headquarters than with the DFID country adviser, even if as in this case, they are based in-country. This is perhaps primarily due to large-scale education research programmes being commissioned out of the UK and therefore researchers and DFID country advisers not necessarily having opportunities to form formal relationships, therefore missing opportunities to identify benefits of partnerships or closer involvement. One researcher14 based in Uganda challenged that ‘funders need to be more aware of who is doing what where, including investments in research in-country and centrally’. However, in contexts where there is often only one education adviser managing a large education portfolio in-country, DFID advisers do not necessarily self-identify as knowledge brokers, as is the case in Uganda. Engaging in research and with researchers is perhaps viewed as a luxury and a nice thing to have. Therefore, the strength of this relationship with the local country DFID office seems to vary between individual researchers.
Clear examples of evidence informing education policy and programming seem to stem from where there are existing relationships, including informal relationships. DFID Uganda’s existing relationship with Mango Tree researchers meant that when the research gained traction with government, DFID used the momentum to inform the design of the DFID education programme and challenge current government thinking on the language of instruction in primary education. In the case of the classroom observation research, there were no existing relationships with DFID in-country and therefore the absence of informal connections seemed to limit the opportunity for researchers to engage more formally with DFID in-country, and potentially limit further opportunities for partnership and impact. The potential for this researcher–donor relationship to impact policy or practice does, it seems, depend on timing and ability to act during windows of opportunity provided by design phases and strategic plans. Therefore, existing relationships, informal or formal, are essential to enable partnership and research impact.
All researchers interviewed reflected that there are two key requirements for impact in Uganda. First, partnerships for impact work best when research teams are embedded in countries full-time, and are known to partners, donors, and governments and are part of key working groups and technical groups. Second, there is a need to redefine what it means to be a research partner: not just responsible for the production of evidence, but a need to actively engage with policy too. DFID could potentially support the role of partnerships for impact by building in more rigorous requirements, incentives, and expectations when commissioning research, to ensure in this case, closer collaboration at all stages of research with Ugandan researchers and Ugandan decision makers.
4 Pathways to impact in India15
4.1 Background to India case study
India is the fastest growing major economy in the world, projected to grow at 8.2 per cent per year over the next five years. Yet it is also home to the largest absolute number of people living in extreme poverty in the world. Education is seen by many as the way out of poverty, and a means to overcome entrenched social exclusion. The Department for School Education and Literacy develops national policy, including Sarva Shiksa Abhiyan (SSA or ‘Education for All’). Responsibility for monitoring educational and administrative activities of schools rests primarily with district and block education officers, through village education committees, which include community members. Education spending by central government is steadily rising, and accounts for 3.7 per cent of proposed budget expenditure in 2017–18 (Ministry of Finance 2017). It is a key political priority for the government.
This case study draws on two research projects:16 Ricardo Sabates at the University of Cambridge, investigating the potential of community-based accountability relationships to raise children’s learning outcomes and Karthik Muralidharan’s research at the University of California, which used an RCT to test the impact of Madhya Pradesh’s (MP) School Quality Assessment programme on school governance and learning outcomes as the programme went to scale across the state. Abhijeet Singh from the Stockholm School of Economics was interviewed as a co-investigator on the MP research. Colin Bangay, DFID’s education adviser in India during both projects’ design phases was also interviewed. The DFID India office no longer has an education adviser, although the senior adviser leading on skills and business engagement contributes, where time allows, to policy debate for the sector.
4.2 India government–donor intergroup relations
The DFID adviser interviewed revealed a strong understanding of India’s complex system of government, where central government comprises over 90 ministries, leading to challenges for policy coordination between ministries and between the central state and local governments. The cadre of professional civil servants are recruited to take a range of leadership roles across different ministries during their career. These officers are the educated elites, many of whom have received Western education. Respect for evidence-based policymaking was mutual between the donors and government. Indeed, the adviser to the minister actively sought to engage with the latest international research, to identify the best route to implement change and deliver for a reformist government. Bangay’s professional knowledge of global evidence was valued. Thus, he was able to build trust over time through responsive sharing of data that responded to the needs of policymakers. One example was his commissioning of a review into activity-based learning (DFID 2016). This research fitted with India’s 2009 Right to Education Act which mandated child-centred and child-friendly education.
However, Indian officials can exhibit a degree of scepticism of outsiders and their agendas, be they donors or consultants. The well-known ‘seagull’ metaphor is used of experts flying in, making a lot of noise, and flying off leaving mounds of guano [reports] but few practical ways forward.17
The psychology of presenting evidence is missing… research is seen to be driven by external agendas, and this can, sometimes, affect trust in such research… I think even if evidence comes from donors the initial government reaction is to be defensive and to reject the research. (Colin Bangay)
Bangay argued that uptake of the evidence will remain a challenge as long as investments in critical relationships are neglected.
The importance of the DFID adviser being resident and building a shared history of engagement was highlighted as significant in building trust. However, deeper relationships were often held by DFID’s professional advisers appointed in-country, and their knowledge of the cultural context enabled more effective engagement politically. Bangay worked in close partnership with the local advisers, and benefited from their social capital. His engagement in government processes also demonstrated commitment beyond his own programme: the establishment of such informal relationships seems to have been key to open up space for policy dialogue.
Bangay stressed the importance of a realistic understanding of the pressures and constraints government policymakers are subject to (see Bangay and Little 2015):
Policymakers are not experts. They are time poor and they want surety and not probability. I think there is a mismatch between the culture of research and researchers and what policymakers are after. You need to repackage research for policymakers, otherwise they won’t look at it. The two groups are thinking at two different levels.
When sharing research findings, two factors were revealed as significant: timing and nuancing. First, the ‘communicator’ of evidence needs to be available in-country and actively identifying key moments when evidence might influence critical decisions. Second, insight into the specific preferences of individual policymakers enables a nuanced adjustment in how messages are communicated. Who is best placed to communicate research was also carefully considered. Bangay gives an example of ‘low-key’ research on girls’ access to education which was received well, despite it demonstrating unintended flaws in current government policy: ‘The government was not particularly fond of this research, but they engaged with it… [W]e took a low-key approach and presented the results through the local research teams.’18 He explained how this contrasted to most research being ‘sold’ to the government by ‘expert middle-aged white men’.
4.3 India researcher–donor intergroup relations
Bangay suggested that there is more that donors could do to support access between academics and policymakers. One researcher noted: ‘we tried repeatedly to get a meeting with the donor group, but our requests were not considered to be a priority, and as an external academic there is limited time to meet with those taking decisions in government’.19 Advisers are being overwhelmed with the number of research initiatives at a country level and need to make strategic decisions about which to engage with. It appears that researchers may lack the initial connection with the relevant donor, and time restrictions on both sides act as a barrier to interaction. Financial resources were mentioned as a constraint on several occasions: ‘The donors fund research but they don’t always invest in the use or uptake of that research. This is a shame because, the people who will use or ignore research are policymakers’ (Colin Bangay). In a meeting organised by the RLO team, several of the research teams bid for resources in a ‘dragons’ den’ and were highly articulate around the need for funding for uptake.20
There is a spectrum of engagement in research by advisers, often associated with their own academic background and interest. Bangay ensured uptake of new research being produced, through international fora, engagement with DFID’s own education research team, and policymakers. For example, he worked closely with the CREATE researchers21 and linked to Muralidharan’s research that built on a body of work in India (see Kingdon and Banerji 2009, and Chavan and Banerji 2012) that pointed towards two major bottlenecks to improving learning outcomes. Firstly, high levels of teacher absenteeism and little time spent on task leading, linked to poor governance and accountability mechanisms. Secondly, pedagogy that revolves around rote learning and progression dictated by the curriculum without adaptation to learners’ specific needs. Bangay was able to use this knowledge in discussions with policymakers in India and the UK, to shape new research directions and policy for DFID centrally.
Abhijeet Singh22 explained the importance for him of engaging with the donor in order to understand what DFID was looking to achieve from their investment in research and DFID’s role of broker between researchers, government, and non-profit partners: ‘Colin [Bangay] was the person brokering this relation between Ark, the government, and non-profit consultants’. However, given DFID’s move out of India ‘getting traction with the country office was quite hard’. Despite this, Singh persevered in his engagement with DFID. ‘There also seemed to be other venues to talk to [a] DFID audience’ he noted, ‘such as the Research on Improving Systems of Education (RISE).’ Sabates23 also noted that ‘currently we have not had many opportunities to engage with DFID country advisers’.
4.4 India government–researcher intergroup relations
Researchers Muralidharan and Singh also succeeded in building trust with senior policymakers.24 Singh notes the positive effects of Muralidharan’s willingness to offer regular expert advice to officials beyond the scope of his current research projects. His global research reputation resulted in Muralidharan becoming an honorary adviser to the National Institute for Transforming India (NITI Aayog), which is a key space for policy influence. Singh stressed the importance of his colleague Karthik Muralidharan being a credible authority on economics and education by government:
Karthik’s work gives openings for collaborations and conversations with officials because they saw the evidence [from him] before.
The incentives for Indian policymakers at the central level to engage with research were strong. They were keen to have a media presence at key events and would accept opportunities to speak, particularly with high-profile external academics.25 The fact that both research teams’ focus of study aligned with ministerial priorities made space for interaction with government easier. The political context and practicability of implementation are also crucial. As Singh noted:
When you tell a policymaker that your education results are bad and they will be bad for the next 200 years, you won’t be incentivising them to work for a change. For that to happen you need to give them actionable steps.
In contrast to the top tiers of the civil service, there is wide variation in the skills and motivation of frontline staff. District education officials may be without support on IT, finance, procurement, and project management. This creates a challenge for the fidelity of research. The relationships between donors and governments at this level of government were more limited. In contrast, local research teams were often well embedded, making responsive policy debate possible. Ricardo Sabates stresses the importance of local expertise for his programme:
Pratham [an Indian NGO] have been engaged from the beginning, thinking together about research questions, research design and the experimental mixed methods… I think it is a very equal partnership. I know I am the principal investigator of this project. However, I think this is misleading. There are two PIs, one in Cambridge and one at Pratham.
The embeddedness of local research teams was impressive and clearly is crucial to assess whether failures of policy implementation, rather than the intervention per se, were resulting in the lack of impact on learning. Genuine debate with government about implementation failures and the robustness of the research further deepened the government–researcher relationship.
Both research teams noted that time to support government officials outside the remit of the research was important, arguing that this was key to building the social capital and trust required when it came to dissemination of their own findings:
Once the collaboration is going, you need some easy way of communication… you can’t have anything going on unless you have a [local] team in place… You signal your commitment (for a long-term relationship) by showing a constant stream of work. (Singh)
This was particularly important for overcoming problems associated with personnel changes in ministries: ‘During four years of work, we saw three secretaries and one interim secretary change, so the only institutional memory was the written documents we have provided’ (Singh).
Researchers from Sabates’ research project highlight the ability of local research to make a significant impact. Purnima Ramanujan26 at the ASER Centre, Pratham states that:
Over the last decade, there has been more demand for and recognition of evidence in education… More and more, we find that the government, besides NGOs, are getting into more research and studies looking at how to improve learning in schools.
Sabates highlights the work of the Annual Status of Education Report as an example of citizen-led research making policy impacts: ‘It has been informing policy in India and informing the learning crisis globally. It has had an incredible impact in terms of how policymakers are responding to what they see in terms of learning levels in schools.’27 One result of this, according to Sabates, has been the adoption by six Indian states of Teaching at the Right Level Methodology.
The case studies highlight three critical factors that are common to each context: understanding the political dynamics of research uptake; a culture of evidence-informed policy; and the importance of supercommunicators, who are trusted by government, to disseminate evidence. These factors are discussed in relation to the commonalities and differences between the Uganda and India case studies.
5.1 Understanding the political dynamics of research uptake
The case studies highlight the attention paid by impactful researchers and advisers to the cycles of policymaking. A common characteristic was their political intelligence and connections to the inner machinery of government, with their tentacles in the system. This was seen in the Indian case study where the user norms were understood by the researcher who had an advisory role at the centre of government. The ability to identify champions of policy change and flag the timings for influence was also crucial. It was key in the way Sabates, as the international principal investigator, worked in joint partnership with Pratham, recognising their long-established relationships and cultural knowledge.
In both case studies, effective evidence engagement did not conform to the pre-designated timings of communication plans. Rather, agile ways of working and political intelligence were key, with a recognition that opportunities for bold reform are fleeting. It was during times of disruption to the system – such as new political appointments – that windows of opportunity arose. Interviews with DFID advisers revealed the constraints they are under. Despite this, the advisers interviewed sought out evidence to equip themselves as technical professionals. Some had formed alliances with local academics. Their reflections indicate that, with increasing external private philanthropy in the sector, financial aid was becoming less important than evidence as a ‘currency’ valued at the policy table.
5.2 A culture of evidence-informed policy
The case studies revealed the importance of a culture of evidence-informed policy in government. Where there were ‘evidence champions’ within government, they were key to the development of strong government–donor and government–researcher intergroup relations that enabled effective knowledge exchange. This culture was most developed in the Indian case where the demand for data was coming directly from the ministry, rather than topics for dissemination being externally driven. If research was perceived as helpful to deliver more effective services, so opportunities for uptake increased. Established relationships between the government and donor or government and researchers opened space for dialogue and dissemination of evidence. The researchers and advisers in both cases were aware of the importance of internal champions to maximise the opportunities for research uptake. Interviews also highlighted that government demand for evidence led to a proactive approach by the donor to respond, and in turn to more actively develop researcher–donor intergoup relations.
5.3 Supercommunicators to disseminate evidence
Interestingly, there was variation across the countries and case studies as to the key stakeholders feeding evidence into policy decisions. Such people might be called ‘supercommunicators’. Whether they are a local academic, an international ‘expert’, a donor, or a multilateral partner, the critical factor was trust. We introduce the term supercommunicator to describe a good communicator who has additionally established trust with key policymakers. These are the people who have developed social and political capital, enabling them to influence most effectively. It was clear that the ‘right’ supercommunicator at one point in time or level of government, may be different from another. Understanding the drivers of change to help select the ‘supercommunicator’ and careful attention to the choice of messenger was key, as discussed by Bangay in the India case study. Rather than it being a reluctance on the part of the academics to be ‘supercommunicators’, the challenge was often the lack of an established relationship with the donor or central government for policy dialogue. Interviews revealed a donor tendency to put the responsibility on the academic to organise government policy engagement, rather than recognising their potential role as a broker.
5.4 The challenges for research impact
We close with four challenges that might be kept in mind while reading other articles in this IDS Bulletin, and that we at DFID take away from this study as actions to consider in our own engagement to encourage impact from research.
Civil society has a unique reach, and networks that are valuable in the research impact pathways. These are networks that advisers are not necessarily embedded within. This was a common factor across the case studies. It raises the question of whether civil society voices are being sufficiently heard at the policy table. Is there, for example, a subconscious privileging by both donors and governments of the ‘expert’ external voice? Could NGOs and civil society be further supported as a voice in the pathways to impact? This has implications for our theoretical framework and prompts us to reconsider how we engage with NGOs, particularly as the sector grapples with the implementation challenge of taking cost-effective innovations to scale.
Drivers to develop intragroup relations will always remain strong, yet the interviews have revealed some powerful intergroup relations that have enabled research to influence policy and practice. The pillars of policy impact need not be siloed. We need to dig deeper to understand how to further develop these intergroup incentives.
Research impact does not emerge from a ‘pathway to impact’ plan. DFID can play a role in supporting agile ways of working, in sharing political knowledge of key moments for change, and being a broker for researchers requesting links with policymakers where those are not already established. For researchers, there is a tension between the time required to produce high-quality research, and time to engage with policy actors, which is insufficiently appreciated, and thus insufficiently budgeted for. Do donors have more of a role to play in supporting this process?
With increased programme delivery demands, DFID education advisers’ time is increasingly constrained to take evidence into action. There is less time for thought leadership and advice on sector reform. Rather, advisers need to invest in oversight and advisory input to the individual programmes that they manage. The skill set and confidence of DFID country advisers interviewed was a critical factor in the time they made to engage with research. The length of their posting and historic engagement in-country were important. This matters not only for the trust and depth of relationships with researchers and government, but also their ability to justify research engagement with their line management. These challenges have implications for DFID’s own operating model, particularly because governments are requesting knowledge, rather than money as the ‘new currency of development’. Ministries of education are seeking to support both the production and the consumption of research, as partners in the global dialogue of how to enhance learning.
* This issue grew out of the Impact Initiative for International Development Research which seeks to maximise impact and learning from ESRC-DFID’s Joint Fund for Poverty Alleviation Research and their Raising Learning Outcomes in Education Systems Research Programme.
✝ Views expressed in this article are those of the authors and do not reflect an official DFID position.
1 Department for International Development (DFID), UK.
2 Department for International Development (DFID), UK.
3 Department for International Development (DFID), UK.
4 We wish to acknowledge support from Kate Ross for editing and with a review of the literature.
5 We recognise the key role of civil society, NGOs, the private sector, teachers, parents, and wider community members on research uptake. Further attention on their role is warranted and deserves to be the focus of more serious debate.
6 Thanks are due to the academic contributors to this issue for their discussions and in particular to Mark Henstridge from Oxford Policy Management for early debate on the competing incentives of academics and donors.
7 Defined as reflecting ‘the place of a journal within its field, the relative difficulty of publishing in that journal, and the prestige associated with it’ https://en.wikipedia.org/wiki/Journal_ranking (accessed 11 February 2019).
8 This is the staff member responsible for the oversight, finances, and ultimate delivery of a programme.
9 The Muscat Agreement (March 2014) recommends that government spends between 4 per cent and 6 per cent of gross domestic product (GDP) on education and between 15 per cent and 20 per cent of public expenditure on education. Uganda’s education share of national income is 3.2 per cent of GDP, half of what Kenya and Tanzania spend.
10 Quotes are based on interviews with Rona Bronwin, between October and November 2018.
11 Interview, February 2019.
12 The collective skills, knowledge, or other intangible assets of individuals that can be used to create economic value for the individuals, their employers, or their community.
13 Interview, September 2018.
14 Interview, September 2018.
15 We wish to acknowledge the inputs and research provided by Josh Lowe, Blavatnik School of Government, in the production of this case study and to Salim Salamah for support with interviews.
16 Quotes are based on interviews with Rachel Hinton and/or Salim Salamah, between December 2018 and January 2019.
17 Thanks to Robin Todd, Cambridge Education, for discussions on the African context of government perceptions of donors.
18 Interview, February 2019.
19 Interview, September 2018.
20 ESRC-DFID grant holder workshop ‘Knowledge Synthesis, Policy Directions, and Implications for Impact’, Oxford, January 2019.
21 See www.create-rpc.org.
22 Quotes from Abhijeet Singh interview with Rachel Hinton, 13 December 2018.
23 Interview, February 2019.
24 Quotes from Abhijeet Singh and Karthik Muralidharan interview with Rachel Hinton, 13 December 2018.
25 This was evident at a strong speech by the minister in Delhi on the importance of research. Ministers are keen to demonstrate that they have delivered manifesto commitments.
26 Interview, February 2019.
27 Interview, February 2019.
Bangay, C. and Little, E. (2015) Researchers Ask Questions and Policymakers Want Answers: How Can Both Do Better?, HEART Blog, 9 December, www.heart-resources.org/blog/researchers-asboth-do-better/ (accessed 25 February 2019)
Boateng, P. (2018) Think Tanks’ Influence on Education Policymaking in Low-Income Contexts, K4D Helpdesk Report 406, Brighton: IDS
Building Evidence in Education (BE2) (forthcoming, 2019) ‘Guidance Note on Measuring Costs of Donor-Funded Education Programming’, Washington DC: USAID
Carden, F. (2009) Knowledge to Policy: Making the Most of Development Research, Ottawa ON: International Development Research Centre
Carter, S.; Katticaran, J.B.; Macias, C. and Walsh, C. (2018) What Have We Learned about Building a Culture of Data and Evidence Use in Government?, J-PAL blog, 2 May, www.povertyactionlab.org/blog/2-5-18/what-have-we-learned-about-building-culture-data-and-evidence-use-government (accessed 1 March 2019)
Chavan, M. and Banerji, R. (2012) ‘The Challenge of Achieving Desirable Levels of Learning in Elementary Education’, Journal of Governance 5: 33–46, www.iccfg.net/files/specialissuesofjournal/Volume%205%20%20-%20July%202012%20-%20Special%20Issue%20on%20Education.pdf (accessed 1 May 2019)
Custer, S.; King, E.M.; Atinc, T.M.; Read, L. and Sethi, T. (2018) 6 Key Insights into the Data and Information Education Leaders Want Most, Education plus Development blog, 20 February, www.brookings.edu/blog/education-plus-development/2018/02/20/6-key-insights-into-the-data-and-information-education-leaders-want-most/ (accessed 2 May 2019)
Datta, A. (2018) Strengthening Research Systems: Concepts, Actions and Actors, K4D Helpdesk Report, Brighton: IDS
DFID (2018) DFID-ESRC Joint Research Fund on Raising Learning Outcomes (RLO), Annual Review 2018, London: DFID, https://devtracker.dfid.gov.uk/projects/GB-1-204324/documents (accessed 22 May 2019)
DFID (2016) Activity Based Learning (ABL): An Evaluation of the Pedagogy, Impact on Learning Outcomes, Political Economy of Adaptation and Subsequent Scale-Up of the Programme in Tamil Nadu, India, London: DFID, https://assets.publishing.service.gov.uk/media/58db95a3e5274a06b0000046/Report_1.pdf (accessed 1 March 2019)
Education World Forum (2019) www.theewf.org (accessed 21 February 2019)
Georgalakis, J.; Jessani, N.; Oronje, R. and Ramalingam, B. (eds) (2017) The Social Realities of Knowledge for Development, Brighton: IDS/Impact Initiative
Gévaudan, C. (2017) Doing Research Assessments: Understanding Research Systems in Developing Countries, New Delhi: Global Development Network, www.gdn.int/sites/default/files/GDN-2017-DRA-method.pdf (accessed 29 March 2019)
Hinton, R. (2015) Assessing the Strength of Evidence in the Education Sector, report prepared for Building Evidence in Education (BE2), London: DFID
Kingdon, G. and Banerji, R. (2009) ‘Addressing School Quality: Some Policy Pointers from Rural North India’, RECOUP Policy Brief 5, Cambridge: Research Consortium on Educational Outcomes and Poverty, University of Cambridge
Lewis, L. and Mosse, D. (eds) (2006) Development Brokers and Translators: The Ethnography of Aid and Agencies, Bloomfield CT: Kumarian Press
Ministry of Finance, Government of India (2017) Union Budget 2017–18, New Delhi: Government of India, www.indiabudget.gov.in/budget2017-2018/budget.asp (accessed 1 March 2019)
Newman, K.; Fisher, C. and Shaxson, L. (2012) ‘Stimulating Demand for Research Evidence: What Role for Capacity‐Building?’, IDS Bulletin 43.5: 17–24, https://bulletin.ids.ac.uk/idsbo/article/view/277 (accessed 1 March 2019)
Oliver K.; Lorenc T. and Innvaer, S. (2014) ‘New Directions in Evidence-Based Policy Research: A Critical Analysis of the Literature’, Health Research Policy and Systems 12: 34
Patrinos, H. and Cross, J. (2015) Generating Evidence in Education – Impact Evaluations, report prepared for Building Evidence in Education (BE2), Washington DC: World Bank
RISE (2015) Vision Document 2: Ambitious Learning Goals Need Audacious New Approaches, Research on Improving Systems of Education (RISE) Vision Document 2, www.riseprogramme.org/publications/vision-document-2-ambitious-learning-goals-need-audacious-new-approaches (accessed 1 March 2019)
Uwezo (2015) Are Our Children Learning? Five Stories on the State of Education in Uganda in 2015 and Beyond, Fifth Uwezo Annual Learning Assessment Report, Kampala: Twaweza East Africa, www.uwezo.net/wp-content/uploads/2016/06/Uwezo2014-ALA-UG-FINAL-web.pdf (accessed 1 May 2019)
World Bank Group (2015) Systematic Country Diagnostic: Boosting Inclusive Growth and Accelerating Poverty Reduction in Uganda, Washington DC: World Bank, http://documents.worldbank.org/curated/en/401861467998756667/pdf/97145-CSD-P151913-IDA-SecM2015-0247-IFC-SecM2015-0173-MIGA-SecM2015-0119-Box393266B-OUO-9.pdf (accessed 1 May 2019)
© 2019 The Authors. IDS Bulletin © Institute of Development Studies | DOI: 10.19088/1968-2019.105
This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International licence (CC BY), which permits unrestricted use, distribution, and reproduction in any medium, provided the original authors and source are credited and any modifications or adaptations are indicated. http://creativecommons.org/licenses/by/4.0/legalcode
The IDS Bulletin is published by Institute of Development Studies, Library Road, Brighton BN1 9RE, UK This article is part of IDS Bulletin Vol. 50 No. 1 May 2019 ‘Exploring Research–Policy Partnerships in International Development’; the Introduction is also recommended reading.