Untitled Document

Learning About Integrated Development Using Longitudinal Mixed Methods Programme Evaluation*

Emily Namey,1 Lisa C. Laumann2 and Annette N. Brown3

Abstract Globally, millions of children are living without parental care. Families experience multiple challenges that lead to family–child separation, with financial hardship a common theme. The integration of interventions to strengthen families socially, emotionally, and economically is therefore a priority, but requires knowing which combination(s) of interventions might work for which households. This article reflects on an ongoing evaluation of two projects in Uganda that implemented integrated family and economic strengthening interventions with families at risk of a child being separated and those where a separated child was reunited. We discuss how the practical realities of integrated programming influence research design options and our attempts to mitigate the potential limitations of a non-experimental design with the use of a mixed methods approach.

Keywords: evaluation, integrated development, child protection, economic strengthening, family strengthening, separated children, reintegration, mixed methods.

1 Background
1.1 Description of the problem
Millions of children around the world are living in situations without parental care (EveryChild 2009; Petrowski, Cappa and Gross 2017). These children might live in residential care; in alternative, family-based care; in child-only households; in juvenile detention; on the streets without their parents; or with their employers/exploiters. While in some cases parental care is not possible or in the child’s best interests, children outside of parental care are more likely to be neglected and exposed to abuse and exploitation, are more vulnerable to health and behavioural problems, and face challenges in developing relationships, a sense of identity, and self-esteem (Bakker, Elings-Pels and Reis 2009; Berens and Nelson 2015; Browne 2009; Csáky 2009; Dozier et al. 2012; Tobis 2000; Wedge, Krumholz and Jones 2013).

Assessments by practitioners suggest that families experience multiple challenges that can lead to the placement of a child in institutional care or domestic service, or lead a child to the streets (Delap 2013; EveryChild 2009; Laumann 2015). A lack of financial resources within the home to address children’s needs is a common theme. Household economic interventions that aim to reduce poverty and build financial resources can therefore supplement traditional social work-driven efforts to help prevent family separation, support children’s reintegration in family care, and improve outcomes for children (Chaffin and Kalyanpur 2014; United Nations General Assembly 2010). However, social work practitioners, with skills in psychosocial and emotional support, report that they lack knowledge and skill in addressing economic matters (Chaffin and Kalyanpur 2014), particularly in low-resource environments where government assistance programmes are limited or non-existent.

At the same time, practitioners in the economic/livelihoods domain with expertise in material support, asset creation, income-generation activities, and employability skills typically work at a broader level and often lack experience working with individual families on the verge of breakdown. Understanding how to successfully integrate protective family social and economic strengthening interventions is therefore a priority, as is bridging the gap between economic development and child protection practitioners (Boothby et al. 2012), particularly given the adoption of the Guidelines for the Alternative Care of Children by the United Nations General Assembly in 20094 which has propelled state and civil society action on separated children.

1.2 Introduction to the study
In 2014, under the FHI 360-managed5 Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES) umbrella, the United States Agency for International Development’s (USAID) Displaced Children and Orphans Fund commissioned the Family Care6 project. The goal of the Family Care project is to develop evidence and guidance for how economic strengthening interventions can help separated children return to and remain in their families and to help highly vulnerable families stay together. A principle source of this evidence is a study that includes the evaluation of two projects funded by Family Care that integrated family social strengthening and household economic strengthening. Ultimately, the Family Care team will combine the evaluation evidence with a literature review and other data sources to develop global guidance on the topic, including how to match families with the forms of economic strengthening activities most appropriate to their needs and circumstances, and how these activities can contribute to family strengthening efforts. In this article, we present the considerations that influenced our evaluation design, highlighting the design elements and approaches we incorporated to better assess the integrated, cross-sectoral nature of the projects.

1.3 Theory of change
Prior to developing an evaluation framework, and indeed, prior to issuing a solicitation for the projects we would evaluate, we needed to understand the drivers of family–child separation and how family social and economic strengthening activities might be expected to mitigate them. We began with the drivers identified in the literature (Delap 2013: 13; Laumann 2015) and validated and supplemented these with learning from field visits with practitioners working on deinstitutionalisation and reunification in Uganda and Rwanda. As illustrated in Figure 1, there are drivers related to both economic and familial characteristics of households – from inadequate shelter and food insecurity to high stress levels – and these often build on, reinforce, or influence each other. The diversity of drivers identified, the interplay among them, and research suggesting that families with a child in institutional care likely experience the negative condition for five to six of these drivers (Kraguli and Pop 2012) means that there are tens of thousands of possible combinations of drivers forming pathways to separation.

To understand where interventions might disrupt some of those pathways, we reviewed the available grey literature and limited number of relevant peer-reviewed studies that discuss factors that may help families stay together (Laumann 2015). These include a sense of belonging or community acceptance and existence of social safety nets (4Children 2015), along with access to childcare (CCF Moldova and Hope and Homes for Children UK 2015), higher parental level of education (Akwara et al. 2010), parenting capacity (Annan et al. 2013), and positive emotional states (Roelen 2015). On the economic side, there are few studies that directly examine the link between economic activities and family preservation or retention in care, with those few focusing primarily on income support through government social protection programmes (Barrientos et al. 2013; Roelen and Kharki Chettri 2014; Roelen and Shelmerdine 2014). Common economic interventions in the field of child protection include limited consumption support (cash transfers), microloans or small grants, small business start-up kits, and vocational skills training and income-generating activities. There is also practitioner interest in using savings groups and sequenced ‘graduation’ approaches (Mattingly 2015; Chaffin and Kalyanpur 2014). However, a review by Ellis and Chaffin (2015) suggests that, as yet, there is no clear evidence indicating which economic strengthening interventions are effective in supporting family–child unity. We therefore mapped a number of economic and family strengthening activities onto the concept map of drivers of family–child separation, to begin to build a theory of change indicating how and where we might expect interventions to affect outcomes (see Figure 2).

1.4 Main features of the projects
Against the backdrop of limited evidence and with a draft theory of change, we put out a call for proposals for projects that would integrate a core set of traditional social work and family strengthening activities (e.g. case management, counselling, parenting skills, community dialogues) with at least three commonly used or promising economic strengthening activities (e.g. cash transfers, financial literacy training, savings groups). We requested that proposed projects serve families at risk of separation (and devise a methodology to identify them), and support the reintegration of separated children into their families and communities, a process that continues past the time of a child’s reunification with family.7

Following a competitive solicitation process, we selected two projects, both in Uganda.8 Family Resilience (FaRe), led by the Association of Volunteers in International Service (AVSI) Foundation with partner Retrak, focused on (a) supporting the reintegration of street-connected children and children from a government remand home into family care, and (b) preventing family separation in at-risk households in urban/peri-urban slums thought to be hotspots for child separation. Economic Strengthening to Keep and Reintegrate Children in Families (ESFAM), implemented by ChildFund, focused on (a) supporting the reintegration of children who had been living in child care institutions and were returned to their families in response to a government deinstitutionalisation policy, and (b) preventing unnecessary family separation among at-risk households in districts with several orphanages. The identification and selection of households ‘at risk’ of separation differed slightly between FaRe and ESFAM, but each included a mix of quantitative and peer-informed (qualitative) vulnerability assessments that assessed both economic and child protection vulnerabilities. Households eligible for inclusion demonstrated need in both domains.

Both projects included case management processes focused on developing household plans, assessing child welfare, providing counselling, and referring to outside support and services as needed as part of family social strengthening activities. Some households also received parenting skills training or coaching, individually or in groups, and/or participated in community dialogues on issues related to family dynamics and relationships. These activities accompanied regular home visits that were intended to occur once per month to once per quarter, depending on the sub-population. The primary household economic strengthening activities across the projects included time-limited unconditional cash transfers, financial literacy training, matched savings accounts, and village saving and lending associations (VSLAs), with some apprenticeships, income-generation training, and child- and youth-savings groups.

2 Evaluation design considerations
The absence of a clear and simple x -> y -> z causal pathway is perhaps not unusual for development programmes, but the added complexity of cross-sector factors affecting family–child separation generates a number of challenges for integrated development evaluation design. A full-factorial (2x2) randomised control trial (RCT) could help to isolate the effects of separate project elements and provide information on their interaction by allowing the comparison of ‘family social strengthening-only’ to ‘family social strengthening activities plus (different) economic strengthening activities’ to a control group that received only the standard of care.

We concluded, however, that important features supporting such a stringent study were not in place. First, we have limited evidence in support of any particular household economic strengthening interventions in this context to justify the selection of one economic intervention over another. We also did not know baseline rates of separation to inform estimations of statistical power. The numerous potential pathways to the primary outcome further complicated the comparison of the integrated intervention to any single intervention. And we also considered contextual factors, such as the geographic complexities of reintegrating children (research staff cannot control which children are ready for reunification, or when or where), and logistical factors, such as limited staff experience implementing a combined family and economic strengthening programme. Based on these considerations, along with time and resource constraints, we ruled out a factorial RCT design fairly quickly. Instead, our focus from the beginning was on building an evaluation framework that would generate robust data to address programmatic learning needs while filling in some of the evidence gaps.

2.1 Evaluation scope and objectives
The scope of the Family Care project evaluations is largely defined by two related objectives.

Objective 1: Demonstrate and assess the effectiveness of different household economic strengthening programmes in the context of family–child reintegration and prevention of family–child separation.

The focus of this objective is on evaluating how well different types of economic strengthening activities work when employed in support of family unity. We seek to understand the extent to which household economic strengthening activities provided in this context help to prevent family–child separation and produce positive changes in family social and economic indicators, and for which types of families. We recognise that our data, which only cover project participants, do not allow for an impact evaluation design to measure effectiveness as a net attributable impact. Instead, to assess effectiveness, we employ several methods to examine, rather than measure, the outcomes of the projects on their beneficiaries.

Objective 2: Demonstrate and assess the feasibility and appropriateness of integrating targeted household economic strengthening interventions into programming in the context of family–child reintegration and prevention of family–child separation.

The intent of this objective is to generate learning from the implementation of the projects that will inform guidance on how reintegration and prevention-of-separation programmes can best incorporate economic strengthening interventions. We seek to understand the mechanics of how economic strengthening programmes work for families, including (a) the complementarities of the household economic strengthening activities and other family strengthening activities, (b) any areas where economic strengthening activities are disruptive to family strengthening goals, and (c) the comparative costs to provide economic strengthening programming within the reintegration and prevention contexts.

The scope of our programme evaluation therefore includes outcome evaluation questions and formative evaluation questions, with specific intention to learn about activity integration and interaction effects from the answers to both sets of questions.

2.2 Outcomes
We used the concept map in Figure 2 along with the objectives above to help inform the selection of appropriate outcomes to reflect family social and economic strengthening. Selection of the outcomes – and development of the tools to measure them – was a multidisciplinary effort, reflecting input from experts in child protection and economic strengthening, local practitioners, local reintegrating families, and researchers. Given our objective to understand the effects of integrated family and economic strengthening on family–child separation, we designate permanency of placement for reintegrating children and continued family care for at-risk children as our primary outcome. We also consider this to be our ‘synergistic’ outcome as, according to the theory of change, successfully preventing separation or re-separation requires, in most cases, addressing drivers in both family and economic domains (see Figure 3). However, given the diversity of potential pathways to separation, and an unknown base rate of family–child separation, we also want to be sure to assess family social and economic outcomes separately. These intermediate outcomes also allow us to understand the relative importance of changes in different areas of family life.

3 Mixed methods evaluation design
The combination of outcome and formative evaluation questions in our study resulted in a multilevel mixed methods evaluation design. Our design comprises four elements based on four types of data – longitudinal quantitative data, costing data, qualitative longitudinal data, and activity-based process evaluations – each intended to contribute specific types of information associated with our learning objectives (see Figure 4). We describe our evaluation framework as multilevel, in addition to mixed method, to describe the ‘layering’ of data sources and analysis methods within concurrent related-but-separate assessments. In this section, we present each element, the types of data collected, and how they address our integrated development objectives.

3.1 Evaluation population
We invited caregivers in all FaRe and ESFAM households (n=~1400) to participate in the evaluation research, along with one ‘index’ child from each household. For households reintegrating a child, the reunified child was the index child. For households at risk of separation, the index child was selected based on a quantitative assessment of relative vulnerability to separation; the child between the ages of seven and 17 with the highest vulnerability score was selected as the index child in these families.

3.2 Longitudinal quantitative data
We collected information from all participating households at three points over the 18-month implementation period, to gather longitudinal quantitative (closed-form) data about both economic and wellbeing outcomes. All quantitative data were collected in-person, using questionnaires administered by external research staff (ESFAM) or project staff (FaRe). The projects used a Family Care-developed instrument to collect information from caregivers on five domains of wellbeing (social, parent–child attachment, community belonging, emotional belonging, and child care and protection) and from children on six domains of wellbeing (enjoyment of education, social, parent–child attachment, community belonging, emotional belonging, and child protection). These domains track to family strengthening activities. The projects used a separate instrument to solicit household information from the caregiver on financial and economic indicators (e.g. income, savings, assets), food security, access to basic needs and education, and child protection issues (e.g. harsh discipline, child separation). The two projects used slightly different instruments to collect this economic strengthening information, but we harmonised the questions related to the main indicators.

We designed these baseline–midline–endline instruments to address Objective 1, the extent to which economic strengthening interventions provided in the context of family strengthening affect the outcomes of interest. With these observational data from a relatively large sample, we can examine many characteristics and outcomes related to our integrated development research questions, including: (a) rates of separation for both reintegrating and at-risk households, (b) changes in a family’s wellbeing (from both caregiver and child perspective), (c) changes in a household’s economic indicators and vulnerability, (d) characteristics of households that experience a separation compared to those that do not, (e) correlations between changes in family wellbeing or economic status and separation rates, and (f) how the types of changes observed in analyses a–c differ (or not) according to the type (and amount) of economic strengthening intervention received.

Analyses e and f address questions associated with the study’s general objective to identify which economic strengthening activities might work for which types of families, necessary to inform the generation of guidelines for practitioners. The large sample size allows the use of difference-in-differences analysis and structural equation modelling to help us investigate sub-group variations and likely contributors to observed effects. For example, preliminary analyses suggest that at-risk households and reintegrating households that received a cash transfer were similarly vulnerable at baseline but experienced statistically significantly different changes between baseline and endline, with the at-risk households showing more improvement. While we do not have a comparison group to allow for measurement of an attributable net programme effect for either group of households, the cross-group statistical analysis using longitudinal quantitative data offers insights for programming consideration. The collection of midline data in addition to the before and after data allows us to better examine the trends within and across households over time, relative to both the family and economic strengthening outcomes. We can explore whether and when certain covariates impact the trajectory of household changes.

3.2.1 Set-theoretic analyses
Even with longitudinal data, what we cannot do in the absence of a comparison group to estimate a counterfactual is attribute observed changes to specific programme activities. However, there are other strategies we can employ to unpack the interactions and importance of project activities relative to the primary outcomes. In this evaluation, we use qualitative comparative analysis (QCA). QCA is a set-theoretic data analysis technique that relies on inferential logic or Boolean algebra, rather than statistical correlation, ‘to specify the different combinations of conditions linked to the selected outcome, based on the features of the positive cases that consistently distinguish them from the negative cases’ (Ragin n.d.: 11). QCA was developed in the field of political science within the discipline of case study research as a way to identify, among a small number of cases (usually countries), the policy conditions necessary and sufficient to generate a particular outcome.

Yet despite its name and its original purpose, QCA is neither limited to qualitative data nor small-n samples. Rather, it is a highly structured and quantitative – but not statistical – means of identifying patterns within a data set based on the creation of like sets. The UK Department for International Development (DFID) describes QCA as a ‘newer method’ of evaluation for which there is compelling rationale and scope (Stern et al. 2012: 49), and though QCA has not been used extensively in development evaluation, there are a handful of recent examples (Welle et al. 2015; Stedman-Bryce et al. 2015; Schatz and Welle 2016). Unlike linear regression-based approaches, set-theoretic techniques like QCA (and also coincidence analysis) do not try to organise all data points to one best-fit line, but rather they allow for multiple ‘lines’ of connection from inputs to outcomes, recognising that in complex systems, there is often more than one route from starting point to desired effect. This perspective is exactly what integrated development is about – different activities/inputs and how they combine to generate effects – and consequently aligns with both our concept map and research objectives.

In our evaluation, we use QCA to examine how some project participants succeeded in keeping children in family care (positive cases) while others did not (negative cases), based on the combination of project activity inputs they received. The analysis of these positive and negative cases allows us to identify the necessary and sufficient conditions for pathways through which the ASPIRES Family Care project participants achieve prevention of family–child separation and successful reintegration of children into family care. And indeed, the QCA analyses help both to clarify the linkages in our concept map and to identify which combinations of project activities contribute to observed effects. The analyses can also inform aspects of Objective 2, namely how family strengthening activities might be bolstering (or detracting from) economic strengthening activities, or vice versa.

Though QCA does not require a large sample, we are using our endline quantitative data to populate the ‘truth tables’ that serve as input for the QCA analyses. A large sample provides greater variation between households within the sets of positive and negative cases, which is particularly helpful in the Family Care project evaluations, given the number of economic and family strengthening activity inputs. And, because the incidence of family–child separation is unknown, a larger sample provides a greater chance to observe the negative cases, i.e. where separation does occur. The larger sample also allows for disaggregation by type of household (at-risk/reintegrating) or geography (rural/urban) to address the question of which types of economic strengthening activities appear to help which types of households.

3.3 Costing data
In response to Objective 2, we also collected cost information on all project activities over the life of the project, to examine the cost of the integrated programme and the costs of the different interventions. While integrated programming could produce economies of scale, where an integrated programme costs less than two separate interventions, the complexity of an integrated programme could also cause costs to multiply. It is therefore as important to understand the effects of integrated programming on costs as on integration outcomes. We prepared a Microsoft Excel tracking workbook and written guidance to enable FaRe and ESFAM finance staff to capture costs in a way that allows the aggregation and disaggregation necessary to examine several formative evaluation questions (e.g. the average costs of family strengthening and economic strengthening activities independently and combined). Within our larger evaluation design, these data provide an opportunity to derive costs per beneficiary and, in conjunction with the quantitative outcome data, may provide suggestive information about cost-effectiveness.

In practice, the cost data proved the most challenging to collect. Despite their willingness and intent to provide quarterly costing updates, our partners found this activity to be very difficult as their financial systems did not capture information in ways that easily translated to the costing templates we provided. Rather, programme implementation, management, and finance staff often had to sit together and review physical documentation to produce the updates. We reduced our requirements for precision to make the exercise more manageable, acknowledging that less precision affects the quality of the data and the analyses we can perform. Nonetheless, the cost estimates for project elements are essential to the programmatic guidance we are generating for the integration of economic and family strengthening activities.

3.4 Longitudinal qualitative data
We also collected longitudinal qualitative data from a stratified purposive sample of 80 households, 20 each of reintegration households and at-risk households per project, stratified further by sub-county to ensure geographic variation. Households were randomly ordered within each sample stratum and then systematically selected in order according to the gender of the index child to maintain gender diversity. For each selected household, the study team conducted separate open-ended qualitative interviews with both a caregiver and the index child. We interviewed reunified families three times; families at risk of separation four times. All qualitative interviews were conducted by external research staff in pairs, who conducted the caregiver and child interviews separately and in private. All interviews began by building a guided ‘life narrative’ that provides a snapshot of life within the household on specific domains and captures changes in these domains over time.

In an integrated development context, qualitative data are extremely useful for identifying and analysing interactions and synergies. We asked direct questions about interactions, such as ‘What connections do you see between the [family strengthening activity received] and the [economic strengthening intervention]?’ And we see examples of complementary effects in both directions:

They support each other because when you learn how to save money and spend it well, definitely it will help you become a better parent because then you can provide for their needs better and even pay their school fees. (Gulu at-risk household)9

The caregiver thinks that the two project activities are both important and if delivered well they can positively impact her household. For example, she thinks that if there’s cooperation among family members they can achieve positive decision of labour where for instance as she attends to the garden, her daughter attends to the grocery stall. This would promote unity and improve on the household productivity. (Luwero reintegration household)10

We also look for examples of interactions and synergies in participant narratives. The longitudinal nature of the open-ended interviews enabled us to develop household-level case histories that include rich descriptions of changes in each family from caregiver and child perspectives, on both the economic and livelihoods situation and on the interpersonal dynamics of wellbeing. A caregiver may describe how her ability to generate income (an economic outcome) reduces her stress, which in turn allows her to parent better and improve the caregiver–child connection (a family wellbeing outcome), which she in turn credits for the lack of family–child separation in her household.

The children are happy because we are no longer bothered by the land lady. Even recently I told her that [we] had sent this month’s instalment and she was happy. Our relationship is now better because we are no longer constrained by money problems. I am no longer worried as before, so I do not take out my stress on the children by shouting at them. I talk to them in case they have done something wrong. (Kampala at-risk household)11

In addition to answering a range of programmatic questions on their own, the qualitative data provide crucial contextual information to facilitate interpretation of the quantitative findings. For instance, when we see in the preliminary quantitative data that cash transfer households with a reunified child increased their income but decreased their savings, on average, from baseline to endline, we can look to the qualitative data for explanations of why this may be the case.

3.5 Formative evaluations of specific interventions
The quantitative, costing, and qualitative data described above contribute information related to Objective 2 and the relationships between family social and economic strengthening project elements. However, questions around the mechanics of overall implementation and specific project components require dedicated inquiry to generate information on how the household economic interventions functioned under the specific contexts of ESFAM and FaRe, along with perspectives on what worked from an implementation and supply point of view and what could be improved in similar situations. To do this, we undertook two formative evaluations per project on common economic strengthening activities: one on cash transfer interventions and one on VSLA interventions. We treated these evaluations as an ‘external consultancy’, with the scope of work drafted by our research team, shared with FaRe and ESFAM for their client perspective, and then carried out by two ASPIRES team members with some knowledge of the projects but little prior contact with them. Our ASPIRES colleagues then conducted fieldwork for both evaluations concurrently, including interviews and focus group discussions with project staff, finance staff, local government, cash transfer recipients, and VSLA members.

These evaluations provide important data on the feasibility and acceptability of specific economic strengthening activities conducted with a family strengthening aim. For example, we learned about the appropriateness of the amount and duration of cash transfers (likely too little cash for too short a period for some families), the ability of very poor people to save (they can, and less-poor VSLA members are willing to accommodate them), and the workforce needs and challenges in doing this type of integrated programming (there are many). In addition, the findings from these formative evaluations also help us to interpret the data on effects, in the same way the qualitative data do, but with more specificity on designated interventions. For example, a finding from the VSLA formative evaluation – that VSLA participants felt reduced social isolation and a sense of having a social safety net in the community – shows a complementary effect of the integrated programming, in that the participants’ sense of community belonging and social wellbeing (two of the family strengthening objectives) are accomplished through an economic strengthening activity.

4 Discussion
In an evaluator’s ideal world, development problems would come with a neat underlying cause–effect structure against which to design and test interventions. But the real world is messy. As Neil deGrasse Tyson said (2016), ‘In science, when human behavior enters the equation, things go non-linear. That is why physics is easy and sociology is hard.’ Integrated development is sociology – and anthropology, epidemiology, economics, social work, public health, diplomacy, and a dozen other disciplines working together. It is usually, as in the case of the Family Care project, closer to circular than linear, with multiple interacting feedback loop ‘causes’ for each effect. Response to this complexity – and evaluation of it – requires integrated and complementary solutions.

An obvious choice for evaluating a two-sector integrated development programme is a 2x2 factorial design RCT. The factorial design facilitates measurement of interaction and synergy effects and random assignment eliminates selection bias. A recent review found, however, that while 70 per cent of integrated development impact evaluations employed randomisation, only 7 per cent used a factorial design (Ahner-McHaffie et al. 2018). In our case, we did not have the resources to conduct an RCT, but as important, we concluded there was not enough evidence regarding the individual interventions or information on how they should be implemented and integrated to justify a factorial RCT. The ‘gold standard’ approach is less golden if the results are null because the interventions being tested are not well understood (in theory or practice) (Brown 2017).

In situations like this, where the focus is still on understanding how an integrated programme might work, rather than measuring net impacts, descriptive mixed method observational research plays a critical role in building the evidence base. Done well, it can identify trends, surface unintended or unexpected consequences of new integrated programming, provide explanations to help interpret those trends, and inform programme design and implementation decisions. Relatedly, in the absence of an impact evaluation (and even with it), qualitative data can provide critical explanatory power to a research design.

Ahner-McHaffie and colleagues concluded in their review that ‘using a mixed method approach to include examination of cost efficiencies and qualitatively assessing synergies offer[s] value for determining how integration factors in to the program findings’ (2018: 10). Our multilevel mixed methods approach was perhaps unorthodox in its blend of formative and outcome evaluation objectives, yet because the elements were designed as complementary, they yielded a broad range of robust data to address those objectives. The longitudinal data – quantitative and qualitative – provide different perspectives from which to assess questions about both the process and effects of integration and synergy. The large and varied sample for the quantitative data also allows disaggregation for within-group comparisons, and flexibility to conduct QCA to help understand which of the many project elements are necessary for success. The formative evaluations, along with the costing data, help us to understand the feasibility of interventions being used in a new and different context and even reveal some complementary effects. In these ways, the Family Care evaluation data will substantially contribute to building the evidence base and providing programmatic guidance on the process of implementing economic strengthening activities in support of family unity, indicating directions for future integrated programming – and evaluation – in this area.

* This issue of the IDS Bulletin was prepared as part of the impact evaluation of the Millennium Villages Project in northern Ghana, 2012–17, funded by the UK Department for International Development (DFID) (www.dfid.gov.uk). The evaluation was carried out by Itad (www.itad.com) in partnership with IDS (www.ids.ac.uk) and PDA-Ghana (www.pdaghana.com). The contents are the responsibility of the evaluation team and named authors, and do not necessarily reflect the views of DFID or the UK Government.

✝ This project was carried out under United States Agency for International Development (USAID) Cooperative Agreement No. AID-OAA-LA-13-00001. The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States government.

1 Corresponding author. FHI 360, Durham, North Carolina, USA. 359 Blackwell Street, Suite 200, Durham NC 27701, USA. enamey@fhi360.org.

2 FHI 360, Altadena, California, USA. 1825 Connecticut Avenue, NW, Suite 8000, Washington DC 20009, USA. llaumann@fhi360.org.

3 FHI 360, Washington, District of Columbia, USA. 1825 Connecticut Avenue, NW, Suite 8000, Washington DC 20009, USA. abrown@fhi360.org.

4 The Guidelines for the Alternative Care of Children by the United Nations General Assembly in 2009 affirm the ‘family as the fundamental group of society and the natural environment for the growth, wellbeing and protection of children’, children’s right to ‘live in a supportive, protective and nurturing environment’, and the responsibility of the state to return children to family care, ensure that families have access to support as caregivers, and ensure appropriate alternative care if needed (United Nations General Assembly 2010: 2–3).

5 www.fhi360.org.

6 The Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES) project, supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) and USAID and managed by FHI 360, supports gender-sensitive programming, research, and learning to improve the economic security of highly vulnerable individuals, families, and children. The Family Care project under ASPIRES, funded by the Displaced Children and Orphans Fund of USAID, developed the work described here.

7 The Guidelines on Children’s Reintegration define reintegration as the ‘… process of a separated child making what is anticipated to be a permanent transition back to his or her family and community (usually of origin), in order to receive protection and care and to find a sense of belonging and purpose in all spheres of life’ (Interagency Group on Children’s Reintegration 2016: 1).

8 The initial intention had been to select projects in two different regions (eligible countries included Rwanda, Uganda, Colombia, Guatemala, and Haiti), but the projects that scored highest in the competitive process were both proposed for implementation in Uganda.

9 Interview conducted July 2017.

10 Interview conducted July 2017.

11 Interview conducted July 2017.

4Children (2015) Expanding the Evidence Base to Inform Next Generation Programming: Keeping Children in Healthy and Protective Families – Research Inception Meeting Synthesis Report, Baltimore MD: Catholic Relief Services

Ahner-McHaffie, T.W.; Guest, G.; Petruney, T.; Eterno, A. and Dooley, B. (2018) ‘Evaluating the Impact of Integrated Development: Are We Asking the Right Questions? A Systematic Review’, Gates Open Research 1.6: 1–23 (accessed 1 October 2018)

Akwara, P.A. et al. (2010) ‘Who is the Vulnerable Child? Using Survey Data to Identify Children at Risk in the Era of HIV and AIDS’, AIDS Care 22.9: 1066–85

Annan, J.; Bundervoet, T.; Seban, J. and Costigan, J. (2013) A Randomized Impact Evaluation of Village Savings and Loans Associations and Family-Based Interventions in Burundi, New York NY: International Rescue Committee

Bakker, C.; Elings-Pels, M. and Reis, M. (2009) The Impact of Migration on Children in the Caribbean, New York NY: UNICEF

Barrientos, A.; Byrne, J.; Villa, J.M. and Peña, P. (2013) Social Transfers and Child Protection, Innocenti Working Papers 2013-05, Florence: UNICEF Office of Research

Berens, A.E. and Nelson, C.A. (2015) ‘The Science of Early Adversity: Is There a Role for Large Institutions in the Care of Vulnerable Children?’, The Lancet 386.9991: 388–98 (accessed 14 September 2018)

Boothby, N. et al. (2012) ‘What are the Most Effective Early Response Strategies and Interventions to Assess and Address the Immediate Needs of Children Outside of Family Care?’, Child Abuse and Neglect 36.10: 711–21

Brown, A.N. (2017) Null Results Should Produce Answers, Not Excuses, R&E Search for Evidence, Washington DC: FHI 360 (accessed 1 October 2018)

Browne, K. (2009) The Risk of Harm to Young Children in Institutional Care, London: Save the Children

CCF Moldova and Hope and Homes for Children UK (2015) Preventing Family Separation/Supporting the Tracing, Reunification and Reintegration of Separated Children into their Families and Communities, Chișinău and Salisbury: CCF Moldova and HHC UK

Chaffin, J. and Kalyanpur, A. (2014) What Do We Know about Economic Strengthening for Family Reintegration of Separated Children? New York NY: CPC Network

Csáky, C. (2009) Keeping Children Out of Harmful Institutions: Why We Should be Investing in Family-Based Care, London: Save the Children

deGrasse Tyson, N. [@neiltyson] (2016) ‘In Science, When Human Behavior Enters the Equation, Things Go Nonlinear. That’s Why Physics is Easy and Sociology is Hard’, tweet (accessed 26 September 2018)

Delap, E. (2013) Towards a Family for Every Child: A Conceptual Framework, London: Family for Every Child

Dozier, M.; Zeanah, C.H.; Wallin, A.R. and Shauffer, C. (2012) ‘Institutional Care for Young Children: Review of Literature and Policy Implications’, Social Issues and Policy Review 6.1: 1–25

Ellis, C.M. and Chaffin, J. (2015) ‘Evaluations of Outcomes for Children and Youth from NGO-Supported Microeconomic Interventions: A Research Synthesis’, Enterprise Development and Microfinance 26.2: 104–21 (accessed 14 September 2018)

EveryChild (2009) Every Child Deserves a Family: EveryChild’s Approach to Children without Parental Care, London: EveryChild (accessed 1 October 2018)

Interagency Group on Children’s Reintegration (2016) Guidelines on Children’s Reintegration (accessed 25 July 2018)

Kraguli, J. and Pop, D. (2012) Preventing the Separation of Children from their Families in Bosnia and Herzegovina: Review of Hope and Homes for Children ACTIVE Family Support Programme in Bosnia and Herzegovina 2003–2010, Salisbury: Hope and Homes for Children

Laumann, L. (2015) Household Economic Strengthening in Support of Prevention of Family–Child Separation and Children’s Reintegration in Family Care, Washington DC: FHI 360

Mattingly, S. (2015) Symposium Report: Keeping Children and Families Together with Economic Strengthening (accessed 1 October 2018)

Petrowski, N.; Cappa, C. and Gross, P. (2017) ‘Estimating the Number of Children in Formal Alternative Care: Challenges and Results’, Child Abuse and Neglect 70: 388–98 (accessed 14 September 2018)

Ragin, C.C. (n.d.) What is Qualitative Comparative Analysis (QCA)? (accessed 26 September 2018)

Roelen, K. (2015) ‘Economic Strengthening and Child Wellbeing and Care: Positive Effects and Adverse Consequences’, paper presented at the STRIVE Symposium, Washington DC, 6 March (accessed 1 October 2018)

Roelen, K. and Kharki Chettri, H. (2014) Researching the Linkages between Social Protection and Children’s Care in Ghana: LEAP and its Effects on Child Well-Being, Care, and Family Cohesion, London, Brighton and Accra: Family for Every Child, IDS and Challenging Heights

Roelen, K. and Shelmerdine, H. (2014) Researching the Linkages between Social Protection and Children’s Care in Rwanda: The VUP and its Effects on Child Well-Being, Care and Family Reunification, London: Family for Every Child

Schatz, F. and Welle, K. (2016) Qualitative Comparative Analysis: A Valuable Approach to Add to the Evaluator’s ‘Toolbox’? Lessons from Recent Applications, Centre for Development Impact (CDI) Practice Paper 13, Brighton: IDS

Stedman-Bryce, G.; Schatz, F.; Hodgkin, C. and Balogun, P. (2015) Medicines Transparency Alliance (MeTA) Evaluation: Testing MeTA’s Underlying Intervention Logic, London: e-Pact

Stern, E. et al. (2012) Broadening the Range of Designs and Methods for Impact Evaluations, DFID Working Paper 38, London: Department for International Development

Tobis, D. (2000) Moving from Residential Institutions to Community-Based Social Services in Central and Eastern Europe and the Former Soviet Union, A World Free of Poverty Series, Washington DC: World Bank

United Nations General Assembly (2010) Guidelines for the Alternative Care of Children, Resolution Adopted by the General Assembly, A/RES/64/142, New York NY: United Nations (accessed 26 September 2018)

Wedge, J.; Krumholz, A. and Jones, L. (2013) Reaching for Home: Global Learning on Family Reintegration in Low and Lower-Middle Income Countries, New York NY: Better Care Network

Welle, K.; Williams, J.; Pearce, J. and Befani, B. (2015) Testing the Waters: A Qualitative Comparative Analysis of the Factors Affecting Success in Rendering Water Services Sustainable Based on ICT Reporting, Brighton: IDS and WaterAid

© 2018 The Authors. IDS Bulletin © Institute of Development Studies | DOI: 10.19088/1968-2018.164

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non Commercial No Derivatives 4.0 International licence (CC BY-NC-ND), which permits use and distribution in any medium, provided the original authors and source are credited, the work is not used for commercial purposes, and no modifications or adaptations are made. https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode

The IDS Bulletin is published by Institute of Development Studies, Library Road, Brighton BN1 9RE, UK. This article is part of IDS Bulletin Vol. 49 No. 4 September 2018 ‘The Millennium Villages: Lessons on Evaluating Integrated Rural Development’; the Introduction is also recommended reading.