ILO response to a paper in 'Frontiers in Public Health' journal

On 17 May 2021, a paper published in this journal about the ILO's Work in Freedom (WIF) programme misrepresents facts and draws conclusions based on insufficient evidence. Here are the observations from the ILO.

Press release | 31 May 2021
A paper Human Trafficking: Results of a 5-Year Theory-Based Evaluation of Interventions to Prevent Trafficking of Women From South Asia was published on 17 May 2021 in Frontiers in Public Health journal. The ILO observes that:

1. As important background information for this evaluation, the paper should have explained the original scope of its evaluation role and many of the changes that occurred thereafter. The original scope of the evaluation consisted of an integrated process evaluation intended to demonstrate what works on trafficking in persons. The funding LSHTM received from DFID (a matter of public record) was commensurate with such a task which was intended to include clustered randomized trials (CRT) and other evidence pieces on all of WIF’s work and not just on the community awareness component. The paper fails to mention this. In fact, LSHTM’s decision to focus exclusively on community work (in countries or states of origin of migration) and apply a theory based critical realist method only came much later towards the end of the evaluation.

The field research undertaken as part of the LSHTM evaluation was concentrated in particular periods, was non-continuous, and - following a series of pivots in the LSHTM led overarching evaluation design - linked only indirectly to WIF programme activities in the three countries selected for evaluation purposes. In Nepal a “formative” (single round) survey was undertaken in one district in 2014. Subsequent surveys in three other districts were undertaken in 2015/2016. Fieldwork in Bangladesh commenced in December 2015. In India, fieldwork commenced in 2016. It is notable that the paper's presentation of the WIF programme logic refers to a version already updated and modified by the initiation of the evaluation's data collection in the three study countries. It is also notable that the periods for the fieldwork, the sampling design and the areas (if any) for which the surveys are representative (and the use or otherwise of sample weights -, the number, frequency, and timing of qualitative interviews and other such basic information to permit the journal editor and the reader to assess the authors' statement that they "prioritized robust methodological approaches" (page 10) are either not presented or are inconsistently presented. This may lead the reader to imagine that the evaluation efforts were conducted continuously or at regularly spaced intervals over the period of five years stated at the outset of the paper.

While the evaluation design was unstable over the project duration, at no point in the five-year project is it expressed in terms of ‘Realist principles’ in the publicly available review documentation (annual reviews available here). From 2012 to late 2013, the evaluation design is experimental (cluster randomised trials). In 2014, the initial experimental evaluation design is somewhat relaxed to a quasi-experimental design. The final design (not established until 2015, or three years into the 5-year project) combines survey, interview, and ethnographic methods to varying degrees in the three study countries. This is a matter of public record (here). While there may be a rationale for modifying the evaluation design, the paper misrepresents this evolution. This is particularly notable in relation to the authors' statement (page 3) that “…the intervention’s feasibility, acceptability, and uptake had not been previously assessed to justify a complex and costly experimental design”. . It is notable in relation to the authors' repudiation of a "what works" approach to evaluation ("Following realist principles, we did not expect to find a definitive answer to the question of what works to prevent human trafficking but rather aimed to investigate what mechanisms were relevant to preventing human trafficking in the intervention contexts" (page 3 of the paper)).

None of the authors of this new paper were involved in the design of the studies in India and Bangladesh. In fact, the authors of both the India and Bangladesh studies are different. The "findings" the authors selectively present for Bangladesh are taken from a 90-page report (reference 32) and journal article (reference 35), the later into which they had no input. In the case of the independently prepared India study report (CWDS 2019), none of the recommendations it listed are even mentioned in the new paper. The paper also conflates the CWDS India study with the review of WIF training. In reality the new paper principally relies on the data collected in Nepal, where the sample used to estimate prevalence was not a representative one as the authors admit themselves in the limitations section.

2. The authors of the paper base their argumentation on chronologically misrepresented information. The paper refers to old programme assumptions (from 2013) when the programme had barely started (e.g. the need to advise women against migration through recruitment agents was an old programme premise that was quickly corrected in 2015 when WIF’s work on recruitment began in earnest). As mentioned above the paper uses data that was primarily collected between 2015 to 2016. Throughout the latter years of the project, constructive inputs received from the independent research groups in Bangladesh and India along with other focussed research studies directly commissioned by ILO and inputs from the project's dedicated Advisory Group, informed the project evolution. In response to such feedback WIF reviewed its own programme assumptions in 2015 and adopted an adaptive learning approach. It documented lessons learned as early as 2017 and published new iterations on a biannual basis – all of which are on public record. Many of the lessons WIF has documented over the past five years relate directly to some of the findings presented in this new paper. The assertion that the programme did not learn the lessons offered in their paper when its second phase was being prepared, is inaccurate. The quote linked to the Independent Commission for Aid Impact (ICAI) is misleading. This leads to the next point.

3. No reference is provided for the ICAI quote, which is nevertheless used to support the narrative of this paper. The paper refers to the following quote: “As noted by the Independent Commission for Aid Impact, “There is also no evidence that these findings were used to inform the design of the second phase of the work.” But this quote is not from the published ICAI report on the UK’s Approach to Tackling Modern Slavery Through the Aid Programme. The authors need to give a reference for this quote. The published ICAI review takes a different view and acknowledges that the second phase of Work in Freedom has shifted the focus of the programme as part of its ‘whole trafficking chain’ approach, with more support for migrant workers and national authorities in destination countries. This is set out on page 40, paragraph 4.61 and the authors need to explain why they have used an unsourced quotation attributed to ICAI rather than using the official ICAI report to represent ICAI views on whether the second phase of Work in Freedom has adapted its approach.

Furthermore, where the paper does quote from ICAI report it appears selective in what it includes and omits. It quotes the following sentence from the report ‘It also makes little sense to address international trafficking and forced migration in source countries without also taking necessary action in destination countries and along migration pathways” (Footnote 52). But it omits to say that in the same paragraph of the report (para 4.29) the ICAI review explicitly acknowledges that “The Work in Freedom programme is one of the few examples in which programmes have been extended to destination countries, such as Jordan and Lebanon”.

4. The paper largely ignores four of the five main areas of WIF interventions. For example, WIF was involved in destination areas where the working relationships between workers and employers and/or contractors exist. That is why ILO, as the UN’s specialized agency for the world of work, was involved in WIF in the first place. In particular, even though the paper acknowledges that other aspects of the programme existed, it makes no mention of the project’s law and policy advocacy, collective organizing of workers, employer awareness, research and recruitment angles of WIF interventions. Instead, the paper focuses almost exclusively on the pre-migration angle of WIF. Even within the field of pre-migration, the paper assumes that ‘trainings’ were the main intervention. That was not the case. A variety of intervention methods were used, including door-to-door visits, women-led events and collective organising, which arguably played a more important role in informing women who were considering migration as an option. Unfortunately, the authors of the paper did not assess that. Many of the gaps mentioned in the paper, may have been informed otherwise had the authors assessed those other components. In the discussion section, the paper suggests that: “it may be problematic to invest in premigration interventions alone, as these single location initiatives do not take sufficient account of the full migration trajectory, especially the later stages toward the destination and workplace, where the power asymmetry widens and most of the abuses occur…”. This is not a new finding since WIF at its early stage, even in 2013, already planned multiple other angles of interventions. The authors of the paper were not involved in assessing those other angles, but this does not mean that they did not exist – they did and it is objectively verifiable. Such an assertion, which constitutes an important recommendation of the paper, cannot be presented as something new for the ILO or others implementing similar programmes.


Chronology for reference

2012
The LSHTM-led evaluation was initially commissioned to deliver "two Impact Evaluation studies which will contribute a credible, robust evidence base to work on trafficking prevention (this has not been done before)." (DFID Business Case and Summary 2012: page 36, available here).
Note: More details of the initial evaluation design and objectives are provided on page 43 of the same document: "DFID has contracted the London School of Hygiene and Tropical Medicine (LSHTM) through an OJEU Competitive Tendering Process to lead two large impact evaluations to build a robust evidence base. These are intended to not only monitor and assess impact of the DFID programme, but to contribute to global knowledge goods on trafficking”. “Using a clustered randomized trials (CRT) methodology, the impact evaluations will be designed in consultation with the implementation partner (the ILO), independent Monitoring and Evaluation Experts at the LSHTM, and DFID’s Evaluation Department".

2013
In the 2013 Annual Review, (published here), the LSHTM team present the first major revision to the evaluation design: "Two quasi-experimental evaluation trials and longitudinal case study series to strengthen the evidence base on ‘what works’ for trafficking prevention", retaining a concern to elucidate "what works".

2014
At the time of the 2014 Annual Review (published here), the LSHTM team retain the same language: "Two quasi-experimental evaluation trials and longitudinal case study series to strengthen the evidence base on ‘what works’ for trafficking prevention", page 7 - together with the commitment to demonstrating "what works".

Note: On pages 6 - 7 of the same report, the LSHTM team describe the project in the following terms "A large portion of funding (approx. £2m) has been allocated to building the evidence base, in response to the lack of rigorous research on effective anti-trafficking interventions in South Asia (…) This output is on track and shows promise to provide the most ambitious and robust evidence base yet on trafficking prevention. LSHTM are conducting an independent impact evaluation to identify ‘what works’ in human trafficking prevention".

2015
By the time of the 2015 Annual Review (available here), the LSHTM team had revised the evaluation design once again to consist of "Three mixed-methods studies strengthen the evidence base on forced labour, trafficking prevention and safe migration" (page 11). It is during this 2015 Annual Review Cycle that the LSHTM team request authorisation to omit the planned Bangladesh component of the (reduced scope) evaluation, owing to "resource constraints" (p12).

2018
As late as the final 2018 Annual Review, the LSHTM team had at no stage mentioned that the evaluation design was based on "realist evaluation techniques" as the paper states (available here).