Annex 1: Methods for the development of the guide and repository of tools

Process

To develop this guide, the Evidence to Policy and Impact Unit of the Research for Health Department, Science Division engaged a wide range of experts and stakeholders. Overall, the methodology consisted of the following steps:

• Conduct of a cross-Organizational consultative meeting in November 2020 to establish a WHO Working Group to strengthen the use of research evidence for health impact. The heads of departments identified as closely linked to EIDM were invited to attend. The consultative meeting agreed that an Organizational framework and guidance were needed for staff on good EIDM practices and principles. It was decided to create a cross- Organizational Working Group on Evidence-to-Action to initiate a mapping exercise of existing EIDM tools and approaches, identify good practices and gaps, and to develop a related guidance for WHO staff.

• The internal WHO Working Group on Evidence to Action, with representation from departmental focal points, facilitated and led by the Evidence to Policy and Impact Unit of the Research for Health Department, was formed in February 2021. The Working Group met on a number of occasions to review and guide progress, and also contributed expertise outside of formal meetings.

• An external Editorial Board made up of eight experts in the area of EIDM was convened in May 2021 to oversee the development of the guidance document and repository of EIDM tools, and to provide external peer-review of the guidance document. They met monthly until September 2021.

The Guide

The Guide was developed based on a review of the literature in the area and the experience and expertise of the author group (see Acknowledgements section). The development was led by Tanja Kuchenmüller, who also constructed the evidence ecosystem framework and wrote the first draft of the document. The framework was modelled based on Graham’s knowledge-to-action cycle1 and initial exchanges with the EVIPNet Europe Steering Group. The original detailed funnel came from Ludovic Reveiz, Pan American Health Organization.

The Guide was reviewed by the Editorial Board and by an internal review group, which was a subgroup of the WHO Working Group on Evidence to Action. After each revision, the document was updated and strengthened based on their review.

Mapping of EIDM tools

A systematic process was applied to develop the repository that included clear inclusion and priority-setting criteria; a process of assessment and data extraction (mapping) undertaken by two reviewers independently; and use of an external expert group to oversee the process and for referral of discrepancies in assessments. Tools from across the Organization and workstreams (including guideline development, evidence briefs for policy, etc.) within the evidence ecosystem were eligible. For the first round, tools were suggested by technical areas, which also undertook the preliminary mapping/data extraction. Further updates are planned with wider searching for tools.

Process

The process undertaken can be summarized as follows:

• Tools were suggested by technical areas within the WHO head office or regional offices.

• These tools were assessed against the inclusion criteria by two independent reviewers from the project team (Evidence to Policy and Impact Unit with the assistance of consultants working on the guidance). Any discrepancies in assessment were discussed, with consultation of a third reviewer where needed.

• Tools that met the inclusion criteria were mapped against the evidence ecosystem framework (Fig. 2.2) that includes the evidence creation funnel and the policy/action cycle. This involved checking (and updating if necessary) the initial mapping plus completion of more detailed mapping.

• The data extracted included whether: the tool was targeted at the funnel or cycle or provided general support/capacity-building for EIDM; the level of the funnel or step/s of the cycle targeted; focus on clinical, public health and/or health system interventions; capacity-building strategies being implemented in relation to the tool (Yes/No); plus details if “yes” to the previous question; focus on the co-production of research between researchers and decision-makers (Yes/No/Partially); focus on practice (patient–provider) or policy (policy-maker) or both; whether it is a general tool that applies across topic areas/health issues or to a specific topic area; and the target audience for the tool. For tools that focused on one or more steps of the policy/action cycle, data were extracted on the specific substeps covered.

• For tools that met the inclusion criteria and were mapped, the priority- setting criteria were applied to help choose which tools to promote where more than one tool covered the same step or process.

• The results of the mapping of tools and application of the priority-setting criteria were shared with the technical areas for checking and feedback.

Inclusion criteria for tools

The tool must fulfil all five of these criteria to be included in the repository.

1. Documents described as a guidance, handbook, manual, method or tool that outline a methodology or process for creating or applying research evidence; AND

2. Relevant to the policy/action cycle OR to the creation of tertiary research OR to the processes or actions of linkage and exchange between researchers and decision-makers; AND

3. A WHO tool or an external tool that is used by WHO in the process of knowledge translation/evidence-informed decision-making; AND

4. Document available in English; AND

5. Published in the year 2005 or after.

Priority-setting criteria for tools

These criteria will help to choose between tools that meet the inclusion criteria. For example, to achieve the label of “recommended” or “good practice”, the tool must achieve a higher score than the alternatives. For the first three criteria we are interested in the process of development of the tool rather than its application.

1. The tool is supported by research evidence in its development.

2. Decision-makers, researchers and other stakeholders were involved in its development (e.g. through consultation, on a working group).

3. The tool was piloted/evaluated in the process of its development.

4. The tool is used by Member States.

5. The tool is of high quality, in that the method described supports the use of the best available evidence from research, while recognizing that other factors such as context, acceptability to stakeholders, etc. also play an important role in EIDM (see Chapter 1 for full definition). The tool covers a range of steps from the policy/action cycle.

Criteria were scored 0, 1 or 2 – where 0 = “not at all” (or “not specified”); 1 = “partly” and 2 = “mostly/yes”. Given the importance of criterion 5, this was weighted more heavily than the other criteria. Thus, the corresponding scores for criterion 5 are: 0, 2, or 4 – where 0 = “not at all” (or “not specified”); 2 = “partly” and 4 = “mostly/yes”.
Maximum total score = 12