Skip to main content

What do we know about evidence-informed priority setting processes to set population-level health-research agendas: an overview of reviews

Abstract

Background

This overview aimed to synthesize existing systematic reviews to produce a draft framework of evidence-informed health priority setting that supports countries in identifying appropriate steps and methods when developing and implementing national research agendas.

Main body

We searched Ovid MEDLINE® and the WHO Institutional Repository for Information Sharing from 2010 to 2020 for critical or systematic reviews that evaluated research priority setting exercises. We adapted the AMSTAR checklist to assess the quality of included reviews and used adapted frameworks for data extraction and analysis. The search resulted in 2395 titles, of which 31 were included. Populations included in the reviews typically involved patients, families and carers, researchers, clinicians, policymakers and research funders. The topics covered in the reviews varied from specific diseases or conditions, approaches for healthcare practice or research priority setting methods itself. All the included systematic reviews were of low or critically low quality. The studies were thematically grouped based on their main focus: identifying and engaging with stakeholders; methods; context; and health area.

Conclusion

Our overview of reviews has reconfirmed aspects of existing frameworks, but has also identified new concepts for countries to consider while developing their national research agendas. We propose a preliminary framework for consideration that highlights four key phases: (1) preparatory, (2) priority setting, (3) follow-up phase and (4) sustainability phase, which have thirteen sub-domains to consider.

Background

In recent years, an evidence-based approach has not only become a cornerstone for informing health practice policy, but it has also become a key part of informing decisions on how to conduct and organize research. This includes how to set priorities for research, which is a political and social process that is informed by the views and experiences of the stakeholders along with available data and information. The research priority setting approach depends on the context, stakeholders and why an organization is conducting the priority setting exercise in the first place. Due to its complexity, more recent studies focus on individual elements or steps of the priority setting process and how these can lead to certain outcomes rather than evaluating the broader priority setting process.

The past two decades have also seen an increase in efforts to develop better stakeholder engagement, especially with patients, to set research priorities (Stewart et al. 2011). This growing movement has been accompanied by a proliferation of primary literature on the conduct, evaluation, and reporting of research priority setting, as well as a subsequent increase of systematic reviews focusing on those areas. There are also numerous guidance documents and manuals on how to set priorities for research (World Health Organization 2020a; Ghaffar et al. 2009; McGettigan and Henry 2011).

While the documents, such as the World Health Organization (WHO) guidance (World Health Organization 2020a), draw on existing research priority setting experiences (tacit knowledge) and examples of good practice, the methods used in the development of these documents are not always inclusive of a systematic evaluation of the literature. Systematic reviews to date have consolidated published priority setting exercises across a variety of fields and disciplines, such as a systematic review of research priority setting in childhood chronic disease (Odgers et al. 2018) and a systematic review on research priority setting exercises in Zambia (Chanda-Kapata et al. 2016). Recent systematic reviews have also acknowledged the high degree of variability in methods used for collecting priority topics and have expressed the need for more standardized methods (Roche et al. 2021).

The COVID-19 crisis has placed enormous pressure on healthcare systems, health research systems and policymakers. These pressures underscore the importance and need, more so now than ever, of consolidated frameworks and processes for performing and evaluating research priority setting exercises, particularly for use at a system and country level. To facilitate the widespread use of evidence, information and research, a systems approach in health research is critical to ensure effective coordination and knowledge translation (Hanney et al. 2020). During the COVID-19 crisis and indeed, in the aftermath, when countries emerge from the acute stages of managing outbreaks, frameworks are needed to facilitate decision makers in allocating limited time, funding and resources to the most relevant and pressing research topics (World Health Organization 2020b).

Systematic reviews primarily address a focused research question. Given that research priority setting processes contain multiple components and stages, a single systematic review is unlikely to be sufficiently comprehensive to assess all these components. Overviews of reviews (hereafter referred to as overviews) “compile data from multiple systematic reviews to provide a single synthesis of relevant evidence for decision making” (Pollock et al. 2016) and can therefore be more accessible to decision makers. Overviews can also avoid uncertainty created by conflicting conclusions from different reviews, which may vary in scope and quality. While the volume of reviews assessing different aspects of research priority setting continues to grow, there is an increasing need amongst policymakers for an overarching overview of reviews that consolidates existing reviews and identifies which practices can be translated into policy, and which areas require further research. Therefore, this overview aimed to synthesize existing systematic reviews to produce a framework for evidence-informed health priority setting processes that could be considered by countries to support the development of national public health research agendas.

Main text

Protocol and registration

The protocol of this overview was drafted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) checklist (Moher et al. 2015). As there is no defined guidance on how to conduct systematic reviews on research priority setting methods or for overviews of methodological reviews, we utilized an adapted approach of the methodology developed for performing overviews of clinical topics (Pollock et al. 2016). The complete methods included in the protocol are described in the registered protocol on Open Science Framework (https://osf.io/necm9) and are summarized below.

Eligibility criteria

Our primary objective was to identify methods for setting priorities in research that inform population level health-related research agenda (context). However, not all studies clearly report whether they focus on population-level or patient-related issues. Thus, for the purposes of our review, we did not restrict our studies based on target population and included all health-related priorities. The core concept examined is any critical or systematic review that evaluates research priority setting exercises that either (1) involve stakeholders; or (2) utilize a transparent and data-driven approach to analyse the process. Though preferably we identified priority setting exercises that contain both elements. Stakeholders are defined as patients, caregivers, general community, health professionals, researchers, policymakers, non-governmental organizations, government, industry, as well as specific groups, including vulnerable and marginalized populations (Tong et al. 2019).

Types of studies

We included any critical or systematic review on health research priority setting that focused on a specific topic or methodology aspect, for example, stakeholder engagement in setting priorities. A critical review is defined as a review that has “extensively researched the literature and critically evaluated its quality” (Grant and Booth 2009). Given the rapid nature of this review, we only considered published, full-text studies. Abstracts, editorials, letters, commentaries, opinion pieces and case studies were excluded. Studies were limited to English language and those published from 2010 to 2020. This date range was chosen as systematic reviews before this period were expected to be too out of date for the purposes of our overview and would not be expected to reflect current best practice. As there is currently no clear definition outlining the standards for high quality systematic reviews in setting priorities for research, we considered any critical or systematic review that transparently defined how they searched and selected included studies.

Some of these reviews have a more descriptive approach to synthesizing reports of research priority setting, while others have evaluated and compared the methods used. We included all these types.

Types of outcomes

We expected reviews to measure and evaluate the success or effectiveness of a research priority setting exercise through either a process evaluation (focusing on whether the research priority setting activities were implemented as intended and resulted in certain outputs) or as an outcome evaluation (focusing on assessing the progress in the outcomes that the research priority setting activities are trying to address) (Nasser et al. 2013). We included outcomes of either type.

Data sources and search for studies

The review team identified search strategies utilized in relevant systematic reviews and piloted the strategy in Ovid MEDLINE®. The final search was peer-reviewed using the Peer-Review of Electronic Search Strategies (PRESS) guideline (McGowan et al. 2016) and the full electronic search strategy for MEDLINE can be found in “Appendix A”. We searched Ovid MEDLINE® and the WHO Institutional Repository for Information Sharing (IRIS). Vocabulary and syntax were adjusted across databases. Strategies utilized a combination of controlled vocabulary (e.g. review) and keywords (e.g. priority setting).

We adopted a pragmatic approach for conducting the search. Ovid MEDLINE® and the WHO Institutional Repository for Information Sharing (IRIS) were searched in May and June 2021, respectively. We expected that most of the research priority setting studies would be identified through this search. Vocabulary and syntax were adjusted across databases. Publications only after 2010 were considered using appropriate limit functions. Strategies utilized a combination of controlled vocabulary (e.g. review) and keywords (e.g. priority setting).

Additionally, we contacted methods experts in the WHO and Cochrane to identify any additional relevant material and hand-searched the reference lists of included studies.

Data management

The final search results the systematic reviews from all databases were imported into Reference Manager v12 (RefMan 2011), and duplicates were removed by (MN). De-duplicated results were imported into Rayyan, an online systematic review management software (Ouzzani et al. 2016).

Study selection

De-duplicated titles from Rayyan were then imported to Covidence for study selection. Title and abstract screening was performed by two authors (AT, SKN). To ensure a high level of agreement, AT and SKN performed a pilot screening of 20 titles. The minimum threshold agreement of 80% was met (90% agreement), so AT and SKN then divided the task of screening the records (non-blinded, single screening).

Full text screening was performed by two authors (AT, SKN). To ensure a high level of agreement, AT and SKN performed a pilot screening of 20 titles. The minimum threshold agreement of 80% was met (90% agreement), so AT and SKN then divided the task of screening the full texts (non-blinded, single screening).

If the same primary study was reported in multiple systematic reviews, the systematic reviews and data from the individual study were included in our synthesis regardless. While the issue of overlapping systematic reviews is a methodological concern for overviews of clinical systematic reviews, this issue is not as applicable for our review as we did not conduct meta-analyses and thus not affected by the issue of double counting of primary studies. If the same primary study is reported in multiple systematic reviews, the systematic reviews and data from the individual study was included in our synthesis regardless of whether it has been included in more than one systematic review. However, we performed a holistic assessment of overlapping systematic reviews to view how the interpretations and conclusions varied and whether this affected our own conclusions.

Data extraction

We drafted the data-extraction form using MS Excel 2016, based on consolidated reporting items identified in the REporting guideline for PRIority SEtting of health research (REPRISE) checklist, which contains ten domains covering: context and scope, governance and team, framework for priority setting, stakeholders/participants, identification and collection of priorities, prioritization of research topics, output, evaluation and feedback, translation and implementation, and funding and conflict of interest (Tong et al. 2019). Any assumptions or definitions are explained in the relevant sections of this review. Two reviewers (AT, SKN) pilot-tested the data-extraction template on one study and discussed the results with the wider team. AT and SKN worked in tandem to extract the data, discussed the results and continuously updated the data-extraction form in an iterative process (Levac et al. 2010). A list of major changes to the data-extraction form is provided as an appendix.

Quality assessment and synthesis

Quality assessment

The A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR 2) checklist assesses the quality of systematic reviews and is commonly used to assess the quality of systematic reviews on effectiveness of interventions (Shea et al. 2017). However, this checklist is less relevant to systematic reviews of priority setting methods as the structure of the research is different. For the purposes of this review, we developed an adapted version of the AMSTAR 2 checklist with guidance that was relevant for priority setting methods (see “Appendix B”). While the original 16 items were maintained, the questions were adapted so that the research priority setting method, as opposed to the intervention, was the focus. We built the adapted checklist based on the experience of the authors in the field, items identified in the REPRISE reporting guideline (Tong et al. 2019) and evaluation frameworks for priority setting (Nasser et al. 2020). The method for determining the overall confidence followed that of the standard AMSTAR 2 guidance with ratings from high (zero or one non-critical weakness), moderate (more than one non-critical weakness), low (one critical weakness with or without non-critical weaknesses) and critically low (more than one critical weakness with or without non-critical weaknesses).

Data synthesis

To manage the heterogeneity (in both focus and methods used) in the reviews, we first iteratively organized studies based on their main focus (i.e. engaging with stakeholders; methods; context; or health area). These themes were then further organized by geographical area and context (i.e. global, high-income countries [HICs], low- and middle-income countries [LMICs], specific countries/regions or no geographical specification) in recognition that the complexity of a priority setting process is dependent on the resources available. Studies were thematically analysed within these subthemes. We used the World Bank’s definition of HICs or LMICs as this was also the practice of many of the included studies. The synthesis was then used to develop an evidence-informed framework for research priority setting, which built on guidance included in four of the most recent, previously published frameworks (Fadlallah et al. 2020; Nasser et al. 2020; Viergever et al. 2010; World Health Organization 2020a).

Results

Results of search

The PRISMA flowchart of study selection (Fig. 1) shows the selection process of 2388 records identified from the literature search and an additional seven records that were identified through hand searching the bibliographies of included studies and consulting experts in the field. After duplicates were removed, two review authors (AT and SKN) excluded 1732 records at the title and abstract screening stage. AT and SKN assessed 69 full-text articles for eligibility and included 31 studies in this review. The reasons for exclusion are listed in “Appendix C” and included: duplicate studies, the core concept not focusing on research priority setting exercises or an ineligible study design.

Fig. 1
figure 1

PRISMA flowchart of study selection

The characteristics of included studies (Table 1) provides an overview of the studies’ characteristics. Of the 31 systematic reviews included in this overview, sixteen reviews (Alqahtani et al. 2021; Bragge et al. 2015; El-Harakeh et al. 2019; El-Harakeh et al. 2020; Fadallah et al. 2020; Garcia et al. 2017; Graham et al. 2020; Roche et al. 2021; Rudan et al. 2017; Rylance et al. 2010; Stewart et al. 2011; Tong et al. 2015; Tong et al. 2017; Viergever et al. 2010; Wade et al. 2021; Yoshida 2016) had a global scope. Additionally, five reviews (Cadorin et al. 2020; Hasson et al. 2020; Hawwash et al. 2018; Manafo et al. 2018; Mörelius et al., 2020) specifically focused on HICs, two reviews (McGregor et al. 2014; Tomlinson et al. 2011) focused on LMICs, five reviews (Badakhshan et al. 2018; Booth et al. 2018; Bryant et al. 2014; Garcia et al. 2015; Reveiz et al. 2013) focused on specific countries or regions of countries and three (Bourne et al. 2018; Odgers et al. 2018; Terry et al. 2018) did not specify geographic settings.

Table 1 Characteristics of included studies

Populations included in the reviews varied, but typically would involve patients and their representatives, families and carers, researchers, clinicians, policymakers and research funders. The topics covered in the reviews varied from specific diseases or conditions, approaches for healthcare practice or research priority setting methods itself. Most studies reported process-related outcomes. Only one review (Odgers et al. 2018) included studies that provided sufficient details on implementation and evaluation to enable the reporting of outcome evaluations. None of the studies were able to provide evidence regarding changes in healthcare practice.

Results of quality assessment

We used an adapted version of the AMSTAR 2 checklist for quality assessment. A full description of the results of the quality assessment are provided in “Appendix D”, and are also summarized here.

None of the included reviews were assessed to have high or moderate overall quality. Only two reviews (Tong et al. 2015; Cadorin et al. 2020) were assessed as low quality and the remaining 29 reviews were assessed to be of critically low quality overall. All the included reviews clearly reported the research questions and inclusion criteria for the review. However, more than 50% of the included reviews had the following issues that led to the low or critically low quality assessment:

  • Did not report if the review protocol was available or report any deviations from the protocol;

  • Did not extract data in duplicate, missing list of excluded studies with justifications for exclusions;

  • Lack of using satisfactory technique for assessing the quality of individual studies and the research priority setting process;

  • Did not report how sources of conflict of interest were managed in the included studies;

  • Did not consider quality of the studies when performing the synthesis, or in the interpretation or the discussion sections; and

  • Did not consider the impact of unpublished literature on the results of the priority setting process.

Results of data synthesis

The results of the thematic analysis of the data synthesis are summarized below. Themes are organized by the main focus of the studies (i.e. engaging with stakeholders; methods; context; or health area) and further organized by geographical area and context (i.e. global, HICs, LMICs, specific countries/regions or no geographical specification) in recognition that the complexity of a priority setting process is dependent on the resources available.

Identifying and engaging with stakeholders

Global

Two reviews (Stewart et al. 2011; Tong et al. 2017) showed that most studies reported collaborating with stakeholders, including patients, caregivers, physicians, allied health workers, researchers and policymakers to identify research questions. Stakeholders were engaged via surveys, telephone and face-to-face interviews, Delphi surveys and online and in-person forums. Tong et al. (2017) found that 32% (9/28) of studies identified differences between the priorities of patients/caregivers and health professionals and that only 4% (1/28) of studies indicated that stakeholder feedback was obtained and integrated into the proposed research priorities.

High-income countries

One review (Manafo et al. 2018) found that patient and public health involvement in setting research priorities resulted in subsequent studies focusing on these areas. However, operational details of the public involvement, cost, infrastructure and timelines are missing and make it difficult to replicate the process. The review found that local-level initiatives (versus regional or national) were most likely to ultimately impact patient-centredness and quality of care.

Assessed all stages: focus on methods

Global

One review (Yoshida 2016) identified that the two most used approaches were the Child Health Nutrition Research Initiative (CHNRI) (26%, 43 of 165 studies) and Delphi methods (24%, 40 of 165). The other methods (in order of usage) were—expert consultation, literature reviews, James Lind Alliance (JLA), and online surveys. Health care providers and researchers were well represented in most initiatives reviewed; however, policymakers, funders, and affected populations were far less involved (Fadlallah et al. 2020). Few studies addressed translation of the priority topics into research questions or implementation and evaluation plans.

High-income countries

One review (Hawwash et al. 2018) assessed research priority setting methods in high-income countries and found that a diverse range of methods were used, including the Delphi method and CHNRI method. Many of the studies did not describe the stakeholders involved or the follow-up activities of the proposed activities.

Guidelines (clinical, public health, health systems)

Three reviews (El-Harakeh et al. 2019, 2020; Garcia et al. 2017) found that the most frequently reported criteria for prioritizing approaches for guideline development were the health burden of the disease; available evidence; potential impact of the intervention on health outcomes; and users’ interest. While health care providers were often involved in the prioritization exercises, very few involved patients. Generally, the methods for institutionalizing and implementing the prioritization processes were varied and poorly reported.

World Health Organization (WHO)

Two studies (Viergever et al. 2010; Terry et al. 2018) examined priority setting methods within the WHO. Terry et al. (2018) found that the most common research priority setting method was expert consultation, sometimes in conjunction with a literature review, but almost 70% of the identified research priorities were developed without using any additional criteria to rank the priorities with respect to potential health impact, feasibility or cost. Viergever et al. (2010) developed nine common themes of good practice in health research priority setting, which were categorized into three domains—preparation phase; methods for deciding upon priorities; and work performed after priorities have been set, which highlighted the importance of transparently reporting and disseminating the results of such work.

Assessed all stages: focused on context

Region of the Americas

One review (Garcia et al. 2015) found that more than half of the priority setting documents did not have clear selection criteria and most did not have an implementation or evaluation plan.

High-income countries

Two reviews (Bryant et al. 2014; Hasson et al. 2020) highlighted the lack of consistent reporting and evaluation of priority setting processes and recommended that a multi-disciplinary advisory group should oversee the priority setting process; broad representation of stakeholders is critical; objective, clearly defined criteria should guide the generation of priorities; and the impact of the priority setting processes should be evaluated. Specific focus needs to be placed on elevating the voice of patients to enhance the validity of identified priorities (Hasson et al. 2020).

Low and middle-income countries

Four reviews (Badakhshan et al. 2018; McGregor et al. 2014; Tomlinson et al. 2011; Reveiz et al. 2013) recommended incorporating mechanisms for disseminating priority setting results, creating implementation plans and processes for revising priorities, and engaging stakeholders. Importantly, these reviews suggested establishing a communication channel with neighbouring countries about the priority setting process. The reviews recommended establishing regional health research agendas, harmonizing research setting approaches to enable greater comparability and strengthening collaboration between groups of researchers sharing the same interests.

No geographic specification

Rudan et al. (2017) did not have a specific geographic focus but examined studies that specifically used the CHNRI method. It found that from 2016 onwards it was often adapted to suit respective contexts. It was used in a range of different health fields due to its: systematic and democratic nature, acceptable framework to handle many research questions, transparency and replicability, clear definition of the context and priority setting criteria, adaptability, and ease of conduct.

Assessed all stages: focus on health area

High-income countries

Three reviews (Booth et al. 2018; Cadorin et al. 2020; Mörelius et al., 2020) found that consensus building methods, such as Delphi processes, multi-staged surveys and focus groups, were commonly used to develop priorities. However, a key limitation of many studies was the lack of engagement with patients and carers or professional groups outside of healthcare sectors, such as social workers and teachers, in the research prioritization exercise. Where sought, only minimal involvement was secured. This was concerning given that priorities of patients often differ from those of clinicians and the priority setting processes did not reflect on whether the priorities identified reflected the priorities of the key consumers of the healthcare services (Booth et al. 2018; Cadorin et al. 2020; Mörelius et al., 2020). Integrating qualitative studies involving patients in focus or discussion groups to share the results of the literature and establish with them the main priorities would be a starting point to develop a new research priority agenda that can be adapted locally according to each context (Cadorin et al. 2020). This would involve going beyond lists of priorities to explore what these priorities mean for patients and consumers. In addition, greater collaboration with other professionals is desirable (Cadorin et al. 2020).

Two studies (Cadorin et al. 2020; Mörelius et al., 2020) discussed practical aspects of priority setting processes. Considering the economic and financial context in which the priority setting processes are situated is important for increasing the likelihood that research priorities will be taken forward. Additionally, Mörelius et al. (2020) was the only study that highlighted the need to improve the actual implementation of evidence into practice.

Global

Commonly used methods for developing priorities included surveys, interviews or focus groups with consumers and stakeholders (Alqahtani et al. 2021; Roche et al. 2021). Methods for recruiting participants included through support groups, research networks, websites, social media, special interest groups, conferences and contacts of the research team (Roche et al. 2021).

Seven reviews (Bourne et al. 2018; Bragge et al. 2015; Graham et al. 2020; Odgers et al. 2018; Roche et al. 2021; Rylance et al. 2010; Tong et al. 2015) recommended that future research priority setting initiatives have a clear aim, use robust methods and include all relevant stakeholders. Future studies could improve the robustness of methods by utilizing good practice guidelines in research and reporting of priority setting, such as those proposed by the JLA, WHO (2020a) or Viergever et al. (2010). Integrated research strategies that incorporate a co-design or participatory research perspective have the potential to drive better outcomes (Roche et al. 2021; Wade et al. 2021). Consumers should be involved at every stage of the priority setting process, especially when determining how to implement the priorities to improve future practice (Wade et al. 2021). Roche et al. (2021) underscored this need by questioning whether priorities identified by healthcare professionals, researchers etc. are truly reflective of the target communities and people with lived experience. Using a ‘bottom up’ approach where consumers can express their priorities in their own words would increase the relevance of priorities (Roche et al. 2021).

Strategies for implementing, institutionalizing and measuring the impact of the research priorities included liaising with key stakeholders, disseminating priorities through key organizations and monitoring the impact of priorities on grant applications and grants awarded (Odgers et al. 2018). Bragge et al. (2015) reported that the findings from their review subsequently informed the development of a regional research strategy.

Notably, two of the most recently published reviews (Roche et al. 2021; Wade et al. 2021) highlighted that translation of research priorities across community or international settings will also need to account for societal and cultural differences.

Discussion

Summary of main findings

This study identified, appraised and synthesized the systematic reviews in the recent literature on research priority setting to propose a draft evidence-informed framework for countries to consider while developing their national research agendas. The findings of our review reflected those in other systematic reviews in the field. Like other evidence syntheses (Hanney et al. 2020), this overview highlighted the value and importance of establishing evidence-informed priority setting structures for the development of research agendas. Involving a diverse range of stakeholders in the priority setting process can ‘increase the legitimacy, credibility, transparency, and acceptability of the identified priorities’ (Fadlallah et al. 2020). However, engaging with stakeholders is challenging in terms of resources, capacity and feasibility (Fadlallah et al. 2020).

As was found by Sharma et al. (2018) and others, many priority setting documents do not clearly or consistently report methods (Garcia et al. 2015; Bryant et al. 2014; Hasson et al. 2020). Commonly used priority setting approaches included formalized methods such as the CHNRI and Delphi approaches, and informal approaches, such as expert consultation. Having an implementation and evaluation strategy could assist in translating the results of the priority setting exercises into research (Manafo et al. 2018; Tong et al. 2017).

The WHO has recently reviewed and updated methodologies related to health research priority setting and developed guidance for its staff and the research priority setting exercises for which the WHO has responsibility (World Health Organization 2020a). The Guide, A Systematic Approach for Undertaking a Research Priority-Setting Exercise, contains a framework for developing research priority setting exercises. This Guide, which was initially developed for use amongst WHO Headquarter staff, appears to be mainly based on good practice examples and methodologies already in use across the WHO. In comparison, our overview used a systematic process to identify literature published within the last decade on health research priority setting methods and used a transparent and reproducible approach to create a framework.

Our framework draws from four existing frameworks (Fadlallah et al. 2020; Nasser et al. 2020; Viergever et al. 2010; World Health Organization 2020a) and incorporates evidence from the thematic analysis of the included reviews to create a comprehensive approach for countries to develop future research agendas. The priority setting framework (Fig. 2) demonstrates the preliminary framework with four key phases: (1) preparatory, (2) priority setting, (3) follow-up phase and (4) sustainability phase.

Fig. 2
figure 2

Priority setting framework. The gradient of grey reflects the frequency with which domains have been considered in existing frameworks, with dark grey representing frequent consideration to white highlighting novel elements identified in the current overview. CHNRI: Child Health Nutrition Research Initiative; JLA: James Lind Alliance; KT: knowledge translation

Preparatory phase

Four existing frameworks (Fadlallah et al. 2020; Nasser et al. 2020; Viergever et al. 2010; World Health Organization 2020a) note the importance of including relevant stakeholders in the process and having a participatory approach, which is also valued as a key component in most of the included reviews in our study. Additionally, our overview highlighted the importance of having a bottom-up approach where researchers work with consumers in a co-design and co-production process so that resulting priorities are reflective of patient needs (Cadorin et al. 2020; Roche et al. 2021; Wade et al. 2021). Only one of the previous frameworks (Nasser et al. 2020) highlights the importance of having a dedicated team to lead the priority setting process.

While only one of the previous frameworks (World Health Organization 2020a) highlighted the need to outline the objective of the priority setting process, all our included reviews indicated the objective of their review, which was then used to guide their inclusion of their primary studies and synthesis. Many of the included reviews also discussed the need for a formal search of the background literature to situate the priority setting within the current literature base and this contextualization was also noted by all four of the existing frameworks (Nasser et al. 2020; Viergever et al. 2010; Fadlallah et al. 2020; World Health Organization 2020a),

Priority setting phase

Our overview found that included studies usually used a combination of mechanisms, such as surveys, workshops and expert consultation, often as part of formal processes for determining priorities, such as CHNRI, Delphi and JLA methods (Cowan and Oliver 2013; Fadlallah et al. 2020; Yoshida 2016). The existing frameworks do not clearly outline what methods should be followed when setting priorities, nor do they discuss the importance of integrating qualitative with quantitative data to understand the reasoning behind stakeholders’ rankings (Roche et al. 2021; Wade et al. 2021). The qualitative evidence gained through focus groups and interviews can provide valuable insights into the lived experiences of patients and consumers to contextualize quantitative rankings of priorities and should be included into processes (Roche et al. 2021; Wade et al. 2021).

The importance of reporting the findings is seen in two existing frameworks (Nasser et al. 2020; World Health Organization 2020a). From our overview we know that there is a lack of consistent reporting and the authors of the reviews recommend having multi-disciplinary advisory groups to oversee the whole process (Bryant et al. 2014; Hasson et al. 2020). It was also noted that often the costs, mechanisms and details of patient and public participation are not reported, making it difficult for replication (Manafo et al. 2018), though some advice around recruitment is available from one review (Roche et al. 2021). We also suspect the presence of non-reporting, in addition to poor reporting, which could be improved by following guidance for appropriate, systematic and transparent reporting of priority setting practices, such as the REPRISE checklist (Tong et al. 2019).

Follow-up phase

Two of the existing frameworks (Fadlallah et al. 2020; World Health Organization 2020a) and our overview discuss the importance of having a structured dissemination strategy that shares the priorities with key organizations and stakeholders to facilitate their subsequent implementation (Badakhshan et al. 2018; McGregor et al. 2014; Tomlinson et al. 2011; Reveiz et al. 2013; Odgers et al. 2018). Our overview also highlighted that the translation of these research priorities should account for societal and cultural differences as they are translated across different communities and settings (Roche et al. 2021; Wade et al. 2021). The evaluation of the priority setting process was considered in all four of the existing frameworks (Nasser et al. 2020; Viergever et al. 2010; Fadlallah et al. 2020; World Health Organization 2020a) as it was in our included studies (Bryant et al. 2014; Hasson et al. 2020). Two of the frameworks (Nasser et al. 2020; World Health Organization 2020a) consider some elements of research funding cycles within their feedback and evaluation domains. Our overview noted that the impact of priorities should be monitored by assessing whether priorities are reflected in grants being awarded (Bragge et al. 2015; Odgers et al. 2018).

Sustainability phase

The most notable difference between our framework and that of WHO (2020a) is our sustainability phase, which includes considerations for developing mechanisms for institutionalizing the priority setting process into the research cycle and creating peer-learning amongst colleagues and regions. This element of sustainability is crucial for ensuring that policymaking is underpinned by prioritized research and that the priority setting process can build a wider community of practice. One included review (Odgers et al. 2018) highlights the value of institutionalization (i.e. ensuring that priority setting processes are cyclical, as opposed to singular, events within an organization). Two reviews (Cadorin et al. 2020; Mörelius et al., 2020) also noted the importance of economic aspects of priority setting processes to ensure that an environment or context is made whereby research priorities are taken forward. The four LMIC reviews (Badakhshan et al. 2018; McGregor et al. 2014; Tomlinson et al. 2011; Reveiz et al. 2013) recommended establishing communication channels with neighbouring countries to share best practices, establish regional health research agendas where possible, harmonize priority setting processes to enable greater comparability and strengthen collaboration.

Methodological strengths and limitations

A rapid review approach requires that components of the systematic review process are simplified or omitted to produce information in a timely manner and are increasingly used when facing resource and time constraints (Tricco et al. 2015). One of the limitations of this overview is that we only searched in two databases. However, the searches and the experts we contacted in our network identified all the relevant studies included in this overview. Due to the limited use of indexing for studies related to priority setting, a broader search strategy would have increased the number of irrelevant results to a degree that the review would have been too resource intensive to complete. Moreover, there is no indication that our approach to searching introduced any meta-biases. While it is theoretically possible that additional studies may have been conducted but not yet published, or that additional studies may not have been identified, this is unlikely.

While we did not conduct an extensive search of the grey literature as this was outside of the scope of our rapid review, our search in the WHO IRIS database captured relevant WHO reports, which is the most relevant grey literature for the context of our review. We also searched the reference lists of included studies to ensure we had captured all relevant reviews. Additionally, several of the included studies in our review, such as Viergever et al. (2010) included a search of the grey literature. Though study selection was not completed fully in a double-blind manner, the authors performed a blinded-dual screen of a portion of the studies to ensure a high degree of agreement (> 80%) before completing the unblended screening on the rest of the records.

A key strength of this study is that this is the first review of its kind that the authors are aware of that uses systematic review evidence to underpin and develop a framework for national research priority setting in a systematic and transparent manner. Notably, this review also created an adapted AMSTAR tool, which enabled us to perform a quality assessment. There are currently no standard methods for conducting priority setting methods, so our overview provides an assessment of the most recently used methods and offers guidance on the conduct of a systematic review of priority setting methods.

Future research

Future studies should confirm and consolidate the framework developed in this review with relevant stakeholders. Further work and validation would be needed to ensure its applicability and appropriateness at a national level. A process evaluation could help assess the extent to which the priorities generated have informed research or research funding, or the extent to which they have been adopted into the field. More research is needed into the practicalities of engaging stakeholders in priority setting exercises.

Differences with protocol

The authors intended to base the data extraction form on consolidated reporting items identified in the REporting guideline for PRIority SEtting of health research (REPRISE) checklist (Tong et al. 2019), but instead used the framework developed by Nasser et al. (2020), which the data fit better. After our overview was completed, the WHO headquarters published their priority setting guidance, A Systematic Approach for Undertaking a Research Priority-Setting Exercise (World Health Organization 2020a). To acknowledge this report and remain transparent, we compared our findings and recommendations with those they recommended in the Discussion section.

Conclusions

Research priority setting is important for strengthening health systems and evidence-informed policies; countries and regions need to invest further within this field. As health systems experience continued pressures and constraints, it is essential that health research funds are allocated in an evidence-informed, transparent and structured manner. Our overview of reviews has reconfirmed aspects of existing frameworks, but has also identified new concepts that we have integrated either as new elements or revisions to existing ones. Notably, our overview has identified that the integration of priority setting exercises into local, national or international level decision making can support sustainability as, currently, some key lessons on how to set priorities for research get lost in administrational, organizational and staff changes. Our overview presents a draft framework and methodological tools to support countries in answering questions that are of value to society and whose outcomes inform policies that improve the health and well-being of populations.

Availability of data and materials

The datasets supporting the conclusions of this article are included within the article and its additional files.

Abbreviations

AMSTAR 2:

A MeaSurement Tool to Assess systematic Reviews 2

CHNRI:

Child health nutrition research initiative

HIC:

High-income country

IRIS:

Institutional repository for information sharing

JLA:

James Lind alliance

LMIC:

Low- and middle-income country

PRESS:

Peer-Review of Electronic Search Strategies

PRISMA-P:

Preferred reporting items for systematic reviews and meta-analysis protocols

REPRISE:

Reporting guideline for PRIority Setting

WHO:

World health organization

References

  • Alqahtani S, Joseph J, Dicianno B, Layton NA, Toro ML, Ferretti E, Tuakli-Wosornu YA, Chhabra H, Neyedli H, Lopes CR, Alqahtani MM, Van de Vliet P, Kumagaya SI, Kim JB, McKinney V, Yang YS, Goldberg M, Cooper R (2021) Stakeholder perspectives on research and development priorities for mobility assistive-technology: a literature review. Disabil Rehabil Assist Technol 16:362–376

    Article  PubMed  Google Scholar 

  • Badakhshan A, Arab M, Rashidian A, Gholipour M, Mohebbi E, Zendehdel K (2018) Systematic review of priority setting studies in health research in the Islamic Republic of Iran. East Mediterr Health J 24:753–769

    Article  PubMed  Google Scholar 

  • Booth A, Maddison J, Wright K, Fraser L, Beresford B (2018) Research prioritisation exercises related to the care of children and young people with life-limiting conditions, their parents and all those who care for them: a systematic scoping review. Palliative Med 32:1552–1566

    Article  Google Scholar 

  • Bourne AM, Johnston RV, Cyril S, Briggs AM, Clavisi O, Duque G, Harris IA, Hill C, Hiller C, Kamper SJ, Latimer J, Lawson A, Lin CC, Maher C, Perriman D, Richards BL, Smitham P, Taylor WJ, Whittle S, Buchbinder R (2018) Scoping review of priority setting of research topics for musculoskeletal conditions. BMJ Open 8:e023962

    Article  PubMed  PubMed Central  Google Scholar 

  • Bragge P, Piccenna L, Middleton J, Williams S, Creasey G, Dunlop S, Brown D, Gruen R (2015) Developing a spinal cord injury research strategy using a structured process of evidence review and stakeholder dialogue. Part II: background to a research strategy. Spinal Cord 53:721–728

    Article  CAS  PubMed  Google Scholar 

  • Bryant J, Sanson-Fisher R, Walsh J, Stewart J (2014) Health research priority setting in selected high income countries: a narrative review of methods used and recommendations for future practice. Cost Eff Resour Alloc 12:1–11

    Article  Google Scholar 

  • Cadorin L, Bressan V, Truccolo I, Suter N (2020) Priorities for cancer research from the viewpoints of cancer nurses and cancer patients: a mixed-method systematic review. Cancer Nurs 43:238–256

    Article  PubMed  Google Scholar 

  • Chanda-Kapata P, Ngosa W, Hamainza B, Kapiriri L (2016) Health research priority setting in Zambia: a stock taking of approaches conducted from 1998 to 2015. Health Res Policy Syst 14:1–9

    Article  Google Scholar 

  • Cowan K, Oliver S (2013) The James Lind Alliance guidebook. National Institute for Health Research Evaluation, Trials and Studies Coordinating Centre, Southampton

  • El-Harakeh A, Lotfi T, Ahmad A, Morsi RZ, Fadlallah R, Bou-Karroum L, Akl EA (2020) The implementation of prioritization exercises in the development and update of health practice guidelines: a scoping review. PLoS ONE 15:e0229249

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • El-Harakeh A, Morsi RZ, Fadlallah R, Bou-Karroum L, Lotfi T, Akl EA (2019) Prioritization approaches in the development of health practice guidelines: a systematic review. BMC Health Serv Res 19:692–692

    Article  PubMed  PubMed Central  Google Scholar 

  • Fadlallah R, El-Harakeh A, Bou-Karroum L, Lotfi T, El-Jardali F, Hishi L, Akl EA (2020) A common framework of steps and criteria for prioritizing topics for evidence syntheses: a systematic review. J Clin Epidemiol 120:67–85

    Article  PubMed  Google Scholar 

  • Garcia AB, Cassiani SHDB, Reveiz L (2015) A systematic review of nursing research priorities on health system and services in the Americas. Rev Panam Salud Publica (pan American Journal of Public Health) 37:162–171

    Google Scholar 

  • Garcia LM, Pardo-Hernandez H, Superchi C, Nino de Guzman E, Ballesteros M, Ibargoyen Roteta N, McFarlane E, Posso M, Roque IFM, Rotaeche Del Campo R, Sanabria AJ, Selva A, Sola I, Vernooij RWM, Alonso-Coello P (2017) Methodological systematic review identifies major limitations in prioritization processes for updating. J Clin Epidemiol 86:11–24

    Article  Google Scholar 

  • Ghaffar A, Collins T, Matlin SA, Olifson S (2009) The 3D combined approach matrix: an improved tool for setting priorities in research for health. In: Global forum for health research. Global Forum for Health Research, Geneva.

  • Graham L, Illingworth B, Showell M, Vercoe M, Crosbie EJ, Gingel LJ, Farquhar CM, Horne AW, Prior M, Stephenson JM, Magee LA, Duffy J (2020) Research priority setting in women’s health: a systematic review. BJOG 127:694–700

    Article  CAS  PubMed  Google Scholar 

  • Grant MJ, Booth A (2009) A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 26:91–108

    Article  PubMed  Google Scholar 

  • Hanney S, Kanya L, Pokhrel S, Jones T, Boaz A (2020) What is the evidence on policies, interventions and tools for establishing and/or strengthening national health research systems and their effectiveness? World Health Organization Regional Office for Europe, Copenhagen

    Google Scholar 

  • Hasson F, Nicholson E, Muldrew D, Bamidele O, Payne S, McIlfatrick S (2020) International palliative care research priorities: a systematic review. BMC Palliat Care 19:1–16

    Article  Google Scholar 

  • Hawwash D, Pinxten W, Bonn NA, Verstraeten R, Kolsteren P, Lachat C (2018) Perspective: consideration of values when setting priorities in nutrition research: guidance for transparency. Adv Nutr 9:671–687

    Article  PubMed  PubMed Central  Google Scholar 

  • Levac D, Colquhoun H, O’Brien KK (2010) Scoping studies: advancing the methodology. Implement Sci 5:1–9

    Article  Google Scholar 

  • Manafo E, Petermann L, Vandall-Walker V, Mason-Lai P (2018) Patient and public engagement in priority setting: a systematic rapid review of the literature. PLoS ONE 13:e0193579

    Article  PubMed  PubMed Central  Google Scholar 

  • McGettigan P, Henry D (2011) Cardiovascular risk with non-steroidal anti-inflammatory drugs: systematic review of population-based controlled observational studies. PLoS Med 8:e1001098

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C (2016) PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol 75:40–46

    Article  PubMed  Google Scholar 

  • McGregor S, Henderson KJ, Kaldor JM (2014) How are health research priorities set in low and middle income countries? A systematic review of published reports. PLoS ONE 9:e108787

    Article  ADS  PubMed  PubMed Central  Google Scholar 

  • Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4:1–9

    Article  PubMed  PubMed Central  Google Scholar 

  • Mörelius E, Foster M, Gill FJ (2020) A scoping review of nursing research priorities in pediatric care. J Pediatr Nurs 52:e57–e69

    Article  PubMed  Google Scholar 

  • Nasser M, Welch V, Tugwell P, Ueffing E, Doyle J, Waters E (2013) Ensuring relevance for Cochrane reviews: evaluating processes and methods for prioritizing topics for Cochrane reviews. J Clin Epidemiol 66:474–482

    Article  PubMed  Google Scholar 

  • Nasser M, Viergever RF, Martin J (2020) Chapter 2.7 Research prioritization. In WHO Guidance on Research Methods for Health Emergency and Disaster Risk Management (under consideration)

  • Odgers HL, Tong A, Lopez-Vargas P, Davidson A, Jaffe A, McKenzie A, Pinkerton R, Wake M, Richmond P, Crowe S, Caldwell PHY, Hill S, Couper J, Haddad S, Kassai B, Craig JC (2018) Research priority setting in childhood chronic disease: a systematic review. Arch Dis Child 103:942–951

    Article  PubMed  Google Scholar 

  • Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A (2016) Rayyan - a web and mobile app for systematic reviews. Syst Rev 5:1–10

    Article  Google Scholar 

  • Pollock M, Fernandes RM, Becker LA, Featherstone R, Hartling L (2016) What guidance is available for researchers’ conducting overviews of reviews of healthcare interventions? A scoping review and qualitative meta-summary. Syst Rev 5:1–15

    Article  Google Scholar 

  • Reference Manager 12 (2011) Thomson Reuters, New York

  • Reveiz L, Elias V, Terry RF, Alger J, Becerra-Posada F (2013) Comparison of national health research priority-setting methods and characteristics in Latin America and the Caribbean, 2002–2012. Rev Panam Salud Publica 34:1–13

    PubMed  Google Scholar 

  • Roche L, Adams D, Clark M (2021) Research priorities of the autism community: a systematic review of key stakeholder perspectives. Autism 25:336–348

    Article  PubMed  Google Scholar 

  • Rudan I, Yoshida S, Chan KY, Sridhar D, Wazny K, Nair H, Sheikh A, Tomlinson M, Lawn JE, Bhutta ZA, Bahl R, Chopra M, Campbell H, El Arifeen S, Black RE, Cousens S (2017) Setting health research priorities using the CHNRI method: VII. A review of the first 50 applications of the CHNRI method. J Glob Health 7:1–7

    Article  Google Scholar 

  • Rylance J, Pai M, Lienhardt C, Garner P (2010) Priorities for tuberculosis research: a systematic review. Lancet Infect Dis 10:886–892

    Article  PubMed  Google Scholar 

  • Sharma T, Choudhury M, Rejón-Parrilla JC, Jonsson P, Garner S (2018) Using HTA and guideline development as a tool for research priority setting the NICE way: reducing research waste by identifying the right research to fund. BMJ Open 8:e019777

    Article  PubMed  PubMed Central  Google Scholar 

  • Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008

    Article  PubMed  PubMed Central  Google Scholar 

  • Stewart RJ, Caird J, Oliver K, Oliver S (2011) Patients’ and clinicians’ research priorities. Health Expect 14:439–448

    Article  PubMed  Google Scholar 

  • Terry RF, Charles E, Purdy B, Sanford A (2018) An analysis of research priority-setting at the World Health Organization - How mapping to a standard template allows for comparison between research priority-setting approaches. Health Res Policy Syst 16:1–11

    Article  Google Scholar 

  • Tomlinson M, Chopra M, Hoosain N, Rudan I (2011) A review of selected research priority setting processes at national level in low- and middle-income countries: towards fair and legitimate priority setting. Health Res Policy Syst 9:1–7

    Article  Google Scholar 

  • Tong A, Chando S, Crowe S, Manns B, Winkelmayer WC, Hemmelgarn B, Craig JC (2015) Research priority setting in kidney disease: a systematic review. Am J Kidney Dis 65:674–683

    Article  PubMed  Google Scholar 

  • Tong A, Sautenet B, Chapman JR, Harper C, MacDonald P, Shackel N, Crowe S, Hanson C, Hill S, Synnot A, Craig JC (2017) Research priority setting in organ transplantation: a systematic review. Transpl Int 30:327–343

    Article  PubMed  Google Scholar 

  • Tong A, Synnot A, Crowe S, Hill S, Matus A, Scholes-Robertson N, Oliver S, Cowan K, Nasser M, Bhaumik S, Gutman T (2019) Reporting guideline for priority setting of health research (REPRISE). BMC Med Res Methodol 19:1–11

    Article  CAS  Google Scholar 

  • Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, Perrier L, Hutton B, Moher D, Straus SE (2015) A scoping review of rapid review methods. BMC Med 13:1–15

    Article  Google Scholar 

  • Viergever RF, Olifson S, Ghaffar A, Terry RF (2010) A checklist for health research priority setting: nine common themes of good practice. Health Res Policy Syst 8:1–9

    Article  Google Scholar 

  • Wade TD, Hart LM, Mitchison D, Hay P (2021) Driving better intervention outcomes in eating disorders: a systematic synthesis of research priority setting and the involvement of consumer input. Eur Eat Disord Rev 29:346–354

    Article  PubMed  Google Scholar 

  • World Health Organization (2020a) A systematic approach for undertaking a research priority-setting exercise: guidance for WHO staff. https://apps.who.int/iris/handle/10665/334408. Accessed 22 June 2020

  • World Health Organization (2020b) WHO to accelerate research and innovation for new coronavirus. Retrieved from https://www.who.int/news-room/detail/06-02-2020-who-to-accelerate-research-and-innovation-for-new-coronavirus. Accessed 22 June 2020

  • Yoshida S (2016) Approaches, tools and methods used for setting priorities in health research in the 21st century. J Glob Health 6:1–10

Excluded Studies

  • Abramowitz SA, Hipgrave DB, Witchard A, Heymann DL (2018) Lessons from the West Africa Ebola epidemic: a systematic review of epidemiological and social and behavioral science research priorities. L Infect Dis 218:1730–1738

    Google Scholar 

  • Anstee S, Price A, Young A, Barnard K, Coates B, Fraser S, Moran R (2011) Developing a matrix to identify and prioritise research recommendations in HIV Prevention. BMC Pub Health 11:1–8

    Article  Google Scholar 

  • Barnes AP, Moxey AP, Ahmadi BV, Borthwick FA (2015) The effect of animal health compensation on ‘positive’ behaviours towards exotic disease reporting and implementing biosecurity: a review, a synthesis and a research agenda. Prev Vet Med 122:42–52

    Article  PubMed  Google Scholar 

  • Bassetti CL, Ferini-Strambi L, Brown S, Adamantidi A, Benedetti F, Bruni O, Cajochen C, Dolenc-Groselj L, Ferri R, Gais S, Huber R (2015) Neurology and psychiatry: waking up to opportunities of sleep. State of the art and clinical/research priorities for the next decade. Eur J Neurol 22:1337–1354

    Article  CAS  PubMed  Google Scholar 

  • Buchholz SW, Klein T, Cooke C, Cook ML, Knestrick J, Dickins K (2018) Introduction to four reviews addressing critical topics identified by the 2015 Nurse Practitioner Research Agenda Roundtable: priorities for policy, workforce, education, and practice. J Am Assoc Nurse Pra 30:667–672

    Article  Google Scholar 

  • Consortium from Altarum Institute, Centers for Disease Control and Prevention, Robert Wood Johnson Foundation, National Coordinating Center for Public Health Services and Systems Research (2012) A national research agenda for public health services and systems. Am J Prev Med 42:S72–S88

  • Foundation RWJ, Centers for Disease Control and Prevention (2012) A national research agenda for public health services and systems. Am J Prev Med 42:S72–S78

  • Crewdson K, Rehn M, Lockey D (2018) Airway management in pre-hospital critical care: a review of the evidence for a ‘top five ’research priority. Scan J Trauma Resus 26:1–6

    Google Scholar 

  • Cumpston MS, Tavender EJ, Buchan HA, Gruen RL (2012) Australian health policy makers’ priorities for research synthesis: a survey. Aust Health Rev 36:401–141

    Article  PubMed  Google Scholar 

  • Davis J, Morgans A, Stewart J (2015) Developing an Australian health and aged care research agenda: a systematic review of evidence at the subacute interface. Aust Health Rev 40:420–427

    Article  Google Scholar 

  • Delost ME, Nadder TS (2014) Guidelines for initiating a research agenda: topic selection and evidence of impact. Clin Lab Sci 27:231–236

    Article  PubMed  Google Scholar 

  • Doosti-Irani A, Holakouie-Naieni K (2016) Determination the research priorities in the field of HIV/AIDS in Iran: a systematic review article. Iran J Public Health 45:1149

    PubMed  PubMed Central  Google Scholar 

  • Fabbri A, Lai A, Grundy Q, Bero LA (2018) The influence of industry sponsorship on the research agenda: a scoping review. Am J Public Health 108:e9–e16

    Article  PubMed  PubMed Central  Google Scholar 

  • Foster J, Bautista C, Ellstrom K, Kalowes P, Manning J, Pasek TA (2018) Creating a research agenda and setting research priorities for clinical nurse specialists. Clin Nurse Spec 32:21–28

    Article  PubMed  Google Scholar 

  • Hill B, Skouteris H, Teede HJ, Bailey C, Baxter JAB, Bergmeier HJ, Borges ALV, Harrison CL, Jack B, Jorgensen L, Lim S (2019) Health in Preconception, Pregnancy and Postpartum Global Alliance: international network preconception research priorities for the prevention of maternal obesity and related pregnancy and long-term complications. J Clin Sci 8:2119

    Google Scholar 

  • Johnston MJ, Paige JT, Aggarwal R, Stefanidis D, Tsuda S, Khajuria A, Arora S, Association for Surgical Education Simulation Committee (2016) An overview of research priorities in surgical simulation: what the literature shows has been achieved during the 21st century and what remains. Am J Surg 211:214–225

  • Jones AC, Geneau R (2012) Assessing research activity on priority interventions for non-communicable disease prevention in low-and middle-income countries: a bibliometric analysis. Glob Health Action 5:18847

    Article  Google Scholar 

  • Karimkhani C, Trikha R, Aksut B, Jones T, Boyers LN, Schlichte M, Pederson H, Okland T, DiGuiseppi C, Nasser M, Naghavi M (2016) Identifying gaps for research prioritisation: global burden of external causes of injury as reflected in the Cochrane Database of Systematic Reviews. Injury 47:1151–1157

    Article  PubMed  PubMed Central  Google Scholar 

  • Knight R, Small W, Pakula B, Thomson K, Shoveller J (2014) A scoping study to identify opportunities to advance the ethical implementation and scale-up of HIV treatment as prevention: priorities for empirical research. BMC Med Ethics 15:1–13

    Article  Google Scholar 

  • Kong G, Creamer MR, Simon P, Cavallo DA, Ross JC, Hinds JT, Fishbein H, Gutierrez K (2019) Systematic review of cigars, cigarillos, and little cigars among adolescents: setting research agenda to inform tobacco control policy. Addict Behav 96:192–197

    Article  PubMed  PubMed Central  Google Scholar 

  • Kühne F, Brütt AL, Otterbeck MJ, Weck F (2021) Research priorities set by people with OCD and OCD researchers: do the commonalities outweigh the differences? Health Expect 24:40–46

    Article  PubMed  Google Scholar 

  • Morton MJ, DeAugustinis ML, Velasquez CA, Singh S, Kelen GD (2015) Developments in surge research priorities: a systematic review of the literature following the academic emergency medicine consensus conference, 2007–2015. Adac Emerg Med 22:1235–1252

    Article  Google Scholar 

  • Nicolau I, Ling D, Tian L, Lienhardt C, Pai M (2012) Research questions and priorities for tuberculosis: a survey of published systematic reviews and meta-analyses. PLoS ONE e42479

  • Okland T, Karimkhani C, Pederson H, Boyers LN, Sawyer MD, Rove KO, Kenny C, Steinberg S, Naghavi M, Dellavalle RP (2017) Research prioritization of men’s health and urologic diseases. Int Braz J Urol 43:289–303

    Article  PubMed  PubMed Central  Google Scholar 

  • Oncology Nursing Forum (2019) Research agenda of the oncology nursing society: 2019–2022. Oncol Nurs Forum 46:654–669

  • Pozzar RA, Berry DL (2017) Patient-centered research priorities in ovarian cancer: a systematic review of potential determinants of guideline care. Gynecol Oncol 147:714–722

    Article  CAS  PubMed  Google Scholar 

  • Pratt B (2020) Developing a toolkit for engagement practice: sharing power with communities in priority-setting for global health research projects. BMC Med Ethics 21:1–14

    Article  Google Scholar 

  • Sebastianski M, Gates M, Gates A, Nuspl M, Bialy LM, Featherstone RM, Breault L, Mason-Lai P, Hartling L (2019) Evidence available for patient-identified priorities in depression research: results of 11 rapid responses. BMJ Open 9:e026847

    Article  PubMed  PubMed Central  Google Scholar 

  • Sigfrid L, Moore C, Salam AP, Maayan N, Hamel C, Garritty C, Lutje V, Buckley B, Soares-Weiser K, Marshall R, Clarke M (2019) A rapid research needs appraisal methodology to identify evidence gaps to inform clinical research priorities in response to outbreaks-results from the Lassa fever pilot. BMC Med 17:1–17

    Article  CAS  Google Scholar 

  • Van Royen P, Beyer M, Chevallier P, Eilat-Tsanani S, Lionis C, Peremans L, Petek D, Rurik I, SolerJK, Stoffers HE, Topsever P (2010) Series: the research agenda for general practice/family medicine and primary health care in Europe. Part 5: needs and implications for future research and policy. Eur J Gen Pract 16:244–8

    Article  PubMed  Google Scholar 

  • Von Ah D, Brown CG, Brown SJ, Bryant AL, Davies M, Dodd M, Ferrell B, Hammer M, Knobf MT, Knoop TJ, LoBiondo-Wood G (2019) Research agenda of the oncology nursing society: 2019–2022. Oncol Nurs Forum 46:654–669

    Article  Google Scholar 

  • Wald HL, Leykum LK, Mattison ML, Vasilevskis EE, Meltzer DO (2014) Road map to a patient-centered research agenda at the intersection of hospital medicine and geriatric medicine. J Gen Intern Med 29:926–931

    Article  PubMed  PubMed Central  Google Scholar 

  • World Health Organization (2011) Building and strengthening partnerships: annual report 2010. Retrieved from https://apps.who.int/iris/bitstream/handle/10665/70758/WHO_AllianceHPSR_11.1_eng.pdf. Accessed 23 July 2021

  • World Health Organization (2021) WHO public health research agenda for managing infodemics. Retrieved from https://apps.who.int/iris/bitstream/handle/10665/339192/9789240019508-eng.pdf?sequence=1. Accessed 23 July 2021

  • Woud ML, Verwoerd J, Krans J (2017) Modification of cognitive biases related to posttraumatic stress: a systematic review and research agenda. Clin Psychol Rev 54:81–95

    Article  PubMed  Google Scholar 

  • Wykes T, Haro JM, Belli SR, Obradors-Tarragó C, Arango C, Ayuso-Mateos JL, Bitter I, Brunn M, Chevreul K, Demotes-Mainard J, Elfeddali I (2015) Mental health research priorities for Europe. Lancet Psychiat 2:1036–1042

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Gabrielle Tan and Laurie Reedman for their editorial support. We would also like to thank all the primary authors of the included articles.

Funding

This study has no funding sources to report.

Author information

Authors and Affiliations

Authors

Contributions

AT performed study selection and data extraction, analysis, quality assessment and write up of the manuscript. SKN performed study selection and data extraction and quality assessment. MN created the data extraction form, adapted the quality assessment and interpreted the findings. TS provided the background, performed the analysis, developed the framework and interpreted the findings. TK provided the background and interpreted the findings. All authors contributed to the development of the protocol and read and approved the final manuscript.

Corresponding author

Correspondence to Audrey Tan.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

MN has previously worked on priority setting of research for other similar organizations and this research was included in this paper. She has received funding from the National Institute of Health Research, Cochrane, Public Health Wales, NHS Devon, EU Marie Skłodowska Curie initiative, EU Cost Action Peninsula Dental Social Enterprise, University of Plymouth, Shahid Behshti University of Medical Science for other research projects, but this had no influence on the current research, financial or otherwise. TS has previously worked on priority setting of research for other similar organizations, but they had no influence on this research, financial or otherwise. The authors have no other competing interests to declare. The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions or policies of the institutions with which they are affiliated.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Commissioning author: Tarang Sharma

Appendices

Appendix A: Search strategy and summary of search results

Search Strategy OVID (MEDLINE) 1 April 2020.

  Searches Results Type
1 "Review"/ 2589030 Advanced
2 (research adj3 (priorit$ or agenda or strateg$)).kw,ti 5213 Advanced
3 (review or analysis or map* or database).ab,kw,ti 5557460 Advanced
4 3 or 1 7064197 Advanced
5 4 and 2 1911 Advanced
6 from 5 keep 1–1911 1911 Advanced

Search Strategy OVID (MEDLINE) 29 May 2021

  Searches Results Type
1 "Review"/ 2756172 Advanced
2 (research adj3 (priorit$ or agenda or strateg$)).kw,ti 5676 Advanced
3 (review or analysis or map* or database).ab,kw,ti 6124712 Advanced
4 1 or 3 7683993 Advanced
5 2 and 4 2102 Advanced
6 limit 5 to (english language and yr = "2020–2021") 211 Advanced

WHO IRIS search strategy (14/04/2020)

  Searches Results of search on 14/04/2020 Results of search on 1 June 2021
1 “research priority setting” 218 15
2 “research agenda setting” 27 6

Summary of searches:

  Database Results of first search Results of updated search Total Total after removal of duplicates
1 MEDLINE via OVID 1911 211 2122 1734
2 WHO IRIS 245 21 266
3 Other sources 7 - 7

Appendix B: Adapted AMSTAR-2 for use with reviews of priority setting

Adaptation developed by Dr Mona Nasser

  • Item 1: Did the research questions and inclusion criteria for the review make it clear what type of research priority setting exercises they are focusing on?

    • For yes—If the description if clear.

    • For no—If the description is not clear.

  • Item 2: Did the report of the review contain an explicit statement that the review methods were established prior to conduct of the review and did the report justify any significant deviations from the protocol?

    • For yes—If there is a protocol available and reported.

    • For no—If there is not available or not reported.

  • Item 3: Did the review authors explain their selection of the study designs for inclusion in the review?

    • For yes—If the authors describe and differentiate different type of studies and their implications in the review. Most these types of reviews just include any reporting of research priority setting without considering whether they have done evaluation or not.

    • For no—If the choice of the study design for evaluation is no appropriate for the objective.

  • Item 4: Did the review authors use a comprehensive literature search strategy appropriate for the research question and context of the study?

    • For yes—If they are looking at all priority setting in an organisation (for example, WHO or Cochrane) and have demonstrated they have taken an appropriate strategy to find all studies. If it is a review of published peer-reviewed literature, it will follow the standard AMSTAR rules, that is, if at least two bibliographic databases were searched and provided key words and publication restrictions.

    • For partial yes—If they are only look at priority setting in an organisation and they have used a selection of the reports of that organisation.

    • For no—If no systematic approach to searching the literature has been outlined.

  • Item 5: Did the review authors perform study selection in duplicate?

    • For yes—At least two reviewers independently agreed on selection of eligible studies and achieved consensus on which studies to include OR two reviewers selected a sample of eligible studies and achieved good agreement (at least 80 percent), with the remainder selected by one reviewer.

    • For no—Study selection not completed in duplicate.

  • Item 6: Did the review authors perform data extraction in duplicate?

    • For yes—At least two reviewers achieved consensus on which data to extract from included studies or two reviewers extracted data from a sample of eligible studies and achieved good agreement (at least 80 percent), with the remainder extracted by one reviewer.

    • For no—Data extraction not performed in duplicate.

  • Item 7: Did the review authors provide a list of excluded studies and justify the exclusions?

    • For yes—Authors provided a list of excluded studies and justified the exclusions.

    • For partial yes—Authors provided a list of all potentially relevant studies that were read in full-text form but excluded from the review.

    • For no—Authors did not provide a list of excluded studies or justifications for the exclusion.

  • Item 8: Did the review authors describe the included studies in adequate detail?

    • For yes—If they provide details of the research priority setting exercises.

    • For partial yes—If they mentioned what type of research priority setting was included.

    • For no—If they did not provide adequate details or descriptions of the included studies.

  • Item 9: Did the review authors use a satisfactory technique for assessing the quality of individual studies and the research priority setting processes in the review.

    • For yes—If they have a process in place and described and used it.

    • For no—If no technique for assessing the quality of individual studies is described or implemented.

  • Item 10: Did the review authors report on the sources of funding for the studies included in the review?

    • For yes—Must have reported on the sources of funding for individual studies included in the review. Reporting that the reviewers looked for this information but it was not reported by study authors also qualifies.

    • For no—Did not report on the sources of funding for individual studies.

  • Item 11: Did the authors use an appropriate approach to synthesize the information?

    • For yes—If they used a systematic and critical approach to evaluate the studies—this might be statistical or narrative or a combination of both.

    • For no—If no approach was outlined for synthesizing the information.

  • Item 12: Did the synthesis process consider the quality of the studies when combining?

    • For yes—If they have considered the quality of the studies when performing the synthesis.

    • For no—If they have not considered the quality of the studies when performing the synthesis.

  • Item 13: Did the interpretation and discussion of the results of the review considered the quality of the individual priority setting process or the evaluation study?

    • For yes—If the interpretation and discussion considered the quality of the individual studies.

    • For no—If the interpretation and discussion did not consider the quality of the individual studies.

  • Item 14: Did the review authors provide a satisfactory explanation for, and discussion of, any major discordance or differences in the results of the priority setting exercise or the evaluation of them?

    • If you see some major differences either in the final priorities or how the process is implemented or evaluation results, the authors need to explain and rationalize them in the synthesis process.

    • For yes—There was no significant heterogeneity in the results OR if heterogeneity was present, the authors discussed the impacts on the results.

    • For no—If heterogeneity is present and they do not explain or report it.

  • Item 15: Did they consider the impact of unpublished literature on the results?

    • For yes—If they discuss and consider the impact of unpublished literature on the results and interpretation.

    • For no—If they do not discuss and consider the impact of unpublished literature on the results and interpretation.

  • Item 16: Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?

    • For yes—The authors reported no competing interests OR The authors described their funding sources and how they managed potential conflicts of interest.

    • For no—The authors did not report whether there were competing interests, conflicts of interest or funding sources.

Appendix C: Excluded studies

References Reasons for exclusion
Abramowitz et al. (2018) Core concept does not evaluate research priority setting exercises
Anstee et al. (2011) Core concept does not evaluate research priority setting exercises
Barnes et al. (2015) Core concept does not evaluate research priority setting exercises
Bassetti et al. (2015) Core concept does not evaluate research priority setting exercises
Bragge et al. (2015) Core concept does not evaluate research priority setting exercises
Buchholz et al. (2018) Ineligible study design, i.e. not a critical or systematic review
Consortium from Altarum (2012) Ineligible study design, i.e. not a critical or systematic review
Crewdson et al. (2018) Ineligible study design, i.e. not a critical or systematic review; Core concept does not evaluate research priority setting exercises
Cumpston et al. (2012) Core concept does not evaluate research priority setting exercises
Davis et al. (2015) Core concept does not evaluate research priority setting exercises
Delost and Nadder (2014) Ineligible study design, i.e. not a critical or systematic review; Core concept does not evaluate research priority setting exercises
Doosti-Irani and Holakouie-Naieni (2016) Core concept does not evaluate research priority setting exercises
Fabbri et al. (2018) Ineligible study design, i.e. not a critical or systematic review; Core concept does not evaluate research priority setting exercises
Foster et al. (2018) Core concept does not evaluate research priority setting exercises
Hasson et al. (2020) Duplicate
Hill et al. (2019) Ineligible study design, i.e. not a critical or systematic review; Core concept does not evaluate research priority setting exercises
Knight et al. (2014) Core concept does not evaluate research priority setting exercises
Johnston et al. (2016) Ineligible study design, i.e. not a critical or systematic review; Core concept does not evaluate research priority setting exercises
Jones and Geneau (2012) Core concept does not evaluate research priority setting exercises
Karimkhani et al. (2016) Core concept does not evaluate research priority setting exercises
Kong et al. (2019) Core concept does not evaluate research priority setting exercises
Kühne et al. (2021) Core concept does not evaluate research priority setting exercises
Morton et al. (2012) Core concept does not evaluate research priority setting exercises
Nicolau et al. (2012) Core concept does not evaluate research priority setting exercises
Okland et al. (2017) Core concept does not evaluate research priority setting exercises
Oncology Nursing Forum (2019) Ineligible study design, i.e. not a critical or systematic review
Pozzar and Berry (2017) Core concept does not evaluate research priority setting exercises
Pratt (2020) Ineligible study design, i.e. not a critical or systematic review
Sebastianski et al. (2019) Core concept does not evaluate research priority setting exercises
Sigfrid et al. (2019) Core concept does not evaluate research priority setting exercises
Van Royen et al. (2010) Core concept does not evaluate research priority setting exercises
Von Ah et al. (2019) Ineligible study design, i.e. not a critical or systematic review
Wald et al. (2014) Ineligible study design, i.e. not a critical or systematic review; Core concept does not evaluate research priority setting exercises
World Health Organization (2011) Ineligible study design, i.e. not a critical or systematic review
World Health Organization (2020a, b) Ineligible study design, i.e. not a critical or systematic review
World Health Organization (2021) Ineligible study design, i.e. not a critical or systematic review
Woud et al. (2017) Core concept does not evaluate research priority setting exercises
Wykes et al. (2015) Ineligible study design, i.e. not a critical or systematic review

Appendix D: Quality assessment

We used adapted version of AMSTAR criteria for quality assessment with 16 items. None of the included reviews were assessed to have high or moderate overall confidence in the results. Only two reviews (Tong et al. 2015; Cadorin et al. 2020) were assessed as low quality and the remaining 29 reviews were assessed to have critically low-quality overall confidence.

All the 29 included reviews clearly reported the research questions and inclusion criteria for the review. Eighteen included reviews did not report if the review protocol was available or report any deviations from the protocol (Alqahtani et al. 2021; Badakhshan et al. 2018; Bourne et al. 2018; Bragge et al. 2015; Bryant et al. 2014; Garcia et al. 2015; Hasson et al. 2020; McGregor et al. 2014; Odgers et al. 2018; Reveiz et al. 2013; Rudan et al. 2017; Terry et al. 2018; Tomlinson et al. 2011; Tong et al. 2015; Tong et al. 2017; Yoshida 2016, Wade et al. 2021). Nine reviews did not explain their selection of the study designs for inclusion (Badakhshan et al. 2018; Booth et al. 2018; Garcia et al. 2015; Hasson et al. 2020; Odgers et al. 2018; Rudan et al. 2017; Rylance et al. 2010; Yoshida 2016).

Only four reviews did not use a comprehensive literature search strategy (Roche et al. 2021; Viergever et al. 2010; Wade et al. 2021; Yoshida 2016). In 13 out of 29 reviews, the review authors had selected the studies in duplicate (Booth et al. 2018; Bourne et al. 2018; Bragge et al. 2015; Cadorin et al. 2020; El-Harakeh et al. 2019; El-Harakeh et al. 2020; Fadlallah et al. 2020; Garcia et al. 2017; Graham et al. 2020; Mörelius et al. 2020; Stewart et al. 2011; Rylance et al. 2010; Tong et al. 2017). Fifteen reviews did not extract data in duplicate (Alqahtani et al. 2021; Bragge et al. 2015; Garcia et al. 2015; Manafo et al. 2018; McGregor et al. 2014; Reveiz et al. 2013; Roche et al. 2021; Rudan et al. 2017; Stewart et al. 2011; Terry et al. 2018; Tomlinson et al. 2011; Viergever et al. 2010; Wade et al. 2021; Yoshida 2016).

Only four reviews provided a list of excluded studies with justifications for exclusions (Bourne et al. 2018; Fadlallah et al. 2020; Garcia et al. 2017; Rylance et al. 2010) and one review mentioned that the list of excluded studies can be requested to the review authors (Booth et al. 2018). Only one review did not describe the included studies in adequate detail (Wade et al. 2021) and three studies just mentioned what type of research priority setting was included (Graham et al. 2020; Manafo et al. 2018; Terry et al. 2018). Fourteen studies used satisfactory technique for assessing the quality of individual studies and the research priority setting process in the review. Only eight studies reported how sources of conflict of interest were managed in the included studies (Booth et al. 2018; Bourne et al. 2018; El-Harakeh et al. 2020; Fadlallah et al. 2020; Odgers et al. 2018; Reveiz et al. 2013; Terry et al. 2018; Tong et al. 2015). Majority of the included studies used an appropriate approach to synthesis the information except for three (Graham et al. 2020; Manafo et al. 2018; Yoshida 2016) which did not outline the information on data synthesis. Only one study considered the quality of the studies when performing the synthesis (Cadorin et al. 2020).

Six studies did the interpretation and discussion of the results considering the quality of the priority setting process or the evaluation of the study (Cadorin et al. 2020; Graham et al. 2020; Manafo et al. 2018; Mörelius et al. 2020; Odgers et al. 2018; Tong et al. 2015). Six studies did not provide any satisfactory explanation or discussion of major discordance in the results of the priority setting exercise (Badakhshan et al. 2018; Rylance et al. 2010; Viergever et al. 2010; McGregor et al. 2014; Bragge et al. 2015; Garcia et al. 2015; Graham et al. 2020). The impact of unpublished literature on the results of the review was considered only in six of the included studies (Alqahtani et al. 2021; Bourne et al. 2018; Cadorin et al. 2020; Garcia et al. 2017; Roche et al. 2021; Tong et al. 2015). Review authors of four studies did not report any potential sources of conflict of interest including any funding they received for conducting the review (Garcia et al. 2015; El-Harakeh et al. 2020; Garcia et al. 2017).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tan, A., Nagraj, S.K., Nasser, M. et al. What do we know about evidence-informed priority setting processes to set population-level health-research agendas: an overview of reviews. Bull Natl Res Cent 46, 6 (2022). https://doi.org/10.1186/s42269-021-00687-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s42269-021-00687-8

Keywords

  • Research priority setting
  • Overview of reviews
  • Systematic review
  • National health research system
  • Research agenda
  • Research funding
  • Resource allocation