Stage 2: Protocol Development & Review
A Guide to Writing and Submitting a BEME Review Protocol
The protocol details the procedures for identifying, evaluating and amalgamating the evidence for a BEME review. The protocol describes the review's context and objectives, defines the review's question and remit, and outlines the methodology to be employed to critically appraise and analyse evidence from the selected studies. Setting out an explicit and thorough protocol before the review commences defines the review process and helps to ensure that the evidence to be considered is systematic and comprehensive and that the possibility of bias is diminished. The protocol is peer reviewed by an assigned BICC and, following any requested revisions, is posted on the BEME website. A BEME review protocol must clearly
A BEME review protocol must clearly:
- Identify what type of review is planned – effectiveness, definitional, scoping , etc.
- Identify what the practical contribution to knowledge will be once the review is complete.
- Argue that there is a sound link between the research question(s) and the secondary research methodology that will be used to identify answers to the question(s).
- Give a clear definition of the key concepts in the review question(s) or show how these will be arrived at.
The protocol comprises the following:
- Cover sheet
- Background to the topic
- Review question(s), objectives and key words
- Search sources and strategies
- Study selection criteria
- Procedure for extracting data
- Synthesis of extracted evidence
- Scoping Search
- Translation into practice
- Project timetable
- Conflict of interest statement
- Plans for updating the review
- Changes to the protocol
- Protocol checklist
1. Cover sheet–The cover sheet should include the following information:
- Review title
- Names, contact details and affiliations of group members, with lead reviewer highlighted
- Contact details for the lead reviewer
- Sources of support
2. Background to the topic– The review topic should be explained and placed in context to make the rationale and need for the review explicit. It should contain:
- a brief overview of the research question
- a conceptual discussion relating to the research question
- a statement of the significance of the research question
Reference should be made to any previous reviews of the research topic where these exist and reasons for the need for the current review.
3. Review topic/question(s), objectives and key words– A BEME review topic/question should focus on and inform on practical issues or problems faced by the teacher or institution in their day-to-day practice. For example, the research topic/question should be phrased in such a way as to provide information on how the teacher or institution should respond to the adoption of a new teaching approach in their own context. The review topic/question should be precisely defined. Typically, a review topic/question will identify:
- population/participants e.g. undergraduate or postgraduate students
- the activity under investigation e.g. the timing of feedback in assessment
- outcomes e.g. change of attitudes or knowledge
BEME review questions may be most helpful where they illuminate the topic area. A valuable question will usually explore aspects of the review topic and will not necessarily be answered with a simple yes or no, e.g.What are the aspects of high fidelity simulators that promote effective learning? The terms used in the question should be defined: e.g.what is a high-fidelity simulator? Having defined the question it may be necessary to broaden or narrow the search (see Search sources and strategies below). Reviewers should identify five key words related to the topic of their review.
4. Search sources and strategies– BEME reviews require systematic searches for evidence across a comprehensive range of sources relevant to the topic. Systematic searching in medical education is particularly challenging given that the evidence is widely dispersed and very often poorly indexed. An appropriate search strategy should be constructed to ensure the remit of the review is met and this will be published as an appendix to the completed review. The protocol should identify the full range of sources to be consulted including databases and specialist registers to be accessed, journals to be hand-searched, the grey literature to be accessed and methods that will be used to contact experts in the field. The protocol should indicate the time limit for the search. If possible, neither language nor geography should restrict searches. Depending upon the timetable for the review (see Project timetable below), it may be necessary to include in the protocol the provision to conduct updating searches during the review (More on literature search can be found here).
5. Study selection criteria– The review group should make explicit the criteria it plans to use for selecting or excluding studies in order that the results are objective and reproducible, bearing in mind the following:
- as part of the BEME approach, studies should not be excluded solely on the basis of type of study design;
- where studies are to be excluded for reasons of methodology, these reasons should be made explicit.
6. Procedure for extracting data– The review group should include a statement of how many group members will code each article, how the review group intends to check for inter-rater reliability and the proposed procedure to resolve differences in coding of studies.
7. Synthesis of extracted evidence– The methodology used in BEME reviews will inevitably evolve during the initial pilot reviews. Review groups should make explicit the methodological procedures they plan to use to synthesise the evidence.
8. Scoping search - Using the search criteria selected the review group should assess the feasibility and practicality of their search sources and strategy as well as judging the amount of material in the literature.
9.Translation into practice - BEME reviews are practical applications of secondary research. The procotol should describe how the authors findings will translate into practice and how they will help and guide heathcare education in the future.
10. Project timetable - Review groups should submit a proposed timetable for the stages involved in producing the review. Timetables will vary between BEME groups, depending on an individual group's resources, its size, geographic distribution of members and complexity of the topic, etc.
11. Conflict of interest statement– The protocol must include a conflict of interest statement, relating to all review group members. These include academic, institutional, political, personal or other conflicts. Special concern must be given to financial conflict of interest, with the acknowledgement of any financial support, including money, hospitality or subsidy.
12. Plans for updating the review– Review groups should indicate how they intend to update the review after completion and if so, include brief details and a potential timeline.
13. Changes to the Protocol– Whilst a well thought out protocol should reduce the need for changes, unanticipated issues may arise and modifications to the review topic/question, coding sheets and/or protocol may become necessary as the review group becomes more familiar with the evidence. Any subsequent changes to the protocol must be carefully recorded, with the reasons for these changes and the date the changes took place. Significant changes to the protocol must be submitted to BEME for approval.
14. Protocol checklist– When the protocol is being reviewed by the assigned BICC they use the Protocol Checklist. It is always useful for protocol authors to refer to this checklist to ensure that all the essential elements are included in their protocol.