The opportunity for CanMEDS competency for the workplace… – BMC Medical Education

We used a web-based Delphi study to gather evidence based on consensus assessments in which the CanMEDS key competencies were assessed, first as possible, then as appropriate for work-based assessments in the Flemish General Practice (GP) in Belgium [13,14,15]. On the basis of the available literature, we discussed and decided on the necessary steps to ensure improved methodological rigor. Table 1 provides an overview of the design steps, based on the Leadership in Leadership and Reporting Dolphin Study (CREDES). [16]. We elaborate further on our methodological decisions considering the stages of designing CREDES.

study design

We want to use the email to recruit maps from different geographic locations in Flanders and reach a larger group in a cost-free way. The online format is also preferred because this study was conducted during the COVID-19 pandemic. We defined feasibility as what can be observed in the workplace, and whether the competency-based formulation is appropriate for the task. We defined a consistency that can be consistently observed across different training settings and levels in the device (Fig. 1). [13,14,15]. Consensus was defined as 70% of respondents agreeing or strongly agreeing that the item is feasible or stands for assessment in the workplace [17]. Non-agreement was defined as less than 70% of respondents agreeing or strongly agreeing, and neither a major change in agreement ratings nor suggestions for any change per panel after round 2.

To ensure the iterative nature of our study, we decided to establish a minimum of three rounds. [18, 19]. After each Delphi round, when consensus was reached for the CanMEDS key competency, it was no longer offered for evaluation. Although the traditional Delphi method starts from an informal round, we preferred to follow a semi-structured approach, since our main objective was to validate the predefined CanMEDS framework. [4]. Therefore, we used a combination of closed and open ended questions [20].

In the first round, respondents were asked whether they could rate the CanMEDS key competency and their agreement was based on a 5-point Likert scale. They will also be able to provide qualitative comments for each key competency [7, 14]. In the second round, we informed the accountants about the agreement of the evaluations of round 1. In this round, the accountants were asked to send suggestions for concrete modifications and to draw two research methods separately. A document has also been added to the questions that arose in round 1, based on the quality of the notes. For some clarification on the competency formula, CanMEDS can help provide a table with suggestions for the competencies of each key competency. Additionally, we have listed and generated the most frequent qualitative comments for your review. Opinions about modifications in key competencies are shared transparently. Again, we asked the panel to rate the CanMEDS key competencies as feasible and consistent on a job-based assessment on a 5-point Likert scale.

In the third round, we compared summaries from the previous rounds. Following the panelists’ request, we included a list of examples of how each CanMEDS key competency would translate into the workplace. In this final round, we asked board members whether or not they agreed that the CanMEDS key competency was feasible and consistent for workplace assessment. if not, reasons are required to abstain from consent [15]. Figure 2 shows the three Delphic scrolls.

Flowchart of three Delphi rounds

study setting

In order to create a coherent approach across Flanders, four Flemish universities (KU Leuven, University of Ghent, University of Antwerp, and Free University of Flanders Brussels) have created an interuniversity GP Training course consisting of three levels. Practical coordination and decision-making regarding the curriculum are the responsibility of the Interuniversity Center for GP Training (ICGPT). The ICGPT is responsible, among others, for placing clinical programs, organizing tests, arranging meetings of GP trainees with educators, and managing trainee portfolios where the assessment of competencies is related.

Choice of floors

To select the panelists, we followed purposive sampling [13, 21]. We set three selection criteria: 1) having sufficient GP experience (> 3 years of experience), 2) experience in mentoring and evaluating practices in the workplace [7, 22]. Seventy participants were invited by the Principal Investigator (BS) via email. In order to incorporate a wide range of opinions, the board consisted of both GP teachers and GP protégés [23]. GP teachers are workshop-based teachers assisted in their internships, while GP tutors are associated with the university to provide guidance and facilitate peer learning and support (10-15 trainings per group) twice monthly. Both groups were responsible for evaluating training in the workplace. The panelists lived in the different provinces of Flanders to bring together great ideas and apply faith [13, 23]. Although there is no consensus on the appropriate sample size for a Delphi design, more than 15–30 respondents could yield reliable results. [23, 24]. In our study, we selected accountants who received the same medical background and held a general understanding of politics. In addition, to determine the size of the sample, we began to look at the feasibility parameters to obtain a good response rate, so that the Delphi rounds were spacious and a reasonable time was required to complete each one.

Progress and pilot Delphi surveys

The 27 CanMEDS key competencies were transferred from English to Dutch because the panel spoke Dutch. Figure 3 graphically illustrates how the Delphi survey was constructed. First, the CanMEDS competencies were translated separately by five researchers. After discussing and evaluating all the translations, we decided to keep the Dutch translation as close as possible to the original English framework. Second, to validate the translation and pilot of the instrument, we sent it to a group of medical professionals for comment. Third, when feedback was received and the Dutch translation was completed, the Dutch version of the framework was back-translated into English to confirm the accuracy of the translation. [25].

Process steps for constructing Delphi surveys

Each Delphi consisted of an introductory section, a CanMEDS key competency assessment, and a concluding section. In the introduction, each item is explained, and the plans are shared. We have added an end section to allow space for board communication and feedback not related to the CanMEDS key competencies (eg, time required for completion, notes on layout). To avoid confusion between the different parts of CanMEDS, the main competencies were divided by part. Figure 4 illustrates how survey results are displayed before a consensus is reached.

1

Data collection and analysis

To collect our data, we used the Qualtrics XM database. This online tool is allowed to maintain anonymity among the panellists [26]. A personal link was sent by email to each panellist. This is allowed by following responses and sending specific reminders to members. Due to the heavy workload caused by the COVID-19 pandemic, each round lasted four weeks. We want a flexible approach to increase the response rate of each round. Weekly reminders are sent to members who have not yet completed the survey [26]. Data collection took place between October 2020 and February 2021. To analyze the quantitative data, we calculated descriptive statistics for all items using SPSS 27 (IBM SPSS Statistics 27). We used Microsoft Excel to qualitatively identify and categorize the data. Panelists’ comments are anonymous and written down verbatim. To examine the quality of the data, we used content analysis [27].

The role of the research team is to prevent prejudice

Methodological decisions made by the research team were documented in the available literature. We pre-defined and agreed methodological steps before starting the study. We used, monitored and evaluated these steps in the study. The results of each round were analyzed by the research teams, while the qualitative data were interpreted by two researchers triangulating the researchers. [28].