Do One, Order One, See One
Conditions
Keywords
implementation study, motivational interviewing, substance use
Brief summary
General medical hospitals provide care for a disproportionate share of patients who misuse substances. Motivational interviewing (MI) is a well-recognized, evidenced-based substance use treatment. However, it is unclear which implementation strategies lead to the efficient and proficient uptake of MI in general medical settings, such as medical inpatient units. Because medical providers have multiple practice demands and time constraints, new practices have the greatest chance of being implemented if they are simple and compatible with existing workflows and systems. Two widely used strategies to bring specialized practices into use within general hospital settings are the apprenticeship model of training and use of consultation-liaison (CL) services. The apprenticeship model requires that appropriate patients and trainers are available with high flexibility for teaching and supervision; when applied to behavioral counseling approaches, this model may be incompatible with the providers' medical role and time constraints. In contrast, ordering MI through CL is relatively simple, minimally burdensome, and highly compatible with the way clinicians secure other specialist services for their patients in the hospital. This cluster randomized controlled trial examines the effectiveness of three different strategies for integrating MI into the practice of medical providers working within an academically affiliated internal medicine hospitalist service. Specifically, the trial randomizes 38 healthcare providers to one of three conditions: (1) a continuing medical education workshop that provides background and shows healthcare providers how to conduct MI (the control condition, called SEE ONE); (2) a see one, do one apprenticeship model involving workshop training plus live supervision of bedside practice (DO ONE); and (3) ordering MI from CL after learning about it in a workshop (ORDER ONE). Following the respective MI trainings, each healthcare provider will be assessed for the provision of MI to 40 study-eligible inpatients, recruited by the research team after admission to our general medical units. Trial hypotheses are 1) the percentage of MI sessions delivered by providers to study-eligible inpatients will be higher in both Do One and Order One than See One, and 2) providers in both Do One and Order One will conduct MI sessions with greater integrity than those in See One. This study is an implementation trial examining provider, not patient, outcomes.
Detailed description
General medical hospitals provide care for a disproportionate share of patients who abuse or are dependent upon substances. This group is among the most costly to treat and has the poorest medical and substance use outcomes. Motivational interviewing(MI) is a well-recognized, evidenced-based substance use treatment that has been adapted for use as a brief intervention in health care settings. MI is applicable to many health-related behavioral problems, and can be taught to a broad range of health care clinicians. However, it is unclear which implementation strategies will lead to the efficient and proficient uptake of MI in general medical settings, such as medical inpatient units. Primary care clinicians have multiple practice demands and time constraints. New practices have the greatest chance of being implemented if they are simple and compatible with existing workflows and systems. Two widely used strategies to bring specialized practices into use within general hospital settings are the see one, do one apprenticeship model of training and use of consultation-liaison (CL) services. See one, do one has been a modus operandi in medical education for centuries and relies upon a competency-based supervision training approach. While it has been empirically validated in the specialty addiction field, less controlled testing of this implementation strategy is available in general medical settings. The apprenticeship approach requires that appropriate patients and trainers are available with high flexibility for teaching and supervision; when applied to behavioral counseling approaches, this may be seen as incompatible with the medical role and time constraints of clinicians. In contrast, ordering MI through CL is a relatively simple, minimally burdensome process and highly compatible with the way clinicians secure other specialist services for their patients in the hospital. We propose to conduct a randomized controlled implementation trial using mixed quantitative and qualitative methods to examine the effectiveness of three different strategies for integrating MI into the practice of healthcare providers working within Yale New Haven Hospital's internal medicine hospitalist service and other general medical inpatient units. Specifically, we will randomize 40 healthcare providers to one of three conditions: (1) a continuing medical education workshop that provides background and shows healthcare providers how to conduct MI (the control condition, called SEE ONE); (2) a see one, do one apprenticeship model involving workshop training plus live supervision of bedside practice (DO ONE); and (3) ordering MI from CL after learning about it in a workshop (ORDER ONE). Following the respective MI trainings, each healthcare provider will be assessed for the provision of MI to 40 study-eligible inpatients, recruited by the research team after admission to our general medical units.We hypothesize that the percentage of MI sessions delivered by providers to study-eligible inpatients would be higher in both Do One and Order One than See One. We also hypothesize that providers in both Do One and Order One would conduct MI sessions with greater integrity (i.e., adherence to core components of MI and delivery of them with competence) than those in See One. Please note, as an implementation trial, the primary outcomes for this study focus on provider behaviors, namely, uptake of MI sessions with patients and the adherence and competence in which they conduct MI sessions. No outcome data will be collected and reported at the patient level.
Interventions
BEHAVIORALSee One
A 1-day workshop conducted by a member of the Motivational Interviewing Network of Trainers (MINT), according to MINT recommendations, to build providers' skills needed to use MI with patients who misuse substances.
BEHAVIORALDo One
A 1-day workshop conducted by a member of the Motivational Interviewing Network of Trainers (MINT), according to MINT recommendations, to build providers' skills needed to use MI with patients who misuse substances. Following the workshop training, providers conduct two motivational interviews bedside with patients under the supervision of one of the MI-trained CL clinicians, who subsequently give them performance feedback and coaching. Providers receive one more supervised practice case mid-trial. In addition, they can request additional supervision at any point during the trial.
BEHAVIORALOrder One
A 1-day workshop conducted by a member of the Motivational Interviewing Network of Trainers (MINT), according to MINT recommendations, to build providers' skills needed to use MI with patients who misuse substances. Following the workshop, providers have the option to conduct MI with patients themselves or to order a MI for delivery by one of the MI-trained CL clinicians.
Sponsors
National Institute on Drug Abuse (NIDA)
Study design
Allocation
RANDOMIZED
Intervention model
PARALLEL
Primary purpose
HEALTH_SERVICES_RESEARCH
Masking
SINGLE (Subject)
Masking description
Research staff independently screen, assess and obtain consent from eligible substance using patients admitted to the general medical hospitalist service. Patients are included if they are assigned to a participating provider according to the hospital's usual clinical administrative procedures. Thus, patients followed the randomization condition of their assigned provider, but they do not know how their providers have been trained to deliver MI and the providers do not know which patients assigned to them have been enrolled in the study. This approach permits a naturalistic test of the providers' ability to identify and intervene using MI with patients who misuse substances without patients or research staff prompting providers to do so.
Intervention model description
In this three-arm parallel assignment, study-eligible and consented provider participants are randomized to one of three conditions (See One, Do One, or Order One) in which they receive their respective motivational interviewing training in parallel with provider participants assigned to the other conditions.
Eligibility
Sex/Gender
ALL
Age
18 Years to 100 Years
Healthy volunteers
No
Inclusion criteria
For Healthcare provider participants: Inclusion criteria : * Assignment to one of the general medical inpatient units during day-time shifts; intensive care units will be excluded given the morbidity of patients in this setting. * Volunteer to serve as study clinicians, attend a workshop about MI, and possibly receive live supervision. * Agree to all procedures of this trial (randomization to training condition and of assigned patients, audio recording MI sessions, and completing assessments).
Exclusion criteria
* Have been formally supervised to use MI with patients on the units. * Intend to give notice that they plan to leave the hospital or are scheduled for medical or family leave such that they will not be able to interview 40 patients during the study period. For patient participants: Inclusion criteria: * Are 18 years of age or older. * Acknowledge use of a substance within past 28 days and meets screening criteria consistent with substance (illicit drugs, licit drugs that are used in a non-medically indicated fashion, alcohol, or nicotine) use disorder. * Are willing to consent to audio recording of interview with the provider or CL clinician.
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| Percentage of Motivation Interviewing Sessions Audio Recorded | Provider participants will be followed for the duration of the trial period, an expected average of 24 weeks to see 40 study-enrolled patients per provider. | The percentage of audio recorded Motivational Interviewing sessions conducted by providers for their 40 consecutively enrolled study-eligible patients |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| The Independent Tape Rater Scale - Fundamental Adherence Score | All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider. | The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 fundamental MI items important for building rapport and understanding patients' experiences (e.g. reflections), rated for adherence (i.e. the extent/frequency of intervention delivery) on a scale from 1 (not at all) to 7 (extensively). The fundamental adherence score is the average of these 5 items, with scores ranging from 1-7 and higher scores indicating greater adherence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis. |
| The Independent Tape Rater Scale- Fundamental Competence Score | All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider. | The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 fundamental MI items important for building rapport and understanding patients' experiences (e.g. reflections), rated for competency (i.e., the skill/quality of intervention delivery) on a scale of 1 (not at all) to 7 (excellent). The fundamental competence score is the average of these 5 items, for which there was adherence, with higher scores indicating greater competence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis. |
| The Independent Tape Rater Scale- Advanced Adherence Score | All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider. | The advanced adherence score is the average of these 5 items. The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 advanced MI items critical for eliciting patients' motivations for change (e.g., asking for reasons for change), rated for adherence (i.e., the extent/frequency of intervention delivery) on a scale of 1 (not at all) to 7 (extensively). The advanced adherence score is the average of these 5 items, with scores ranging from 1-7 and higher scores indicating greater adherence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis. |
| The Independent Tape Rater Scale- Advanced Competence Score | All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider. | The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 advanced MI items critical for eliciting patients' motivations for change (e.g., asking for reasons for change), rated for competency (i.e., the skill/quality of intervention delivery) on a scale of 1 (not at all) to 7 (excellent). The advanced competence score is the average of these 5 items, for which there was adherence, with higher scores indicating greater competence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis. |
| Adequately Performing MI | All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider. | The Independent Tape Rater Scale includes 5 fundamental MI items and 5 advanced MI items. Items are rated for adherence on a scale of 1 (not at all) to 7 (extensively) and competency on a scale of 1 (not at all) to 7 (excellent). Higher scores are better adherence or competence. An adequately performed MI session occurs when at least half the MI consistent items are rated average or above (i.e., score of 4, 5, 6, or 7) for both adherence and competence. Percentage of MI sessions reaching this adequate performance threshold, per arm, served as the secondary outcome measure. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis. |
Countries
United States
Participant flow
Pre-assignment details
The study's primary outcome and secondary outcomes are only collected and reported at the provider level - not at the patient level.
Participants by arm
| Arm | Count |
|---|---|
| See One Providers in See One only receive MI workshop training, giving them an opportunity to see the MI intervention and learn how to conduct it. The trainer encourages them to screen their patients for substance misuse and apply MI as indicated. | 349 |
| Do One Following workshop training, MI-trained CL clinicians directly supervise providers' live bedside provision of MI to patients twice before beginning the trial and once midstream. In addition, providers have the option to request additional live supervision from CL clinicians during the trial, consistent with the apprenticeship model. | 391 |
| Order One Following the workshop, providers either administer MI themselves or order a MI for delivery by one of the MI-trained CL clinicians. Only providers in Order One can specifically request MI through a separate CL order in the electronic health record. The physicians or PAs directly place MI orders. Nurses contact physicians or PAs to place the MI order. The CL clinicians are trained in MI via a clinical trials training approach: 1) a 2-day skill-building workshop; 2) three post-workshop supervised practice cases based on review of audio recorded sessions; and 3) follow-up monthly group supervision to maintain and monitor the CL clinicians' MI practice. CL clinicians also learned supervisory practices to provide live supervision to providers in Do One. | 471 |
| Total | 1,211 |
Baseline characteristics
| Characteristic | See One | Total | Order One | Do One |
|---|---|---|---|---|
| Age, Continuous Patients | 46.8 years STANDARD_DEVIATION 14 | 46.7 years STANDARD_DEVIATION 14.3 | 46.9 years STANDARD_DEVIATION 14.5 | 46.3 years STANDARD_DEVIATION 14.5 |
| Age, Continuous Providers | 39.2 years STANDARD_DEVIATION 13.2 | 35.3 years STANDARD_DEVIATION 11 | 32.2 years STANDARD_DEVIATION 10.1 | 34.4 years STANDARD_DEVIATION 8.9 |
| Race/Ethnicity, Customized Patients Hispanic White | 44 Participants | 144 Participants | 53 Participants | 47 Participants |
| Race/Ethnicity, Customized Patients non-Hispanic African American | 98 Participants | 352 Participants | 149 Participants | 105 Participants |
| Race/Ethnicity, Customized Patients non-Hispanic Asian | 0 Participants | 0 Participants | 0 Participants | 0 Participants |
| Race/Ethnicity, Customized Patients non-Hispanic Other Race | 1 Participants | 11 Participants | 7 Participants | 3 Participants |
| Race/Ethnicity, Customized Patients non-Hispanic White | 193 Participants | 666 Participants | 249 Participants | 224 Participants |
| Race/Ethnicity, Customized Providers Hispanic White | 0 Participants | 2 Participants | 2 Participants | 0 Participants |
| Race/Ethnicity, Customized Providers non-Hispanic African American | 1 Participants | 3 Participants | 1 Participants | 1 Participants |
| Race/Ethnicity, Customized Providers non-Hispanic Asian | 2 Participants | 3 Participants | 1 Participants | 0 Participants |
| Race/Ethnicity, Customized Providers non-Hispanic Other Race | 0 Participants | 1 Participants | 1 Participants | 0 Participants |
| Race/Ethnicity, Customized Providers non-Hispanic White | 10 Participants | 29 Participants | 8 Participants | 11 Participants |
| Sex: Female, Male Patients Female | 152 Participants | 534 Participants | 200 Participants | 182 Participants |
| Sex: Female, Male Patients Male | 184 Participants | 639 Participants | 258 Participants | 197 Participants |
| Sex: Female, Male Providers Female | 11 Participants | 33 Participants | 12 Participants | 10 Participants |
| Sex: Female, Male Providers Male | 2 Participants | 5 Participants | 1 Participants | 2 Participants |
| Type of provider Medical Doctor | 2 Participants | 5 Participants | 1 Participants | 2 Participants |
| Type of provider Physician's Assistant | 5 Participants | 14 Participants | 5 Participants | 4 Participants |
| Type of provider Registered Nurse | 6 Participants | 19 Participants | 7 Participants | 6 Participants |
Adverse events
| Event type | EG000 affected / at risk | EG001 affected / at risk | EG002 affected / at risk |
|---|---|---|---|
| deaths Total, all-cause mortality | 0 / 349 | 0 / 391 | 0 / 471 |
| other Total, other adverse events | 0 / 349 | 0 / 391 | 0 / 471 |
| serious Total, serious adverse events | 0 / 349 | 0 / 391 | 0 / 471 |
Outcome results
Primary
Percentage of Motivation Interviewing Sessions Audio Recorded
The percentage of audio recorded Motivational Interviewing sessions conducted by providers for their 40 consecutively enrolled study-eligible patients
Time frame: Provider participants will be followed for the duration of the trial period, an expected average of 24 weeks to see 40 study-enrolled patients per provider.
Population: A percentage of MI sessions audio recorded was calculated for each provider, and a mean percentage was calculated by group.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| See One | Percentage of Motivation Interviewing Sessions Audio Recorded | 0.8 percentage of MI sessions audio recorded | Standard Deviation 2.3 |
| Do One | Percentage of Motivation Interviewing Sessions Audio Recorded | 3.0 percentage of MI sessions audio recorded | Standard Deviation 5.4 |
| Order One | Percentage of Motivation Interviewing Sessions Audio Recorded | 20.5 percentage of MI sessions audio recorded | Standard Deviation 17.9 |
p-value: 0.0009Kruskal-Wallis
p-value: <0.0001Kruskal-Wallis
p-value: 0.4983Kruskal-Wallis
Secondary
Adequately Performing MI
The Independent Tape Rater Scale includes 5 fundamental MI items and 5 advanced MI items. Items are rated for adherence on a scale of 1 (not at all) to 7 (extensively) and competency on a scale of 1 (not at all) to 7 (excellent). Higher scores are better adherence or competence. An adequately performed MI session occurs when at least half the MI consistent items are rated average or above (i.e., score of 4, 5, 6, or 7) for both adherence and competence. Percentage of MI sessions reaching this adequate performance threshold, per arm, served as the secondary outcome measure. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis.
Time frame: All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider.
| Arm | Measure | Value (COUNT_OF_UNITS) |
|---|---|---|
| See One | Adequately Performing MI | 2 MI Sessions completed by providers |
| Do One | Adequately Performing MI | 11 MI Sessions completed by providers |
| Order One | Adequately Performing MI | 100 MI Sessions completed by providers |
Secondary
The Independent Tape Rater Scale- Advanced Adherence Score
The advanced adherence score is the average of these 5 items. The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 advanced MI items critical for eliciting patients' motivations for change (e.g., asking for reasons for change), rated for adherence (i.e., the extent/frequency of intervention delivery) on a scale of 1 (not at all) to 7 (extensively). The advanced adherence score is the average of these 5 items, with scores ranging from 1-7 and higher scores indicating greater adherence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis.
Time frame: All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider.
Population: Analysis population includes the N=114 MI sessions that were completed by providers across all training groups.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| See One | The Independent Tape Rater Scale- Advanced Adherence Score | 3.93 score on a scale | Standard Deviation 0.58 |
| Do One | The Independent Tape Rater Scale- Advanced Adherence Score | 4.38 score on a scale | Standard Deviation 0.95 |
| Order One | The Independent Tape Rater Scale- Advanced Adherence Score | 4.81 score on a scale | Standard Deviation 0.65 |
p-value: 0.1566Kruskal-Wallis
p-value: 0.0982Kruskal-Wallis
p-value: 0.6631Kruskal-Wallis
Secondary
The Independent Tape Rater Scale- Advanced Competence Score
The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 advanced MI items critical for eliciting patients' motivations for change (e.g., asking for reasons for change), rated for competency (i.e., the skill/quality of intervention delivery) on a scale of 1 (not at all) to 7 (excellent). The advanced competence score is the average of these 5 items, for which there was adherence, with higher scores indicating greater competence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis.
Time frame: All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider.
Population: Analysis population includes the N=114 MI sessions that were completed by providers across all training groups.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| See One | The Independent Tape Rater Scale- Advanced Competence Score | 3.92 score on a scale | Standard Deviation 0.88 |
| Do One | The Independent Tape Rater Scale- Advanced Competence Score | 4.25 score on a scale | Standard Deviation 0.52 |
| Order One | The Independent Tape Rater Scale- Advanced Competence Score | 4.49 score on a scale | Standard Deviation 0.46 |
p-value: 0.1327Kruskal-Wallis
p-value: 0.4097Kruskal-Wallis
p-value: 0.8779Kruskal-Wallis
Secondary
The Independent Tape Rater Scale - Fundamental Adherence Score
The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 fundamental MI items important for building rapport and understanding patients' experiences (e.g. reflections), rated for adherence (i.e. the extent/frequency of intervention delivery) on a scale from 1 (not at all) to 7 (extensively). The fundamental adherence score is the average of these 5 items, with scores ranging from 1-7 and higher scores indicating greater adherence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis.
Time frame: All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider.
Population: Analysis sample includes the N=114 MI sessions completed by providers with study-enrolled patients across all training groups.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| See One | The Independent Tape Rater Scale - Fundamental Adherence Score | 4.00 score on a scale | Standard Deviation 0.35 |
| Do One | The Independent Tape Rater Scale - Fundamental Adherence Score | 4.60 score on a scale | Standard Deviation 0.8 |
| Order One | The Independent Tape Rater Scale - Fundamental Adherence Score | 5.22 score on a scale | Standard Deviation 0.58 |
p-value: 0.0217Kruskal-Wallis
p-value: 0.0126Kruskal-Wallis
p-value: 0.4231Kruskal-Wallis
Secondary
The Independent Tape Rater Scale- Fundamental Competence Score
The Independent Tape Rater Scale assessed the integrity of MI interviews. The Independent Tape Rater Scale includes 5 fundamental MI items important for building rapport and understanding patients' experiences (e.g. reflections), rated for competency (i.e., the skill/quality of intervention delivery) on a scale of 1 (not at all) to 7 (excellent). The fundamental competence score is the average of these 5 items, for which there was adherence, with higher scores indicating greater competence. Only MI sessions conducted within each study arm by the providers with patient participants were included in this analysis. Since 3 MI sessions were conducted by providers in See One, 11 sessions were conducted by providers in Do One, and 100 sessions were conducted by providers in Order One, this analysis includes 114 MI sessions as the unit of analysis.
Time frame: All sessions conducted and recorded by providers with study-enrolled patients during the trial period, an expected 24 weeks per provider.
Population: Analysis population includes the N=114 MI sessions that were completed by providers across all training groups.
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| See One | The Independent Tape Rater Scale- Fundamental Competence Score | 3.67 score on a scale | Standard Deviation 0.61 |
| Do One | The Independent Tape Rater Scale- Fundamental Competence Score | 4.27 score on a scale | Standard Deviation 0.43 |
| Order One | The Independent Tape Rater Scale- Fundamental Competence Score | 4.71 score on a scale | Standard Deviation 0.45 |
p-value: 0.0091Kruskal-Wallis
p-value: 0.0196Kruskal-Wallis
p-value: 0.2832Kruskal-Wallis