Patient Communication
Conditions
Brief summary
The purpose of this study is to refine and test existing enterprise-grade large language model (LLM) based on generative artificial intelligence (AI), to assess the feasibility and acceptability of LLM-based feedback, to assess the ability of LLM-based feedback to improve residents' communications,to explore the ability of standardized patients to assess residents' communication and to explore the ability of residents to self-assess their communication complexity
Interventions
BEHAVIORALeducational LLM-based feedback tool
Participants will have their verbal communications with standardized patients (SP) regarding 3 different scenarios recorded, transcribed, and analyzed in real-time by the large language model (LLM) and will receive feedback as suggestions and alternative scripts. These will be reviewed by residents between SP scenarios
Sponsors
Health Science Education Small Grants Program
The University of Texas Health Science Center, Houston
Study design
Allocation
RANDOMIZED
Intervention model
PARALLEL
Primary purpose
OTHER
Masking
NONE
Eligibility
Sex/Gender
ALL
Age
18 Years to 50 Years
Healthy volunteers
Yes
Inclusion criteria
* McGovern Medical School (MMS) general surgery residents * postgraduate year (PGY) 1-5
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| acceptability of future use | end of intervention ( 1 hour after baseline) | This is scored from 1( very unlikely) to 5 (very likely) |
| Correctness of recommendations as assessed by a survey | end of intervention ( 1 hour after baseline) | This will be reported on a 5 point Likert scale form 1 very incorrect to 5 very correct |
| Applicability of recommendations as assessed by a survey | end of intervention ( 1 hour after baseline) | This will be reported on a 5 point Likert scale form 1 very inapplicable to 5 very applicable |
| Perceived readability of resident-standardized patient (SP) interactions as assessed by a survey: schooling level | end of intervention ( 1 hour after baseline) | This will be categorically reported in the following categories: Elementary middle high college graduate |
| confidence in communication ability | end of intervention ( 1 hour after baseline) | This is scored from 1( very unconfident) to 5 (very confident) |
| usefulness of the LLM | end of intervention ( 1 hour after baseline) | This is scored from 1( very useless) to 5 (very useful) |
| Readability discernment as assessed by a survey | end of intervention ( 1 hour after baseline) | This will be scored by the by Cohen's Kappa values from 1-5. Higher Cohen's kappa scores mean better outcome |
| Quality discernment as assessed by a survey | end of intervention ( 1 hour after baseline) | This will be scored by the by Cohen's Kappa values from 1-5. Higher Cohen's kappa scores mean better outcome |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| readability grade level of resident-SP transcripts as assessed by the Flesch-Kincaid Grade Level (FKGL) readability tool | end of intervention ( 1 hour after baseline) | Readability of resident-SP encounter transcripts will be assessed using the Flesch-Kincaid Grade Level formula, which estimates the U.S. school grade level required to understand the text. Higher scores indicate a higher reading grade level (i.e., lower readability). Formula used: Grade level= 0.39(total words/total sentences) + 11.8 (total syllables/total words)-15.59 |
| Quality based on Ensuring Quality Information for Patients (EQIP) score of resident-SP transcripts | end of intervention ( 1 hour after baseline) | Percentage score based on a validated questionnaire This has 20 questions and each is scored from 1(yes), 0.5(partly), 0 (no) and question is removed if it does not apply.Scores are reported as a percentage and higher percentage score indicates better quality |
| Perceived readability of SP-resident interactions as assessed by a standardized survey | end of intervention ( 1 hour after baseline) | This will be categorically reported in the following categories: Elementary middle high college graduate |
| confidence in communication ability | end of intervention ( 1 hour after baseline) | This is scored from 1( very unconfident) to 5 (very confident) |
| Survey feedback on the LLM interface | end of intervention ( 1 hour after baseline) | This is scored from 1( very unrealistic) to 5 (very realistic) |
Countries
United States
Contacts
Primary ContactKrislynn M Mueck, MD, MS, MPH
Backup ContactWilliam D Rieger
Outcome results
None listed