Artificial Intelligence (AI), Artificial Intelligence Technology, Communication, Interdisciplinary Communication
Conditions
Brief summary
This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers' comprehension and satisfaction with the notes and ophthalmologists' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.
Interventions
OTHERLarge Language Model-generated Plain Language Summary of Ophthalmology notes
Prospective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes.
Sponsors
John J Chen
Study design
Allocation
RANDOMIZED
Intervention model
PARALLEL
Primary purpose
HEALTH_SERVICES_RESEARCH
Masking
NONE
Intervention model description
Randomized study assessing the efficacy of Large Language Model-generated Plain Language Summaries of Ophthalmology notes in increasing non-ophthalmologist note understanding. Notes were randomized to intervention (Summary + Original Note) or control (Original note only) arms. Non-ophthalmology providers were sent surveys assessing note comprehension for both control and treatment arm notes, and Ophthalmologists were sent surveys for intervention arm notes regarding Summary accuracy and other attributes. Conducted as a Quality Improvement initiative.
Eligibility
Sex/Gender
ALL
Age
18 Years to No maximum
Healthy volunteers
No
Inclusion criteria
* Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.
Exclusion criteria
* No in-network non-Ophthalmologist care providers in the medical record * Procedure-only visits containing only a procedure note
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| Non-Ophthalmologist Note Comprehension and Satisfaction | From enrollment to 8 weeks after enrollment | Assessed via survey responses evaluating understanding of the patient's ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary. Degree of understanding was assessed on the following Likert scale: Not at all Neutral Moderately A great deal Satisfaction was graded on the following Likert scale: Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied Preference between note types was assessed on the following Likert scale: Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| Ophthalmologist Evaluation of Summary Accuracy and Time burden | Assessed at a single time point <24 hr after enrollment | Assessed via survey responses regarding Plain Language Summary accuracy in reflecting the Standard Ophthalmology Note findings, time spent reviewing/editing, and perceived burden. Accuracy in reflecting findings was graded on the following Likert scale: Not at all Neutral A little A great deal Time spent reviewing/editing was reported using the following options: <1 minute 1. minute 2. minutes 3. minutes 4. minutes 5. or more minutes Perceived burden was graded on the following Likert scale: Not at all Neutral A little A great deal |
| Semantic and Linguistic Quality of Plain Language Summaries | Assessed at a single time point &lt;24 hr following enrollment | Flesch Reading Ease Scale: 0 (poor) to 100 (good) |
Other
| Measure | Time frame | Description |
|---|---|---|
| Presence of Errors, Type of Errors, and Potential Harm of Errors | Assessed at a single time point &amp;amp;amp;amp;lt;24 hr following enrollment and again at a single time point 8 weeks after enrollment | Presence of any incorrect or missing information in the Plain Language Summary, error rates, types of errors, and potential harm rating. Types of errors were reported as: HPI Exam findings Diagnosis Plan Explanation of the patient&#39;s overall eye health Other N/A Likelihood of harm was graded on the following scale: Low risk Medium risk High risk N/A Potential harm rating was graded on the following scale: Mild or moderate harm Death or severe harm |
Countries
United States
Outcome results
None listed