Evaluating the use of generative AI to improve the quality and timeliness of junior doctor written discharge summaries in Australia: Exploratory blinded simulation study.

Delivered at home 3 junior doctors given AI-assistance using a Heidi Health subscription supplied by the company, accessed by desktop website. 2 junior doctors non-AI assistance. Each doctor given 20 records derived from the MIMIC-IV hospital admissions repository - a retrospective deidentified cohort of ICU admission in US hospitals specifically designed for research purposes containing full transcripts of all features of the admission including pathology, radiology, and other investigations, and referrals and reviews by consult teams. The records were randomly chosen from 400 downloaded records, allocated based on a weighted file size balance (which qualitatively reflected both the length and complexity of admissions). AI-assistance participants were given no special training, only that provided to all first-time users via the Heidi Health scribe software. AI-assisted all returned within the 30 day window. Human only participants given 2 extensions of 30 days each for completion at two follow ups due to non-return of summaries. Adherence was monitored by return of summaries to the pre-specified collection email address. The Heidi discharge summary template was aligned to the ACSQHC template for digital presentation of hospital discharge summaries with no other instructions given in the prompts. Participants could attach the deidentified patient event to the template and generate, review, edit and finalise the script before returning it on identical standardised discharge summary template docx file which all participants (AI-asssist and human-only) used for their summary completion. The study protocol included a Delphi method for managing assessment and any significant disagreements between assessors on the scoring of any one discharge summary. Delphi participants included 3 general practitioners, 2 hospital-based consultants and 2 senior registrars. A prespecified threshold of disagreement (>80% difference in scores on the two assessment tools) amongst reviewers was agreed as a trigger a review. Delphi participants were given drafts of the 100-point assessment rubric with the background of its creation in Australia. There were no amendments or changes to the agreed assessment process of both the assessment rubric and PDQI-9. Discussion regarding limitations of the use of US data and the limits of simulation were discussed prior to commencement, with no major disagreements on strategies to manage these issues. At completion of assigned assessments, no summaries had reached the 80% disagreement threshold triggering further summary review.