Coronary Heart Disease (CHD)
Conditions
Keywords
artificial intelligence
Brief summary
This prospective, multi-reader, randomized crossover trial evaluates SCOUT (Scalable Clinical Oversight via Uncertainty Triangulation), a model-agnostic meta-verification framework that selectively defers unreliable large language model (LLM) predictions to clinicians by triangulating three orthogonal uncertainty signals: model heterogeneity, stochastic inconsistency, and reasoning critique. The trial assesses whether SCOUT-assisted review can reduce physician review time compared with standard manual review of AI-generated diagnoses while maintaining non-inferior diagnostic accuracy in coronary heart disease (CHD) subtyping.
Detailed description
Background: Large language models are increasingly deployed in clinical workflows, yet requiring clinician review of every AI output negates the efficiency gains that motivate their adoption. SCOUT addresses this efficiency-safety paradox through algorithmic meta-verification. The SCOUT framework triangulates three orthogonal external signals to determine case-level uncertainty: (1) Model Heterogeneity - whether a structurally different auxiliary LLM agrees with the primary model; (2) Stochastic Inconsistency - whether repeated sampling from the same model yields divergent outputs; (3) Reasoning Critique - whether an external checker model identifies logical flaws in the chain-of-thought reasoning. In this crossover trial, 7 clinicians of varying seniority (2 junior residents, 3 senior residents, 2 attending physicians) each review all 110 cases under both standard manual review and SCOUT-assisted review workflows. The study evaluates workflow efficiency (primary endpoint) and diagnostic accuracy (secondary endpoint).
Interventions
DIAGNOSTIC_TESTSCOUT-Assisted Review Workflow
SCOUT-Assisted Review (Intervention Arm): Physicians review 56 cases processed through the SCOUT framework. For cases classified as low-uncertainty (D(x)=0), the AI prediction is auto-accepted without physician review. For high-uncertainty cases (D(x)=1), the physician reviews the case with access to the main model's chain-of-thought reasoning and the meta-verification audit results. The main model is DeepSeek-V3.1 with chain-of-thought prompting.
DIAGNOSTIC_TESTStandard Manual Review Workflow
Physicians perform a full manual review of 54 cases using raw medical records with access to the AI model's predictions and reasoning, but without SCOUT uncertainty stratification or selective deferral.
Sponsors
China National Center for Cardiovascular Diseases
Study design
Allocation
RANDOMIZED
Intervention model
CROSSOVER
Primary purpose
DIAGNOSTIC
Masking
NONE
Eligibility
Sex/Gender
ALL
Age
18 Years to No maximum
Healthy volunteers
No
Inclusion criteria
* Board-certified or in-training cardiologists at Fuwai Hospital * Spanning three experience strata: junior residents, senior residents, attending physicians
Exclusion criteria
* Clinicians involved in the development or optimization of the SCOUT framework * Clinicians involved in the gold-standard adjudication process
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| Mean physician review time per case (minutes) | Through study completion, an average of 2 hours. | Mean time spent by each clinician reviewing and rendering a diagnostic decision per case under each arm. Measured in minutes. |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| Diagnostic accuracy (%) | Through study completion, an average of 2 hours. | Proportion of correct CHD subtype classifications (STEMI, NSTEMI, unstable angina, chronic coronary syndromes) under each arm. |
| Computational Return on Investment (ROI) | Through study completion, an average of 2 hours. | Ratio of physician time savings (valued at standardized minute-wages from Sanming healthcare reform benchmarks) to computational cost of SCOUT inference, stratified by clinician seniority level. |
Contacts
CONTACTXiaojin Gao, Dr.
Outcome results
None listed