After Cataract, Cataract
Conditions
Brief summary
This project will apply AI technology to meet the gap between increasing demand and limited capacity of high- volume healthcare services. The project will develop evidence that will support the safe deployment of Ufonia's automated telemedicine platform to deliver calls to cataract surgery patients at two large NHS hospital trusts. The proposed study will implement DORA in addition to the current standard of care for a cohort of patients at Imperial College Healthcare Trust and Oxford University Hospitals NHS Foundation Trust. The study will evaluate the agreement of DORA's decision with an expert clinician. In addition it will test the acceptability of the solution for patients and clinicians; the sensitivity and specificity of the system in deciding if a patient requires additional review; and the health economic benefits of the solution to patients (reduced time and travel) and the local healthcare system. If successful, a proposal will be developed to roll the solution out to all patients at each site in anticipation of an application to a late phase award for wider NHS deployment.
Detailed description
Background Due to an ageing population and increased expectation, the demand for many services is exceeding the capacity of the clinical workforce. As a result, staff are facing a crisis of burnout from being pressured to deliver high- volume workloads, driving increasing costs for providers. Artificial intelligence, in the form of conversational agents, presents a possible opportunity to enable efficiencies in the delivery of care. Aims and Objectives This study aims to evaluate the effectiveness, usability and acceptability of DORA - an AI-enabled autonomous telemedicine call - for detection of post-operative cataract surgery patients who require further assessment. The study's objectives are: to establish efficacy of DORA's decision making in comparison to an expert human clinician; baseline sensitivity and specificity for detection of true complications; evaluation of patient acceptability; evidence for cost-effectiveness; and to capture data that may support further studies. Project plan and methods used Based on implementation science, the interdisciplinary study will be a mixed-methods phase one pilot establishing inter-observer reliability; as well as usability and acceptability. Timelines for delivery The study will last eighteen months: seven months of evaluation and intervention refinement, nine months of implementation and follow-up, and two months of post-evaluation analysis and write-up. Anticipated Impact and Dissemination The project's key contributions will be evidence on artificial intelligence voice conversational agent effectiveness, and associated usability and acceptability. Results will be disseminated in peer-reviewed journals and at international medical sciences and engineering conferences.
Interventions
OTHERDora
DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards.
Sponsors
University of Oxford
Imperial College London
University of Plymouth
Study design
Allocation
NA
Intervention model
SINGLE_GROUP
Primary purpose
SCREENING
Masking
NONE
Masking description
No masking is possible, because all participants receive the same intervention.
Intervention model description
The study will be a multi-centre, mixed-methods clinical investigation to develop evidence regarding the feasibility, acceptability and potential effectiveness of Dora.
Eligibility
Sex/Gender
ALL
Healthy volunteers
No
Inclusion criteria
* Willing and able to provide informed consent; * Aged 18 years or older; * On the waiting list for routine cataract surgery. Cataract surgery as part of a combined procedure with other ocular surgery will not be included; * No history or presence of significant ocular comorbidities that would be expected to alter the risks of cataract surgery or normal post-operative follow-up schedule. Note that significant ocular comorbidities do not include stable, chronic, or inactive ocular conditions such as amblyopia, drop-controlled stable glaucoma or ocular hypertension, previous squint surgery, inactive macular pathology, previous refractive surgery, or previous vitreoretinal surgery with stable retina.
Exclusion criteria
* Individuals with any condition that could preclude the ability to comply with the study or follow-up procedures; * Presence of ocular or systemic uncontrolled disease (unless deemed not clinically significant by the Investigator and Sponsor); * Involved in current research related to this technology or been involved in related research to this technology prior to recruitment; * Cognitive difficulties, hearing impairment or non-English speakers; * History of current or severe, unstable or uncontrolled systemic disease (unless deemed not clinically significant by the Investigator and Sponsor).
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| Agreement | Inter-rater reliability was assessed based on data collected during Dora calls, which lasted an average of 7.5 minutes | Inter-rater reliability: the degree of agreement between DORA and the clinician on their assessments of the individual symptoms and the management plan; Whether or not the clinician had to interrupt the call to ask clarifying questions |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| Calls Completed Without Intervention | Dora calls lasted an average of 7.5 minutes | Number of autonomous calls that were completed without needing any intervention from the supervising clinician; Clinician-reported reasons for asking clarifying questions |
| System Usability | Usability assessments were completed up to 6 months after the Dora call | Measured using the System Usability Scale (minimum of 0, maximum of 100, higher scores indicate better usability) |
| Usability of Telehealth System Implementation | Usability was assessed up to 6 months after the call | Measured using the Telehealth Usability Questionnaire (minimum score of 1, maximum score of 5, averaged across 19 items; higher scores indicate better usability) |
| Qualitative Patient Perspectives of Usability | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call | Qualitative feedback from semi-structured interviews |
| Clinical Complications Identified or Missed by DORA System | Up to 90 days post surgery | Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unexpected management change |
| Satisfaction With AI Follow-up Phone Call | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call | Qualitative feedback from semi-structured interviews |
| Appropriateness of AI for Follow-up Assessment | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call | Qualitative feedback from semi-structured interviews |
| Cost Impact | Conducted 6 months after baseline | A cost analysis compared the direct costs of face-to-face (F2F) follow-up at Imperial with Dora R1 (in Oxford, patients do not have routine postoperative follow-up). Assumptions included annual costs for various healthcare professionals and the duration of F2F follow-up appointments (estimated at 30 min). |
| Subsequent Unplanned Follow-up | Up to 90 days post surgery | Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unplanned review. |
| Acceptability of AI Follow-up Phone Call | Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call | Qualitative feedback from semi-structured interviews |
Countries
United Kingdom
Participant flow
Participants by arm
| Arm | Count |
|---|---|
| Dora Follow-up Phone Call DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. The entire conversation will be supervised by a clinician. This clinician will be able to interrupt the call at any point if the system fails, the patient struggles to interact with it, or DORA does not collect sufficient information from the patient. The clinician will record a clinical assessment which will be compared to the DORA assessment.
Dora: DORA uses a variety of AI technologies to deliver the patient follow-up call, including: speech transcription, natural language understanding, a machine-learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. DORA is configured to deliver calls through a telephone connection as a real-time, stand-alone system: the operator inputs individual patient details to initiate the call and completes a summary in the electronic health record (EHR) afterwards. | 202 |
| Total | 202 |
Withdrawals & dropouts
| Period | Reason | FG000 |
|---|---|---|
| Overall Study | Lost to Follow-up | 12 |
| Overall Study | Protocol Violation | 9 |
| Overall Study | Withdrawal by Subject | 2 |
Baseline characteristics
| Characteristic | Dora Follow-up Phone Call |
|---|---|
| Age, Continuous | 74 years STANDARD_DEVIATION 8.96 |
| Education level Bachelor's Degree or Higher | 43 Participants |
| Education level Lower Than Bachelor's Degree | 88 Participants |
| First or second cataract surgery First eye | 113 Participants |
| First or second cataract surgery Second eye | 89 Participants |
| Income ≤ £19,999/yr | 28 Participants |
| Income £20,000-29,999/yr | 24 Participants |
| Income £30,000-39,999/yr | 5 Participants |
| Income £40,000-49,999/yr | 5 Participants |
| Income £50,000-69,999/yr | 9 Participants |
| Income >£70,000/yr | 6 Participants |
| Income Undefined | 70 Participants |
| Race/Ethnicity, Customized Ethnicity Asian | 12 Participants |
| Race/Ethnicity, Customized Ethnicity Black | 11 Participants |
| Race/Ethnicity, Customized Ethnicity Other | 17 Participants |
| Race/Ethnicity, Customized Ethnicity White | 160 Participants |
| Sex: Female, Male Female | 116 Participants |
| Sex: Female, Male Male | 86 Participants |
Adverse events
| Event type | EG000 affected / at risk |
|---|---|
| deaths Total, all-cause mortality | 0 / 202 |
| other Total, other adverse events | 4 / 202 |
| serious Total, serious adverse events | 0 / 202 |
Outcome results
Primary
Agreement
Inter-rater reliability: the degree of agreement between DORA and the clinician on their assessments of the individual symptoms and the management plan; Whether or not the clinician had to interrupt the call to ask clarifying questions
Time frame: Inter-rater reliability was assessed based on data collected during Dora calls, which lasted an average of 7.5 minutes
Population: Calls to 7 of the 202 were incomplete.
| Arm | Measure | Category | Value (COUNT_OF_PARTICIPANTS) |
|---|---|---|---|
| Dora Outcome Decision | Agreement | Discharge | 117 Participants |
| Dora Outcome Decision | Agreement | Review | 78 Participants |
| Supervising Clinician Decision | Agreement | Discharge | 131 Participants |
| Supervising Clinician Decision | Agreement | Review | 64 Participants |
Secondary
Acceptability of AI Follow-up Phone Call
Qualitative feedback from semi-structured interviews
Time frame: Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call
Population: Participants invited for semi-structured interviews - categories reflect major components of the Theoretical Framework of Acceptability and the number of participants who provided feedback that was coded into those categories.
| Arm | Measure | Group | Value (NUMBER) |
|---|---|---|---|
| Dora Outcome Decision | Acceptability of AI Follow-up Phone Call | Affective attitude (anticipated) | 15 participants |
| Dora Outcome Decision | Acceptability of AI Follow-up Phone Call | Affective attitude (experienced) | 20 participants |
| Dora Outcome Decision | Acceptability of AI Follow-up Phone Call | Ethicality | 5 participants |
Secondary
Appropriateness of AI for Follow-up Assessment
Qualitative feedback from semi-structured interviews
Time frame: Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call
Population: Participants invited for semi-structured interviews - categories reflect major components of the Theoretical Framework of Acceptability and the number of participants who provided feedback that was coded into those categories.
| Arm | Measure | Group | Value (NUMBER) |
|---|---|---|---|
| Dora Outcome Decision | Appropriateness of AI for Follow-up Assessment | Perceived effectiveness (anticipated) | 4 participants |
| Dora Outcome Decision | Appropriateness of AI for Follow-up Assessment | Perceived effectiveness (experienced) | 19 participants |
| Dora Outcome Decision | Appropriateness of AI for Follow-up Assessment | Perceived effectiveness (long-term) | 19 participants |
Secondary
Calls Completed Without Intervention
Number of autonomous calls that were completed without needing any intervention from the supervising clinician; Clinician-reported reasons for asking clarifying questions
Time frame: Dora calls lasted an average of 7.5 minutes
| Arm | Measure | Group | Value (NUMBER) |
|---|---|---|---|
| Dora Outcome Decision | Calls Completed Without Intervention | Call completed autonomously | 195 DORA follow-up calls |
| Dora Outcome Decision | Calls Completed Without Intervention | Call incomplete | 7 DORA follow-up calls |
Secondary
Clinical Complications Identified or Missed by DORA System
Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unexpected management change
Time frame: Up to 90 days post surgery
| Arm | Measure | Value (COUNT_OF_PARTICIPANTS) |
|---|---|---|
| Dora Outcome Decision | Clinical Complications Identified or Missed by DORA System | 10 Participants |
Secondary
Cost Impact
A cost analysis compared the direct costs of face-to-face (F2F) follow-up at Imperial with Dora R1 (in Oxford, patients do not have routine postoperative follow-up). Assumptions included annual costs for various healthcare professionals and the duration of F2F follow-up appointments (estimated at 30 min).
Time frame: Conducted 6 months after baseline
Population: Patients on the Imperial post-cataract surgery pathway
| Arm | Measure | Group | Value (NUMBER) |
|---|---|---|---|
| Dora Outcome Decision | Cost Impact | Total staff cost (standard pathway) | 4555.16 Great British Pounds |
| Dora Outcome Decision | Cost Impact | Total staff cost (Dora pathway) | 1084.21 Great British Pounds |
Secondary
Qualitative Patient Perspectives of Usability
Qualitative feedback from semi-structured interviews
Time frame: Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call
Population: Participants invited for semi-structured interviews - categories reflect major components of the Theoretical Framework of Acceptability and the number of participants who provided feedback that was coded into those categories.
| Arm | Measure | Group | Value (NUMBER) |
|---|---|---|---|
| Dora Outcome Decision | Qualitative Patient Perspectives of Usability | Burden | 18 participants |
| Dora Outcome Decision | Qualitative Patient Perspectives of Usability | Opportunity costs | 12 participants |
| Dora Outcome Decision | Qualitative Patient Perspectives of Usability | Self-efficacy | 5 participants |
| Dora Outcome Decision | Qualitative Patient Perspectives of Usability | Intervention coherence | 17 participants |
Secondary
Satisfaction With AI Follow-up Phone Call
Qualitative feedback from semi-structured interviews
Time frame: Semi-structured interview call, lasting up to 30 minutes, conducted up to 6 months after the Dora call
Population: Interviewed participants were asked if they would be willing to use Dora again; their responses were coded into the three categories below.
| Arm | Measure | Group | Value (NUMBER) |
|---|---|---|---|
| Dora Outcome Decision | Satisfaction With AI Follow-up Phone Call | Willing | 15 participants |
| Dora Outcome Decision | Satisfaction With AI Follow-up Phone Call | Not willing | 1 participants |
| Dora Outcome Decision | Satisfaction With AI Follow-up Phone Call | Hesitant, but willing | 4 participants |
Secondary
Subsequent Unplanned Follow-up
Clinical data was collected from patients' electronic health record (EHR) up to 90 days postoperatively to capture numbers of participants 'recommended discharge' by Dora R1 with subsequent unplanned review.
Time frame: Up to 90 days post surgery
| Arm | Measure | Value (COUNT_OF_PARTICIPANTS) |
|---|---|---|
| Dora Outcome Decision | Subsequent Unplanned Follow-up | 5 Participants |
Secondary
System Usability
Measured using the System Usability Scale (minimum of 0, maximum of 100, higher scores indicate better usability)
Time frame: Usability assessments were completed up to 6 months after the Dora call
Population: Not all participants completed the usability questionnaire
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| Dora Outcome Decision | System Usability | 77.76 units on a scale | Standard Deviation 17.55 |
Secondary
Usability of Telehealth System Implementation
Measured using the Telehealth Usability Questionnaire (minimum score of 1, maximum score of 5, averaged across 19 items; higher scores indicate better usability)
Time frame: Usability was assessed up to 6 months after the call
Population: Not all participants completed the usability questionnaire
| Arm | Measure | Value (MEAN) | Dispersion |
|---|---|---|---|
| Dora Outcome Decision | Usability of Telehealth System Implementation | 3.79 units on a scale | Standard Deviation 0.89 |