Not applicable Not applicable
Conditions
Interventions
Validation framework for ambient listening technologies in Dutch healthcare.
Sponsors
Erasmus MC, Universitair Medisch Centrum Rotterdam
Eligibility
Age
18 Years to 99 Years
Inclusion criteria
Inclusion criteria: Phase 1: Expert input and consensus Study For the Steering Group, experts will be approached by the initiators. Experts will be selected based on experience with research involving AI in healthcare, experience with methodology, or experience with developing guidelines. Phase 1a: Workshops For the Ethical Workshop Participants: must represent one of the key stakeholder perspectives: healthcare professional, patient/client/relative/informal carer, manager or policymaker (including legal or ethical experts), or ICT/technical professional. Must be willing to participate in a 3.5–4-hour workshop following the “Aanpak Begeleidingsethiek” methodology (10). Must be able to provide informed consent to participate in group discussions and the recording of anonymized qualitative data. Are joining the Delphi rounds.Phase 1b: Modified Delphi rounds For the Delphi rounds, experts will be recruited by five different channels: 1) invitation to experts recommended by the Steering Group, 2) invitations to key stakeholders and interests connected to the consortium, 3) call to contribute published in a commentary article in a medical journal, 4) consideration of any expert contacting the Steering Group on their own initiative, and 5) invitation to experts recommended by the Delphi participants (snowballing). Phase 2: On-site Clinical evaluation of the Framework on an existing applicationThe first version of the framework will be tested and applied to the METC approved study: protocol nr. METC-2024-0733. This study investigates the use of a large language model (LLM)–based ambient listening tool for recording and summarizing family conversations in the ICU setting. This pilot validation will serve as a key first test of the framework’s applicability, clarity, and completeness in a high-stakes clinical environment.Following on-site validation teams will be recruited through: 1) Invitations to healthcare institutions participating in the RIGH:T consortium. 2) Invitations to relevant stakeholders and partner organizations connected to the consortium. Institutions will be eligible for participation if they: 1) Have the opportunity to apply all relevant steps of the RIGH:T validation framework within the planned study. 2) Are able to implement and operationalize an ambient listening solution in their care setting. 3) Commit to developing a study-specific research protocol jointly with the RIGH:T research team.
Exclusion criteria
Exclusion criteria: For All Participants:Unwillingness or inability to provide informed consent.Additional for phase 1: expert input and consensus study Participants:Lack of sufficient expertise or background in one of the relevant domains (clinical, ethical, technical, organizational, or patient perspective).Inability to complete the Delphi questionnaire or attend scheduled workshop sessions due to time or language barriers.Additional for phase 2: On-site Clinical evaluation of the Framework on an existing application:Rejection of the research protocol for the clinical evaluation of the framework on an ambient listening application by the METC of that institution.
Design outcomes
Primary
| Measure | Time frame |
|---|---|
| Phase 1: Expert input and consensus StudyPhase 1a: WorkshopsEssential elements that should be included in the RIGH:T validation framework.Phase 1b: Modified Delphi processConsensus on framework components (technical, clinical, and societal dimensions, including error taxonomies and thresholds).Suggestions for altering framework components and their order.Phase 2: Clinical evaluation of the Framework on an existing ApplicationFeedback on the usability and completeness of the RIGH:T validation framework on a prospective clinical validation of a large language model (LLM)–based ambient listening tool. | — |
Secondary
| Measure | Time frame |
|---|---|
| Phase 1: Expert input and consensus StudyPhase 1a: WorkshopsMethodologies to find estimations of possible performance, clinical impact and societal impact thresholds to compare framework outcomes to. Phase 1b: Modified Delphi processAssessment of defined methodologies to find estimations of possible performance, clinical impact and societal impact thresholds to compare framework outcomes to. Participants are also asked to add their own suggestions of methodologies for defining thresholds.Phase 2: Clinical evaluation of the Framework on an existing ApplicationAverages and ranges of performance, clinical impact and societal impact values collected by the local validation team by using the framework. These values are only used for defining a methodology for determining ranges or thresholds for framework metrics. They are not used to compare ambient listening solutions or healthcare institutions. The values are reported during semi-structured interviews. | — |
Countries
Netherlands
Contacts
Public ContactM.E. Genderen
Erasmus MC, Universitair Medisch Centrum Rotterdam
Outcome results
None listed