Atrial Fibrillation (AF), Coronary Artery Disease, Heart Failure, Stroke
Conditions
Brief summary
This study aims to explore the feasibility of using medical large language models to assist in chronic disease management. By setting up experimental and control group interventions and having experts blindly evaluate anonymized cases, it compares different management approaches in chronic disease care, verifying the scientific basis, effectiveness, and potential for broader use of medical large language models in supporting chronic disease management.
Interventions
DEVICEMedical large language model
The primary care physicians in the intervention group completed the management decisions for chronic disease cases, including disease assessment, examination suggestions, treatment plans, selection of management methods, and health education, with the assistance of the medical LLM.
Sponsors
Jiangsu Taizhou People's Hospital
Study design
Allocation
RANDOMIZED
Intervention model
PARALLEL
Primary purpose
OTHER
Masking
SINGLE (Outcomes Assessor)
Eligibility
Sex/Gender
ALL
Age
18 Years to 65 Years
Healthy volunteers
No
Inclusion criteria
1. Work experience of 3 years or more; 2. Able to complete the case assessment tasks required by the study; 3. Have a practicing doctor qualification; 4. Work at a township health center or community health service center; 5. Voluntarily participate in the study and sign the informed consent form.
Exclusion criteria
N/A
Design outcomes
Primary
| Measure | Time frame | Description |
|---|---|---|
| expert overall scores | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale on several aspects: accuracy of condition assessment, reasonableness of test recommendations, reasonableness of treatment plans, appropriateness of management choices, scientific nature of patient education content, feasibility at the primary care level, overall clinical quality. Experts were instructed to rank the seven dimensions from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
Secondary
| Measure | Time frame | Description |
|---|---|---|
| accuracy of condition assessment | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale on accuracy of condition assessment from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| reasonableness of test recommendations | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the reasonableness of test recommendations. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| reasonableness of treatment plans | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the reasonableness of treatment plans. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| appropriateness of management choices | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the ppropriateness of management choices. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| scientific nature of patient education content | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the scientific nature of patient education content. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| feasibility at the primary care level | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the feasibility at the primary care level. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| the overall clinical quality | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. They rated them using a Likert scale regarding the overall clinical quality. Experts were instructed to rank the dimension from lowest to highest quality. These ordinal rankings were then converted into standardized scores of 1 to 5. Higher scores mean better. |
| risk advice | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. If there is no risk advice, the expert selects 'No'; otherwise, the expert selects 'Yes' and specifically lists the content of the risk advice. |
| Any diagnostic or treatment information missing | through study completion, an average of 3 months. | A total of 200 cases (50 each of coronary heart disease, atrial fibrillation, heart failure, and stroke) were grouped by disease type and independently evaluated in a blinded manner by three subspecialty experts. For each case, the two management plans were anonymized in the same way and then randomly labeled by a computer as Plan A and Plan B. Experts could only view the anonymized case information and the two corresponding anonymized management plans. If there is no omission of diagnostic and treatment information, the expert selects "No"; otherwise, the expert selects "Yes" and specifically lists the omitted diagnostic and treatment content. |
| feedback regarding the use of medical LLM by primary care physicians | through study completion, an average of 3 months. | After primary care physicians complete a case assessment assisted by medical LLM , they will fill out an evaluation of the medical LLM usage, including whether the medical LLM helped in disease assessment, assisted in formulating examination plans, aided in developing treatment plans, improved chronic disease management capabilities, supported chronic disease education, assisted in management approach selection, enhanced confidence, saved time, whether there were hallucinations or omissions, whether it provided risky suggestions, and any additional comments on the use of the medical LLM. |
Countries
China
Contacts
CONTACTGuoyu Wang, MD
Outcome results
None listed