Exploring perceptions of an artificial intelligence tool for health information.

The intervention/s employed in this study target the perceptions of warmth and competence in AI systems which participants will use to learn about atrial fibrillation. The study will employ a between-subjects 2x2 design: Variable 1 - Competence (perceived; high vs. low): participants will be randomised to read one of two text-based descriptions of an AI agent, one emphasising the agent’s intelligence, accuracy and capabilities, the other a more general description with no competence descriptors. Variable 2 - Warmth (high vs. low): after reading the competence description, participants will be randomly allocated to receive one of two private access links to distinct Cogniti agents (Cogniti is a site developed by the University of Sydney for people to design AI agents). These agents have differing system messages which have been written by the research team to instruct the agent of the characteristics to express and conversational styles to adopt during interactions. The high warmth agent expresses high social orientation in its conversational style. The low warmth agent has a direct and emotionally neutral conversational style, adopting the persona of a web-based search rather than a conversational agent. Other than the conversational styles and benevolence level, the agents are identical. The resulting groups will be: High competence x high warmth High competence x low warmth Low competence x high warmth Low competence x low warmth = active control group Participants will be randomly allocated to read a high- or low- competence text-based description. These descriptions are quite short and we anticipate that participants will require less than one minute of reading time. These descriptions will be administered within the survey via Qualtrics, not by any specific member of a research team. To ensure sufficient interaction with the AI agent, the participants will not be able to proceed with the experiment until five minutes has passed. Ongoing pilot testing has confirmed that this is sufficient time, and has indicated that at least five meaningful questions should be asked to obtain sufficient information. Transcripts will therefore be checked upon submission for evidence of this sufficient engagement (i.e. at least five questions asked). Participants will be provided with the following instruction to prompt their interaction with the AI agent: "Please spend five minutes using this agent to get a good understanding of atrial fibrillation. Read its responses carefully. You could ask about what the disease is, its symptoms, causes, diagnosis and treatments, and what life looks like with the disease." Information provided by the agents are entirely text-based. The high-warmth agent does use emoticons to communicate the social orientation, and the low-warmth agent does not. The interactions will all occur within the single session and the agent is not accessible by the participants outside this time window. Participants will access the study from their personal devices in a location of their choosing, wherever they have access to the internet. There will not be any in person sessions. To monitor the adherence to the intervention, we will be collecting manipulation confirmation responses (twelve items that measure warmth and competence), and will be conducting a sentiment analysis using Linguistic Inquiry and Word Count (LIWC) Analysis for the AI outputs and if applicable the questions that the participants ask the AI. Pilot Testing clarification: Informal pilot testing will be conducted with a distinct cohort, none of whom will be completing the actual survey. This will occur before recruitment into the study commences. The purpose of the pilot testing is to identify technological and administrative issues only, particularly with logging into Cogniti and accessing the agents. We are anticipating including 16 individuals in this informal pilot testing. The data obtained in this informal pilot testing will not be included in any data analysis in the general study.