Creating a Framework for Large Language Models for Caregiver Support in Dementia

Project Accomplishments: The CAREALL pilot project explored the use of large language models (LLMs) to support caregivers of individuals with dementia. The team developed a set of 75 common caregiver questions through focus groups and interviews, then compared responses generated by a trained LLM, a human geriatric care manager (GCM), and a hybrid of both. Clinicians evaluated the LLM’s responses as appropriate for clinical use, noting their empathy and informativeness. Caregivers were then asked to rank the responses blindly, and while final analysis is ongoing, early findings suggest that LLMs can provide high-quality, timely support with significantly less effort than human-generated responses. A guidebook of the 75 questions and answers was also created as a resource for caregivers. 

In addition to evaluating the LLM’s performance, the project conducted a modified Delphi study with clinicians, ethicists, engineers, and caregivers to develop an ethical framework for integrating LLMs into dementia caregiving. This included considerations around informed consent, explored further through 15 one-on-one caregiver interviews. The project has generated a rich dataset and is expected to result in at least three publications, contributing both practical tools and ethical guidance for the use of AI in caregiving. Future studies are planned to build on these findings and further refine the integration of LLMs into caregiver support systems. 

Initial Proposal Abstract: Caregiving in dementia is associated with a range of negative outcomes including worse mental health (depression, anxiety), caregiver burnout, financial strain and higher mortality. Geriatric care managers (GCMs) are a major resource for solutions, including (i) providing information (ii) enhancing caregiver competency and (iii) providing emotional support. However, GCM services are not typically covered by insurance and are not easy to access because of a shortage of GCMs. Thus, there is a need for solutions to fill this gap. The recent emergence of large language models (LLMs) such as ChatGPT that can provide information, support, and potentially guidance offers tantalizing possibilities. The increasing ubiquity of LLMs means that it is reasonable to anticipate that caregivers will increasingly turn to these tools for a range of solutions.

However, there are major unanswered questions, including the absence of any evidence base or meaningful best practices governing LLM use. Our project aims to make a critical initial foray into this space. We propose the use of an LLM developed by Kinto (now Rippl), a MassAITC awardee, that is based on the OpenAI platform and designed to support caregivers. We will recruit 25-30 caregivers of persons with dementia as project participants/stakeholders from McLean Hospital. We will develop a list of 100 representative caregiver queries and utilize both qualitative and quantitative approaches to compare how GCMs, LLMs, and GCMS with LLM support respond to these queries. Thus, we aim to identify specific domains of care management that may be best served by AI versus GCMs.

Additionally, we will gather diverse stakeholder input through a series of virtual stakeholder roundtables at the Harvard Medical School Center for Bioethics and Harvard Law School Petrie-Flom Center to solicit input on safeguards for LLMs and to develop an ethical framework for meaningful interaction between caregivers and LLMs.