Year 3 Pilots

Authors: Ziping Xu, Hinal Jajal, Sung Won Choi, Inbal Nahum-Shani, Guy Shani, Alexandra M. Psihogios, Pei-Yao Hung, Susan A. Murphy Abstract Medication adherence is critical for the recovery of adolescents and young adults (AYAs) who have undergone hematopoietic cell transplantation. However, maintaining adherence is challenging for AYAs after hospital discharge, who experience both individual (e.g. physical and emotional symptoms) and interpersonal barriers (e.g., relational difficulties with their care partner, who is often involved in medication management). To optimize the effectiveness of a three-component digital intervention targeting both members of the dyad as well as their relationship, we propose a novel Multi-Agent Reinforcement Learning (MARL) approach to personalize the delivery of interventions. By incorporating the domain knowledge, the MARL framework, where each agent is responsible for the delivery of one intervention component, allows for faster learning…

Authors: Luton Zou, Ziping Xu, Daiqi Gao, Susan Murphy Abstract It is well known that in reinforcement learning (RL) different reward functions may lead to the same optimal policy, while some reward functions can be substantially easier to learn. In this paper, we propose a framework for reward design by constructing surrogate rewards with mediators informed by causal directed acyclic graphs (DAGs), which are often available in real-world applications through domain knowledge. We show that under the surrogacy assumption, the proposed reward is unbiased and has lower variance than the primary reward. Specifically, we use an online reward design agent that adaptively learns the target surrogate reward in an unknown environment. Feeding the surrogate rewards to standard online learning oracles, we show that the regret bound can be improved. Our framework provides a theoretical improvement…

Authors: Asim H. Gazi, Daiqi Gao, Susobhan Ghosh, Ziping Xu, Anna Trella, Predrag Klasnja, Susan A. Murphy Abstract Just-in-time adaptive interventions (JITAIs) are nascent precision medicine systems that extend personalized healthcare support to everyday life. A challenge in designing JITAIs is that personalized support often involves sophisticated decision-making algorithms. These decision-making algorithms can require numerous non-trivial design decisions that must be made between successive JITAI deployments (e.g., hyperparameter selection for an artificial intelligence algorithm). Making design decisions between deployments–rather than during deployment–ensures intervention fidelity and enhances the ability to replicate results. Yet, each deployment can be costly, precluding the use of A/B testing for every design decision. How should design decisions be made strategically between JITAI deployments? This paper introduces digital twins for just-in-time adaptive interventions (JITAI-Twins) to address this question. JITAI-Twins are “digital twins…