Pilot A21 – Nahum-Shani

Authors: Kelly W Zhang, Nowell Closser, Anna L Trella, Susan A Murphy Abstract Adaptive treatment assignment algorithms, such as bandit algorithms, are increasingly used in digital health intervention clinical trials. Frequently the data collected from these trials is used to conduct causal inference and related data analyses to decide how to refine the intervention, and whether to roll-out the intervention more broadly. This work studies inference for estimands that depend on the adaptive algorithm itself; a simple example is the mean reward under the adaptive algorithm. Specifically, we investigate the replicability of statistical analyses concerning such estimands when using data from trials deploying adaptive treatment assignment algorithms. We demonstrate that many standard statistical estimators can be inconsistent and fail to be replicable across repetitions of the clinical trial, even as the sample size grows large.…

Authors: Inbal Nahum-Shani, Susan A Murphy Abstract The past decade has seen a surge in developing just-in-time adaptive interventions (JITAIs)-an intervention approach that leverages advancements in digital technologies to address the rapidly changing needs of individuals in daily life. This article provides an overview of the state of science on JITAI development and highlights important directions for future research. We explain what a JITAI is (and what it is not) and review the scientific and practical rationales underlying this approach. We also call attention to three key challenges relating to the development of JITAIs. The first challenge is that individuals may not be able to engage with (i.e., invest energy in) an intervention when they need it most in daily life. The second concerns the generally suboptimal engagement of individuals in interventions that leverage digital…

Authors: Ziping Xu, Hinal Jajal, Sung Won Choi, Inbal Nahum-Shani, Guy Shani, Alexandra M. Psihogios, Pei-Yao Hung, Susan A. Murphy Abstract Medication adherence is critical for the recovery of adolescents and young adults (AYAs) who have undergone hematopoietic cell transplantation. However, maintaining adherence is challenging for AYAs after hospital discharge, who experience both individual (e.g. physical and emotional symptoms) and interpersonal barriers (e.g., relational difficulties with their care partner, who is often involved in medication management). To optimize the effectiveness of a three-component digital intervention targeting both members of the dyad as well as their relationship, we propose a novel Multi-Agent Reinforcement Learning (MARL) approach to personalize the delivery of interventions. By incorporating the domain knowledge, the MARL framework, where each agent is responsible for the delivery of one intervention component, allows for faster learning…

Authors: Luton Zou, Ziping Xu, Daiqi Gao, Susan Murphy Abstract It is well known that in reinforcement learning (RL) different reward functions may lead to the same optimal policy, while some reward functions can be substantially easier to learn. In this paper, we propose a framework for reward design by constructing surrogate rewards with mediators informed by causal directed acyclic graphs (DAGs), which are often available in real-world applications through domain knowledge. We show that under the surrogacy assumption, the proposed reward is unbiased and has lower variance than the primary reward. Specifically, we use an online reward design agent that adaptively learns the target surrogate reward in an unknown environment. Feeding the surrogate rewards to standard online learning oracles, we show that the regret bound can be improved. Our framework provides a theoretical improvement…

Authors: Asim H. Gazi, Daiqi Gao, Susobhan Ghosh, Ziping Xu, Anna Trella, Predrag Klasnja, Susan A. Murphy Abstract Just-in-time adaptive interventions (JITAIs) are nascent precision medicine systems that extend personalized healthcare support to everyday life. A challenge in designing JITAIs is that personalized support often involves sophisticated decision-making algorithms. These decision-making algorithms can require numerous non-trivial design decisions that must be made between successive JITAI deployments (e.g., hyperparameter selection for an artificial intelligence algorithm). Making design decisions between deployments–rather than during deployment–ensures intervention fidelity and enhances the ability to replicate results. Yet, each deployment can be costly, precluding the use of A/B testing for every design decision. How should design decisions be made strategically between JITAI deployments? This paper introduces digital twins for just-in-time adaptive interventions (JITAI-Twins) to address this question. JITAI-Twins are “digital twins…