Inconsistent consistency: evaluating the well-defined intervention assumption in applied epidemiological research

According to textbook guidance, satisfying the well-defined intervention assumption is key for estimating causal effects. However, no studies have systematically evaluated how the assumption is addressed in research. Thus, we reviewed how researchers using g-methods or targeted maximum likelihood estimation (TMLE) interpreted and addressed the well-defined intervention assumption in epidemiological studies. We reviewed observational epidemiological studies that used g-methods or TMLE, were published from 2000-21 in epidemiology journals with the six highest 2020 impact factors and met additional criteria. Among other factors, reviewers assessed if authors of included studies aimed to estimate the effects of hypothetical interventions. Then, among such studies, reviewers assessed whether authors discussed key causal-inference assumptions (e.g. consistency or treatment variation irrelevance), how they interpreted their findings and if they specified well-defined interventions. Just 20% (29/146) of studies aimed to estimate the effects of hypothetical interventions. Of such intervention-effect studies, almost none (1/29) stated 'how' the exposure would be intervened upon; among those that did not state a 'how', the 'how' mattered for consistency (i.e., for treatment variation irrelevance) in 64% of studies (18/28). Moreover, whereas 79% (23/29) of intervention-effect studies mentioned consistency, just 45% (13/29) interpreted findings as corresponding to the effects of hypothetical interventions. Finally, reviewers determined that just 38% (11/29) of intervention-effect studies had well-defined interventions. We found substantial deviations between guidelines regarding meeting the well-defined intervention assumption and researchers' application of the guidelines, with authors of intervention-effect studies rarely critically examining the assumption's validity, let alone specifying well-defined interventions.