Distinguishing efficacy, individual effectiveness and population effectiveness of therapies

We read with great interest the article by Phillips et al. [1] and the editorial by Sabin [2] regarding the effectiveness of therapies in observational studies. Although both articles laudably detail issues regarding the use of observational studies for assessing the effect of treatments, the articles highlight the need to distinguish between three epidemiological concepts: efficacy, individual effectiveness and population effectiveness of therapies (Table 1). Clinical trials measure the efficacy of treatments, in which the responses of treated individuals are compared with untreated individuals, and randomization is expected to remove all confounding factors. Clinical trials demonstrate what works under controlled conditions with the limitation that the results are not necessarily applicable to real-world conditions; effects may differ in populations not represented in such trials or among those who do not precisely adhere to the prescribed regimens.Table 1: Efficacy, individual effectiveness, and population effectiveness in epidemiological studies. In observational studies, two types of effects can be measured. The first can be termed `individual effectiveness', which mimics clinical trials by using treatment data at the individual level. These analyses must, via stratification and regression, overcome the lack of randomization and confounding by indication, whereby those individuals at more advanced disease stages are the ones more likely to receive the therapies [3]. Measures of individual effectiveness supplement (and typically confirm) the results of clinical trials, but are usually subject to residual confounding [1]. The second can be termed `population effectiveness', which compares the occurrence of disease in the population when the most ill are treated to the occurrence of disease in the population when none or only a few are treated with a given therapy. As the introduction and use of therapies are closely linked to the calendar, the primary comparison may be characterized by time periods. To control for survival bias and overall disease progression, this approach requires the comparison of groups reaching a similar time at risk (e.g. duration of infection) in different eras defined by calendar periods [4–6]; or the comparison of groups of individuals with similar markers of disease progression at the beginning of treatment eras [7]. Measures of population effectiveness complement the efficacy measured in clinical trials and provide a key public health index: the amount by which disease burden is reduced when only some (typically the most ill) have been exposed to the therapy of interest. Comprehensive data collected by cohort studies are essential to eliminate possible ecological fallaciousness (i.e. effectiveness caused by changes over time different from the therapies of interest) [2,5]. This includes not only prospectively collected data on therapy use, but also information on access, healthcare utilization and practices, and adherence. These data are important in characterizing calendar periods with heterogeneous disease incidences and trajectories of markers of disease progression. Phillips and colleagues [1] mimicked three clinical trials in each of three major European cohorts to estimate individual effectiveness. In addition to examining the extent by which the cohort studies confirmed the efficacy shown by clinical trials, the data could also have been analysed to provide measures of population effectiveness. Characterization of the introduction of different therapies at different calendar times into the cohorts and the corresponding changes of disease incidence is of central public health interest. The agreement between inferences of individual effectiveness from observational studies and efficacy from clinical trials shown by Phillips et al. [1] is, in general, comforting. However, the largest (by roughly a factor of 10) cohort of the three analysed studies was the French cohort that provided findings that were inconsistent with the results of the clinical trials. Despite the cautionary note of a `trade-off between quantity and quality of data', one would have expected the largest cohort to have the maximum ability of controlling (adjusting away) the confounding by indication. This is particularly relevant to the individual effectiveness of the highly efficacious therapies containing a protease inhibitor. This inability to confirm the results of the trials illustrates the difficulty of removing strong confounding by indication unless we entertain the unlikely possibility that potent antiretroviral therapy was not effective in the French cohort. It is possible that the French cohort resembles more of a surveillance registry [2] than a cohort study with active and close follow-up of subjects. Such cohort studies include observing participants at regular study visits to collect data and specimens for laboratory tests under standardized procedures, and the continuous monitoring and confirmation of outcomes. Often the lack of data on important confounders seriously imperils the ability of surveillance databases and registries to assess the effectiveness of interventions [2,8]. Alternatively, the largest cohort may appear to be discrepant from the clinical trials because the analytical approach used by the authors was rather simple compared with the complexities of the selection factors involved in who received treatment. Instead of the standard extension of including updates of the markers of disease progression (i.e. time-dependent covariates) in proportional hazards models, one needs to use more complex models incorporating the mediating role of markers in capturing the effect of therapies. In addition, there are other factors besides markers that influence the use of therapy (e.g. injecting drug users may be less likely to be prescribed complex regimens, and individuals with histories of drug interactions or toxicity may also be prescribed differently). It will be of great interest to know whether a more elaborate analysis renders better agreement between cohort studies and clinical trials regarding the individual effectiveness of therapies. Although observational studies can provide important data on the effectiveness of therapies at the individual level, they face great challenges in fully mimicking what randomization accomplishes in a clinical trial setting. On the other hand, cohort studies offer the opportunity of providing important measures of the effectiveness of therapies at the population level that clinical trials cannot provide. From the public health perspective, population effectiveness quantifies the reduction of disease achieved by treating the subset of the population that needs therapies the most. In doing so, cohort studies are at the cornerstone of public health and policy. Alvaro Muñoz Stephen J. Gange Lisa P. Jacobson