Drug‐induced pancreatitis: lessons in data mining

Pharmacovigilance is principally concerned with the timely discovery of adverse events that are novel in terms of clinical nature, severity, and/or frequency. With the ever-increasing volume of postmarketing safety surveillance data there is considerable interest in using computer-assisted signal detection algorithms [also known as data mining algorithms (DMAs)] to search extremely large spontaneous reporting system (SRS) databases for statistical dependencies between drugs and events in excess of what would be expected if the drug and event were independently distributed in the database [1-5]. If there is sufficient correlation between the statistical dependencies identified by DMAs and demonstrable causal relationships they could improve pharmacovigilance performance. Data mining algorithms include methods based on simple disproportionality analysis [e.g. proportional reporting ratios (PRRs) [1] and reporting odds ratios (RORs) [2]], and those incorporating Bayesian modelling and other statistical adjustments to reduce the volume of 'signals' generated [e.g. Bayesian Confidence Propagation Neural Network (BCPNN) [3] and the multiitem gamma-Poisson shrinker (MGPS) [4]]. A very high volume of signals, including false-positive signals due to confounding, can strain pharmacovigilance resources with too many unfruitful hypotheses. However, components of the statistical modelling and adjustments that reduce the volume of signals may also be associated with deleterious consequences by concurrently degrading the capacity of the DMA to detect true-positive signals of real causal associations. Drug-induced pancreatitis is of considerable concern in pharmacovigilance because of the potential for significant morbidity and mortality and the numerous drugs that may be aetiological agents [6-8]. The ability of DMAs to provide meaningful signals of drug-induced pancreatitis is therefore of interest and can be used to illustrate several important principles related to the practical evaluation and deployment of DMAs. Many drugs have been associated with pancreatitis. Published reports and observational studies have asserted definite causal relationships with various drugs including azathioprine, cimetidine, didanosine, erythromycin, furosemide, hydrochlorothiazide, interferon-alpha, mesalazine, methyldopa, metronidazole, olsalazine, oxyphenbutazone, simvastatin, sulfasalazine, tetracycline, and valproate [6-8]. Although residual uncertainty about causality may still remain even in well-documented case reports involving positive rechallenge [8], it provides an interesting and diverse data set for examining the performance of DMAs since the evidentiary support of the reported associations may be quite strong. Applying two DMAs (PRRs and MGPS) using commonly cited threshold criteria and Medical Dictionary for Regulatory Affairs (MedDRA) Preferred Terms specific for pancreatitis, data from the United States Food and Drug Administration were analysed for possible signals of drug-induced pancreatitis with the 16 aforementioned drugs. For comparison, the time to appearance of replicated findings (i.e. ≥ 2 case reports) of drug-induced pancreatitis in the published literature (another source of signals) was estimated by manual review of citations generated through a search of MEDLINE using the respective drug name and 'pancreatitis' as key words. Proportional reporting ratios appeared to outperform MGPS on several counts in this analysis. PRRs highlighted 15 out of the 16 associations compared with MGPS, which highlighted 11 out of 16. Of the associations highlighted by both methods, PRRs provided a signal of disproportionate reporting from 1 to 14 years prior to MGPS. The number of reports required to generate a signal with PRRs ranged from one to 19 compared with four to 62 for MGPS in the applicable cases. The majority of signals highlighted by PRRs (9/15) were based on three or fewer reports while the majority (6/11) highlighted by MGPS required over 20 reports. Proportional reporting ratios highlighted eight out of the 16 associations (50%) from 1 to 16 years in advance of the published literature compared with three out of 16 (19%) highlighted by MGPS 3–7 years prior to the published literature. The results of this analysis illustrate the potential of both methods to improve the process of signal detection and the potential of simple forms of disproportionality analysis to identify potentially meaningful associations that fail to be identified by certain Bayesian methods such as MGPS or to identify signals highlighted by both at significantly earlier time points. The cost of such enhanced sensitivity is an over abundance of signals that may require additional triage criteria for practical implementation. Performance gradients between simple disproportionality analysis (higher 'sensitivity', lower 'specificity') and certain methods incorporating Bayesian inference and extensive statistical adjustments (lower 'sensitivity', higher 'specificity') when used in isolation is important information to bear in mind. However, it does not truly reflect 'real life' pharmacovigilance practice, since these methods, if used, should only be used as supplements to, not substitutes for, standard signalling strategies. The observed performance differentials are likely to be significantly mitigated when these methods are used as one element of a comprehensive pharmacovigilance programme. Further complicating any comparisons is the lack of specified standardized data mining procedures (e.g. selection and combination of adverse event terms, lack of reliable and consistent criteria for adjudicating causality and expectedness of adverse events, variations in database and dictionary architectures, multiple biases, confounding factors, and data quality limitations inherent in voluntary reporting systems). The crucial question therefore, is the proper positioning of such techniques within the universe of methods that have been historically used for routine signal detection based on various frequency criteria and clinical pharmacological judgement. Most likely the ideal point on the aforementioned performance gradient for decision-making will be highly situation dependent. Specifically, in assessing the potential added value of DMAs and/or selecting a specific algorithm, potential users of DMAs should assess numerous factors, including the rigour of their standard signalling criteria, the nature and numbers of drugs and adverse events reports to be screened, the respective public health impacts of false-positive and false-negative findings, timing in the product life cycle, and resource constraints. It is quite likely that the incremental utility of DMAs may be higher for health authorities, who have statutory obligations for monitoring the safety of all licensed drugs than for individual pharmaceutical companies, whose surveillance activities are more focused in scope. Another important principle illustrated by this analysis is that the importance of sound clinical, pharmacological and epidemiological judgement in the signal detection process is not diminished, and is in fact increased, with the use of DMAs. This is not only because of the false-positive signals frequently observed with DMAs due to various confounding factors and reporting biases. Pancreatitis is an adverse event that may be included in lists of 'designated medical events' based on rarity, medical importance, and/or a high level of drug-attributable risk in general. As few as one to three reports of such events may be considered signals under some circumstances [9]. Given the findings of this analysis, over-reliance on DMAs for detecting such events, especially those that down-weight ('shrink') signal scores based on small numbers of reports, may be especially hazardous. Further research is needed to determine if DMAs have utility in the detection of designated medical events. More generally, while PRRs identified a greater number of the associations or identified them earlier than MGPS, both failed to highlight a significant proportion of associations either absolutely or relative to the published literatune. Numerous factors may contribute to the absence of a signal of disproportionate reporting of a drug-induced medical event with DMAs including a high background frequency of the drug and event in the database, the overall reported safety profile of the pharmaceutical, and influences on reporting behaviour. This underscores the increased need for clinical judgement when using DMAs and the caveats that the absence of a signal of disproportionate reporting with any DMA is not adequate to refute a signal generated by traditional criteria and that DMAs should never be used in isolation. The converse is not true, in that a potential signal generated by a DMA can possibly be refuted by expert case-level clinical review. A clear understanding of the strengths and weaknesses of the various DMAs, both with respect to traditional signal detection strategies, and relative to each other, will help identify their appropriate application and help minimize their misapplication and misuse.