Generalizability of the Individual Placement and Support (IPS) model of supported employment outside the US

Individual Placement and Support (IPS) is a systematic approach to helping people with severe mental illness achieve competitive employment 1. It is based on eight principles: eligibility based on client choice, focus on competitive employment, integration of mental health and employment services, attention to client preferences, work incentives planning, rapid job search, systematic job development, and individualized job supports 2. Systematic reviews have concluded that IPS is an evidence-based practice 3,4,5,6,7,8,9,10,11,12. With the development of a strong evidence base for IPS in the US, mental health leaders in other countries have interest in the transportability of IPS to their countries. Generalizability of other evidence-based practices developed in the US has been variable and in some cases adoption has been curtailed after failures to replicate US findings 13. The current review has two goals. First, given the growing international attention to IPS, we examined its effectiveness in studies conducted outside the US compared to US studies. Second, we expanded the scope of prior IPS reviews by adding recent randomized controlled trials (RCTs) and enlarging the range of outcome measures in order to examine the hypothesis that IPS yields better competitive employment outcomes across a range of measures than alternative vocational programs. The study inclusion criteria were as follows: RCT; comparison of IPS to a control condition not providing IPS; target population of clients with severe mental illness; longitudinal competitive employment outcomes; intervention monitored with the IPS Fidelity Scale 14. We used a combination of search procedures including formal electronic searches, bibliographic searches of prior reviews and conference proceedings, and inquiries to vocational researchers, especially those in other countries. We cross-checked our findings with an exhaustive search conducted by Cochrane reviewers 9. Consistent with the goal of IPS, the review's main focus was competitive employment, defined as permanent jobs paying commensurate wages in integrated community settings (i.e., employing nondisabled workers) and available to anyone (not just individuals with disabilities). Consistent with the goals of social inclusion, this definition excludes noncompetitive jobs, such as transitional and sheltered employment 15. IPS researchers have not adopted a standardized measurement framework, although some indicators are common across studies. Competitive employment indicators include measures of job acquisition (e.g., percentage of clients obtaining competitive employment and time from study entry to first job start), duration (e.g., cumulative number of weeks worked in all jobs), intensity (e.g., percentage working at least 20 hours a week), and productivity (e.g., total hours worked/wages) 16. Some vocational models place clients in noncompetitive jobs (e.g., sheltered employment, agency-run business, etc.). When reported, we summarize these noncompetitive employment outcomes. We also examined dropout rates from IPS and control programs. Finally, many studies also examined a range of outcomes outside the employment domain; we summarize these findings. Data were recorded directly from published reports or calculated from information presented in the published studies. For the measure of job duration, we converted total weeks worked to an annualized rate, reporting the findings for both the full intent-to-treat sample and the worker subsample (those who obtained at least one competitive job during follow-up). Given the small number of studies, our comparisons between US and non-US studies relied on visual inspection. The one exception was competitive employment rate, where we combined samples within US and within non-US studies and used a 2×2 c2 to compare overall rates. For each study we calculated the effect size for the difference in competitive employment rate between IPS and controls using the arc sine approximation 17. An unweighted overall effect size was calculated as the simple mean of the individual effect sizes. For hours of employment, we first converted data for each study to an annualized rate to accommodate the different follow-up periods. We next calculated the d effect size for the difference in means between IPS and controls 17. Finally, we calculated the unweighted overall effect size. For all other outcome measures, means are reported without standard deviations, because this statistic was usually unavailable from the original studies. Overall means were calculated weighting individual means by sample sizes. We excluded 9 RCTs that evaluated a form of supported employment that either preceded the development of IPS 18,19,20,21,22 or reflected a vocational approach that was not IPS 23,24,25,26. We identified 15 studies, 9 from the US and 6 outside the US, as shown in Table 1. Altogether, these studies enrolled 1063 IPS participants (mean = 70.9 per study) and 1117 control participants (mean = 74.5 per study). The mean length of follow-up was 18.4 months. Except for one three-group design 31, all studies used a two-group design (IPS vs. control). Ten studies were conducted at a single site, while five studies 7,27,29,32,33 had multiple sites. Two studies used nonintegrated supported employment control groups 31,33. Otherwise, all the control groups consisted of either treatment as usual or well-established alternative vocational models. All studies reported using standard methods to assess and monitor IPS fidelity. In most studies, participants were recruited from clients receiving services from community mental health centers. In all the studies, participants were unemployed at the time of study admission. In all but one study 34, the study inclusion criteria included an expressed desire to work. Another eligibility criterion common across most studies was the absence of significant medical conditions, such as end-stage cancer, that would preclude working during the follow-up period or participating in assessment interviews. The Los Angeles study 35 required a two-to-three-month stabilization period before study entry because participants were often in a psychotic state at referral. In most studies, participants were required to attend two or more research information meetings explaining the study purpose 36. The competitive employment rate was significantly higher for the IPS condition than for controls in every one of the studies, as shown in Figure 1. In total, 592 (55.7%) of IPS participants obtained employment, compared with 253 (22.6%) control participants. Averaging the rates across studies, the competitive employment rate was 58.9% (median = 63.6%) for IPS compared to 23.2% (median = 26.0%) for controls. The mean difference in percentage employed between IPS and controls was 35.7%, ranging from 11.0% to 55.5%. The individual study effect sizes ranged from .30 to 1.18. The overall unweighted effect size was .77. Competitive employment rates in 15 randomized controlled trials of Individual Placement and Support (IPS) We next compared the competitive employment rates between the 9 US and 6 non-US studies. Combining samples across studies, 374 (62.1%) of 602 IPS clients from the 9 US studies obtained competitive employment, compared with 218 (47.3%) of 461 IPS clients from the 6 non-US studies, χ² 1 = 23.29, p<0.001. The comparison for the combined control samples was not significant: 150 (23.5%) of 645 control clients from the US studies obtained competitive employment, compared with 103 (21.8%) of 472 control clients from the non-US studies, χ² 1 = 0.32. For the US studies, the unweighted mean competitive employment rate was 65% for IPS and 25% for controls, with an overall unweighted effect size of .84. For the non-US studies, the mean competitive employment rate was 50% for IPS and 20% for controls, with an overall unweighted effect size of .67. Among US studies, the competitive employment rate of 27% for the Maryland IPS sample 34 was an outlier – less than half the rate for IPS in the other 8 US studies and equal to the mean control group rate. With this study removed, the unweighted mean competitive employment rate for US studies increased to 69% for IPS and 28% for controls. Similarly, among non-US studies, the competitive employment rate of 22% for the UK IPS sample 29 was an outlier — less than half the rate for IPS in the other 5 non-US studies and equal to the mean control group rate. With this study removed, the unweighted mean competitive employment rate for non-US studies was 56% for IPS and 22% for controls. Four IPS studies 31,32,33,37 reported the proportion of participants who worked 20 hours or more per week. Aggregating across these studies, 134 (43.6%) of 307 IPS participants and 53 (14.2%) of 374 controls held such jobs, yielding an effect size of .67. One study reporting rates of full-time competitive employment found no difference (8.7% of IPS participants vs. 11.6% of controls) 32. Number of days to first competitive job was reported in 9 IPS studies (6 US, 3 non-US), as shown in Table 2. The UK study was an extreme outlier 29, with mean of 680 days to first job. Excluding this outlier, the average time to first competitive job was 50% faster for IPS participants compared to controls (136 days versus 205 days). The other two non-US studies (from Hong Kong and Canada) had the shortest mean time to first job of the 9 studies. The findings for mean hours worked per year in competitive employment for 5 US and 2 non-US studies are shown in Table 3. The variability across studies was substantial, from a mean of 656 hours for the Alabama 28 study to 126 hours for the Québec 38 study. Nonetheless, the overall unweighted effect size was large (d = .58), and the ratio of IPS to controls in hours worked was threefold overall. No obvious pattern was apparent for the comparison of US to non-US studies. The findings for annualized weeks worked in competitive employment are reported for six US and two non-US studies in Table 3. Overall, the mean weeks worked per year in competitive employment for IPS was more than twice the mean weeks for controls. When the comparisons were limited to participants who obtained competitive employment during follow-up, the weeks worked were virtually the same for IPS and controls. Total paid employment outcomes, including noncompetitive jobs, were reported in seven studies 31,32,33,37,38,39,40. In six of these studies, the rate of noncompetitive employment for IPS was modest (11% or less of IPS participants), though in the Québec 38 study, 20% of IPS participants obtained a noncompetitive job. In three US studies 31,33,39 and one non-US study 40, inclusion of all paid employment did not materially affect the employment findings. Considering all paid employment outcomes, one US study 32 and one non-US study 38 showed no differences between IPS and controls in employment rates and on several other employment measures, while another US study reported no differences in overall earnings between IPS and controls 37. Early program dropouts refer to clients who either discontinue vocational services within an early time period or never make an initial contact. Studies reporting dropout (or attrition) rates did not have a standardized time period or common method of assessing discontinuation. For example, the Washington study 37 reported attrition rates after 2 months, the Illinois study 32 identified early program dropouts as clients who discontinue services within the first 6 months, and the Québec study 38 defined attrition as failure to have at least one contact with vocational staff in each of the first and the second three-month follow-up periods. Averaging across six studies 7,31,32,33,37,38, 9% of IPS participants were early program dropouts, compared to 42% of controls. Nine of the studies included in the review also examined nonvocational outcomes, which most often included psychiatric symptoms, quality of life, and psychiatric hospitalizations 31,32,33,34,37,38,39,41,42. Some also included measures of self-esteem, social functioning, and social network. With rare exception, IPS participants did not differ from controls on any of these measures. Rigorous evaluations of IPS suggest that 60% or more of IPS clients obtain competitive jobs, compared to about 25% of those who receive other types of vocational assistance. One way of interpreting this finding is that approximately 25% of clients who express an interest in competitive employment will succeed in obtaining a job in diverse and ineffective vocational programs or even without any vocational services, but IPS helps an additional 35% of the target group who otherwise would remain unemployed. The finding of a large and statistically significant beneficial impact of IPS is robust, upheld in all 15 studies. The effectiveness of IPS is also suggested by other measures of competitive employment outcome, including time to first job, job duration and total hours employed during the follow-up period. Most IPS clients work part-time, typically half-time; about two-thirds of those who obtain competitive employment work 20 hours or more per week. Few IPS clients work full-time, likely due to preferences, limited stamina, and/or fear of losing health insurance or other benefits. Consistent with the principle of rapid job search, the time to first competitive job for IPS participants is nearly 10 weeks sooner than for controls. The mean length of time to first job for IPS participants (19 weeks) is, however, still lengthy for a model that prescribes rapid job search. This review advances over earlier reviews in several respects. First, it has the largest and most up-to-date collection of pertinent randomized controlled trials. Second, it expands the scope of outcomes examined. Third, it is limited to rigorous evaluations of IPS programs, giving the clearest picture of the potential for IPS. Fourth, it is the first review to systematically compare US to non-US studies. Some comment is warranted about the inclusion of the two studies clearly identified as outliers 29,34. The Maryland 34 study clearly deviated from the other IPS studies in that it was the only study among those reviewed that did not require participants to have a goal of competitive employment. Many participants apparently joined the study to receive the research payments and not because of their interest in employment. This study's poor competitive employment outcomes are consistent with its lenient admission criteria. Regarding the UK study 29, we concur with two commentators 43,44 who noted this study's shortcomings in adhering to the IPS model, according to descriptions provided by the investigators 45. Conversely, an outlier on the upper end was the IPS study of veterans with post-traumatic stress disorder (PTSD) 28. This study had outstanding outcomes on most employment indicators, suggesting that this target population may be especially amenable to IPS interventions, though replication is needed. While PTSD is not usually classified as a severe mental illness, some PTSD researchers have argued that it should be, given its long-term nature and the disability it often engenders 46. Systematic research is needed to determine which diagnoses and disabilities IPS is suited for. An unresolved question is whether noncompetitive employment outcomes are equivalent to competitive jobs with respect to their utility for clients, program managers, funders of rehabilitation services, and society at large. The IPS model is based on the argument that competitive jobs are greatly preferred over noncompetitive ones by clients themselves 47. In addition, a sustained period of competitive employment has been associated with better nonvocational outcomes in some studies 41,48, whereas this has not been shown as clearly for noncompetitive jobs. We assume that the advantages of competitive jobs are best evaluated in long-term studies 49. Nonetheless, several studies in this review found that control interventions were equally effective as IPS in achieving a range of paid employment outcomes when noncompetitive jobs were included. Finally, the costs associated with developing and maintaining noncompetitive job programs should also be considered; anecdotal evidence suggests the costs are often enormous 15. Moreover, the societal burden of developing and maintaining noncompetitive jobs is unsustainable on a large scale, in that costs are usually entirely borne by governmental subsidies rather than by the private sector and clients typically do not pay taxes on noncompetitive jobs. The low dropout rates reported in IPS studies are thought provoking. First, they are in contrast to an early review noting high dropout rates among supported employment clients 3. Consistent with the assertive outreach component of the model, IPS programs have exceptionally low dropout rates, less than 10% in most studies. Conversely, studies often report high dropout rates for control participants. The contrast in termination rates for IPS and control groups raises a different question, whether the superior employment outcomes for IPS can be attributed to attrition. In other words, would the intent-to-treat findings for IPS reported above hold up for treatment exposure analysis? That is, what if the analyses were repeated with program dropouts removed? One study that has conducted this analysis found that IPS exceeded controls in comparisons that excluded dropouts 32. However, treatment exposure analyses were not reported in the other studies. Of course, it could be argued that control participants who dropped out did so because they viewed the control intervention as ineffective. Clearly, this question warrants further study. Enrollment in IPS per se does not improve nonvocational outcomes beyond services as usual. Improved nonvocational outcomes may only accrue for clients who work steadily over time in a competitive job 48. These relationships need further exploration within longitudinal studies. Worldwide interest in the IPS model is suggested by the increased proportion of IPS studies conducted outside the US reported since 2007. One new finding to emerge from the current review was that competitive employment rates are stronger for the US studies than for non-US studies. In particular, the European and Canadian studies had poorer outcomes than the US studies, while the outcomes from the Hong Kong and Australian studies were comparable to those in the US. Understanding the reasons will be important for policy planners and service providers as IPS continues to be disseminated internationally 50. Diminished effectiveness for IPS, particularly in Europe, has been typically attributed to labor and disability policies that can impede returns to work, for example, what Burns et al 7 refer to as the "disability trap". A Swedish study of IPS currently in progress describes in detail the bureaucratic inertia and attitudinal barriers within the Swedish welfare system impeding the development of effective IPS services 51. A Dutch study has also described the challenges in implementing IPS 52. Qualitative studies suggest that these barriers are formidable and to some extent represent challenges not found in the US. IPS leaders in several other countries have pursued strategies to overcome these barriers 53,54. Further international studies are needed to examine the nature and strength of these policy factors and to determine what adaptations are needed. At present, too few international RCTs have been conducted to draw strong conclusions about the influence of policy and of economic, cultural, and societal factors. An alternative explanation for the poorer employment outcomes in several non-US IPS studies is the lack of adequate technical assistance and training for staff, leading to substandard implementation. Without adequate fidelity, the effectiveness of a program is attenuated and the quality of the resulting evaluation is greatly compromised. We note that all of the US studies were either conducted by, or received consultation from, the developers of the IPS, whereas only one-third of the non-US studies 7,38 received direct input from the model developers. Geographic distance is likely a factor for this difference. The quality of implementation of the non-US studies is generally difficult to evaluate because of the lack of process details contained in their published reports. Two non-US multisite studies reported substandard fidelity in a minority of sites 7,27. How do we explain the high ratings for IPS fidelity reported in the UK 29 study? We notice that the ratings were not made by independent assessors familiar with IPS, and a wealth of research has shown that self-ratings by project staff are often inflated 55. Given the strong association between IPS fidelity and competitive employment outcome 14, we propose that future reviews be restricted to evaluations of high-fidelity IPS programs, as verified by independent fidelity reviewers trained in conducting these assessments. The broader issue for the advancement of an evidence-based practice, both for practical reasons and for scientific rigor, is the criticality of adequately trained staff and access to appropriate technical assistance. While the field of implementation science is still in its infancy 56, some general findings are beginning to emerge. Widescale dissemination of IPS has been facilitated by expert technical assistance in the US 57. When IPS technical assistance has been absent, dissemination results have often been dismal 58,59,60,61. The critical need for training and quality assurance in implementation of a program model has led the developers of other evidence-based models to insist that users agree to systematic training and technical assistance regimens to assure quality of implementation 62. As a guide for determining when to intervene, Becker et al 63 have suggested that programs with quarterly competitive employment rate under 33% should be considered still in startup phase or as failing programs in need of immediate technical assistance. Of course, undue influence of model developers on evaluations of their own model has been criticized as introducing the bias of therapeutic allegiance 64. This suggests the continuing need for the training of a second generation of IPS experts to conduct studies independent of the model developers, work that has already begun and has been represented in the current set of studies. To summarize, the question of IPS transportability outside the US remains unanswered. While the published studies suggest that the labor and disability laws in some European countries may make a direct replication of IPS difficult, there are also indications that IPS transports well to other countries, such as Australia and the Hong Kong region of China. Finally, before concluding that the IPS must undergo radical adaptations in another nation, IPS programs should receive sufficient training and guidance to implement the model with high fidelity.