Career success and weak paradigms: the role of activity, resiliency, and true scores

Abstract Questions have been raised about the significant influence that weak paradigm development has on the careers of individuals struggling to make a life in the field of organization science. Based on the fact that a weak paradigm leads to low levels of agreement about the quality of any manuscript, it has been suggested that the marketplace for ideas is beyond comprehension to even the most talented individuals, and that career outcomes are determined largely by chance events over which people have no control. We re‐examine this argument and the influence of weak paradigms on the reliability of evaluations at the career level of analysis. Using Spearman–Brown Prophecy formula, we show that the reliability of evaluations at the career level is very high, even if one makes very pessimistic assumptions regarding inter‐rater agreement at the manuscript/journal level. Although weak paradigms create high levels of uncertainty on any one single pass through the review system, our results suggest that within 5 years, researchers are likely to be reliability evaluated. This reliability converges on 1.0 under specific conditions that are easily controlled by authors (activity level and resiliency) and editors (forming independent judgments and the use of three reviewers rather than two). Copyright © 2007 John Wiley & Sons, Ltd.