Heightened interest in comparative effectiveness research has made head-to-head comparisons between anti-cancer drugs increasingly common. One strategy is to show non-inferiority of a therapy to another in terms of efficacy and to rely on improvements in safety (toxicity), quality of life, convenience, or cost to guide the choice. Many non-inferiority trials have followed this model, which raises questions regarding the use of non-inferiority designs in comparative effectiveness research. We address three issues that affect non-inferiority trials: the arbitrary nature of what constitutes a non-inferior therapy; the problem of censorship and its effect on outcomes; and the arbitrary choice of what represents an advantage if non-inferior efficacy is established.