Strengthening high-entropy alloys (HEAs) by precipitates or second phases is a very appealing direction. However, knowledge regarding the relationship between composition and intermetallic phase type in HEAs is extremely rare. In this paper, ten systematically alloyed CoCrFeNiX alloys, with X being mainly refractory elements, were used as a model system to study how alloying affects the type of major intermetallic phase. It was found that atomic size difference is a dominating factor in the selection of intermetallic phase type. The CoCrFeNiX alloys form a particular major intermetallic structure only when the atomic size ratio (ASR) of the alloy corresponds to the allowable ASR of that structure. This behavior is also observed in other HEAs with similar compositions.