The quality of quality criteria: Replicating the development of the Consolidated Criteria for Reporting Qualitative Research (COREQ)

Using checklists to critically evaluate and report qualitative research is a common practice in the health sciences and there is currently a plethora of checklists available. One strategy for developing such checklists is to identify and amalgamate items from previous checklists into more comprehensive, consolidated ones. This paper aims to critically review the credibility of a widely endorsed checklist of this type, the Consolidated Criteria for Reporting Qualitative Research (COREQ), by replicating the procedures that led to the consolidation of its 32 items. The original development of COREQ consisted of four phases: 1. A literature search and identification of a sample (n = 22) of references, 2. Data extraction through coding the references, 3. A simplification of an intermediate list of items, and 4. Addition of two items. A replication of the four-phased development of COREQ. We used the reported account of the development of COREQ to replicate the procedures and rationale applied in the four phases. However, we were rarely able to replicate them completely because of uncertainty about the actual procedures. The problems with the replication indicated that COREQ's items were not credible because the coding processes were not trustworthy and because they decontextualized original checklist items and significantly distorted their meaning in COREQ's final list of items. We discuss the use of checklists by means of Haraway's figure of the “modest witness”, which emphasizes that checklists can de-politicize research and create an illusion of rationality and objectivity. Considering that COREQ is widely endorsed by scholarly journals, it is alarming that the checklist's particular technical and a-political approach to achieve more complete reporting of qualitative research and its particular perspective on what constitutes quality in qualitative research remains unchallenged.