Background: Tumor microenvironment (TME) significantly affects colorectal cancer (CRC) progression and therapeutic efficacy, particularly the infiltrating stromal components. This study profiled the TME composition of tumor tissue and identify TME-related, especially stroma-related genes having prognosis value in CRC patients. Materials and Methods: We used the ESTIMATE algorithm to assess stromal/immune component and divided 524 CRC cases of public dataset into high- and low-score groups. We analyzed the effect of the score on prognosis and extracted the differential expression genes (DEGs) between groups, which were stromal- and/or immune-related genes, and performed a prognostic investigation of the DEGs. Results: Higher stromal score correlated with poor survival, whereas the immune score was the inverse. By comparing global gene expression of cases with high vs. low stromal/immune scores, we extracted 474 stroma-related genes, 76 immune-related genes, and 498 intersection genes, which were explored by function enrichment and survival analysis. We identified the expression of five stroma-related genes (including ITGA7, PTPN14, SCG2, TNS1, and GRP) significantly associated with poorer survival, which were validated in the other two independent CRC cohorts. Conclusion: These results presented a comprehensive understanding of TME components and identified five stroma-related genes that predict poor outcomes in CRC patients.