A dense desmoplastic stroma formed by abundant extracellular matrix and stromal cells, including cancer-associated fibroblasts (CAFs) and immune cells, is a feature of pancreatic ductal adenocarcinoma (PDAC), one of the most lethal cancer types. As the dominant cellular component of the PDAC stroma, CAFs orchestrate intensive and biologically diverse crosstalk with pancreatic cancer cells and immune cells and contribute to a unique PDAC tumor microenvironment promoting cancer proliferation, metastasis, and resistance against both chemo- and immunotherapies. Therefore, CAFs and CAF-related mechanisms have emerged as promising targets for PDAC therapy. However, several clinical setbacks and accumulating knowledge of the PDAC stroma have revealed the heterogeneity and multifaceted biological roles of CAFs, and concerns regarding “what to deliver” and “how to deliver” have arisen when designing CAF-targeted drug delivery systems to specifically inhibit tumor-supporting CAFs without impairing tumor-restricting CAFs. In this review, we will discuss the complexity of CAFs in the PDAC stroma as well as the potential opportunities and common misconceptions regarding drug delivery efforts targeting PDAC CAFs.