Agricultural drought is a complex and insidious natural hazard further complicated by crop impacts. Univariate, bivariate, and multivariate drought analyses have achieved some success, but the analysis of agricultural drought evolution and integration with crop growth is still lacking. In this study, an Evolution Process-based Multi-sensor Collaboration (EPMC) framework was proposed with the realization that effective agricultural drought assessment requires an integrated approach that considers both drought development and crop phenology. Then the Process-based Accumulated Drought Index (PADI) was designed to quantify the accumulative drought impacts on crops. Based on the monitoring of precipitation, soil moisture, and vegetation conditions, EPMC extracted four main agricultural drought evolution phases termed: (i) latency, (ii) onset, (iii) development, and (iv) recovery. Subsequently, the crop growth stages and water-deficit sensitivity coefficients were integrated with the drought evolution process. Experiments conducted in three different climate regions of China demonstrated that the EPMC framework could clearly depict evolution of the different phases of agricultural drought. Three decades of multi-sensor datasets include monthly precipitation from the Global Precipitation Climatology Centre (GPCC), root zone soil moisture from the satellite-model integrated Global Land Data Assimilation System version 2 (GLDAS-2.0), and vegetation condition data from the Advanced Very High Resolution Radiometer (AVHRR). Results indicated that PADI reliably provided a weekly evaluation of accumulative drought severity instead of a “snapshot”. PADI was also compared with the Palmer Drought Severity Index (PDSI) and multi-time scale of Standardized Precipitation Index (SPI). Results showed good correlation with short-term SPI at the onset of drought as well as long-term SPI at later stages. Additionally, compared to the correlation with precipitation, soil moisture, and vegetation data alone, it was found that as an integrated model, PADI correlated well with wheat yield loss (Spearman rank correlation coefficient ρ was between 0.66 and 0.77, p < 0.05). Therefore, the proposed multi-sensor integrated monitoring framework and index provide a useful and new approach to address the complexity of agricultural drought, with particular relevance to drought impact assessment.