Evidence for Neolithic era rice farming shows that this development occurred significantly later along the coastal zone of South China and Southeast Asia (SC-SEA) than in the lower reaches of the Yangtze River. However, the mechanisms that constrained early agricultural expansion along the coastal regions are controversial, and there has been a lack of research to correlate archaeological evidence with studies on the comparison between natural processes of deltaic deposition and coastal geomorphic evolution. This study first examines pollen data representing local land cover and agriculture in southern China, based on a pollen record covering recent decades from a lake with a small watershed area. The results, associated with the pollen database of China, show that the Holocene pollen records in the coastal plain of South China can accurately capture the signal of past vegetation cover and human activities, especially the Poaceae pollen records are of great value to evaluate the rice-farming history. Secondly, in order to explore the relationship between coastal geomorphic change and prehistoric agricultural development, we integrated 23 pollen records and established a dataset of boreholes containing information for Holocene sedimentary facies and fossil indicators in the coastal areas of SC-SEA. The results show that the coastline in our study area was characteristically mountainous during the Middle Holocene, when sea level rose to its high stand and the marine transgression extended deep into the hilly landscape, resulting in a lack of coastal plain formations suitable for rice farming. After 6−5 ka BP, the sedimentary environment began to transition from marine to terrestrial facies, but the limited amount of arable land in coastal areas constrained the development of rice cultivation. Although some archaeological sites have revealed rice remains dating to around 5−4 ka BP, the broader Neolithic record shows that fishing, hunting and gathering strategies were still the dominant component of economic systems. Deltas in the study area began to expand after 4 ka BP, providing larger coastal plains and wetland areas. After about 2.5 ka BP in particular, rapid growth of the deltaic plains was highly coupled with rapid increases in Poaceae pollen, indicating the large-scale development of rice farming agriculture. In other word, despite the introduction of rice by migrating Neolithic farmers as early as 5−4 ka, the expansion of paddy rice farming in SC-SEA was highly constrained by physiographic factors. Thus, rice agriculture remained a small component of subsistence strategies prior to about 2500 years ago, and intensive rice farming took hold rapidly following the exposure of deltaic plains. Migrations of agricultural population to the south during the Qin and Han dynasties introduced advanced technology for intensive rice farming, which further promoted the development of local agriculture. Accompanying increases in shifting cultivation caused extensive burning, deforestation and soil erosion. Besides, according to the comparison of pollen records at different altitudes, it was found that the age of the early human impact on natural vegetation and development of rice farming in the low altitude plain were earlier than those in the mountains. The disturbance of forest by human activities occurred during 3−2 ka. Whereas, in the mountainous areas higher than 1000 m, significant ecological disturbance and deforestation occurred much later, generally after the Tang or Song Dynasties. This study reveals that the geomorphic evolution of coastal plains in SC-SEA was basically synchronized with the history of agricultural development, highlighting the dynamic relationship between humans and the environment.