Pancreatic duct dilation indicates a high risk of pancreatic ductal adenocarcinoma (PDAC), the deadliest cancer with a poor prognosis. Segmentation of dilated pancreatic duct from CT taken from patients without PDAC shows the potential to assist the early detection of PDAC. Most current researches include pancreatic duct segmentation as one additional class for patients who have already detected PDAC. However, the dilated pancreatic duct for people who have not yet developed PDAC is typically much smaller, making the segmentation difficult. Deep learning-based segmentation on tiny components is challenging because of the large imbalance between the target object and irrelevant regions. In this work, we explore an attention-guided approach for dilated pancreatic duct segmentation as a screening tool for pre-PDAC patients, enhancing the pancreas regions' concentration and ignoring the unnecessary features. We employ a multi-scale aggregation to combine the information at different scales to improve the segmentation performance further. Our proposed multi-scale pancreatic attention-guided approach achieved a Dice score of 54.16% on dilated pancreatic duct dataset, which shows a significant improvement over prior techniques.