With the increasing threat of Unmanned Aerial Vehicle (UAV) intrusions, enhancing the perceptual capabilities of counter-UAV systems has become essential. Although various advanced algorithms have been developed in the domain of object tracking, resources specifically dedicated to UAV target tracking remain scarce. To bridge this gap, we propose a novel tracking algorithm named the Spectrum-Adaptive Transformer with Spatial Awareness for UAV Target Tracking (SAT). This network incorporates a spatial location awareness module, which captures long-range dependencies while retaining precise positional information, thereby enriching target feature representation in complex environments. Additionally, we introduce a spectrum-adaptive attention module, allowing the network to adapt to UAV targets with dynamic scale variations. Experimental results demonstrate that our algorithm achieves a 9.4% increase in success rate and a 12.1% improvement in precision compared to baseline methods on the UAV target datasets, demonstrating superior performance in handling scale variations and complex tracking scenarios. This research provides a significant advancement in UAV tracking, offering a robust and efficient solution for counter-UAV systems, and has the potential to enhance real-world applications in security and surveillance.