Ultra-wideband (UWB) technology has been widely applied in the high-precision indoor positioning system. However, the complicated indoor environment makes signals propagate in non-line-of-sight (NLoS) situations, which seriously deteriorates the positioning accuracy. This work proposes an outlier-aware method to improve the positioning accuracy under NLoS scenarios. End-to-end optimization and positioning are achieved by combining the measurement error mitigation process with the positioning process. Experiments on public benchmarks illustrate that the proposed method enhances the performance of indoor positioning in NLoS situations.