Domain-specific neural network accelerators have seen growing interest in recent years due to their improved energy efficiency and performance compared to CPUs and GPUs. In this paper, we propose a novel cross-layer optimized neural network accelerator called CrossLight that leverages silicon photonics. CrossLight includes device-level engineering for resilience to process variations and thermal crosstalk, circuit-level tuning enhancements for inference latency reduction, and architecture-level optimizations to enable better resolution, energy-efficiency, and throughput. On average, CrossLight offers 9.5x lower energy-per-bit and 15.9x higher performance-per-watt than state-of-the-art photonic deep learning accelerators.