Whey protein-based nanocomplexes were fabricated to improve the stability and bioavailability of marine carotenoid-fucoxanthin (FX) by using whey protein isolate (WPI) as the core and Ca2+ crosslinked flaxseed gum (FG) as the shell. The results of multi-spectroscopic methods indicated that the incorporation of FX and FG changed the structural characteristics of WPI. FX could quench the intrinsic fluorescence of WPI by a static mechanism, indicating that they could spontaneously form a nanocomplex through non-covalent interactions. FX-WPI-FG-Ca nanocomplex with a nano scale (~350 nm) was found to perform the best with the highest encapsulation efficiency (EE, 96.19 ± 0.33%) and loading capacity (LC, 2.47 ± 0.04%) and improved FX stability against ultraviolet B (UVB), heat, NaCl and pH. In addition, the nanocomplex was shown to effectively deliver FX into PC12, and cytotoxicity increased with the concentration. The encapsulated-FX perturbed lipid metabolism, causing ROS accumulation, mitochondrial damage and regulated apoptosis. In conclusion, the nanocomplexes assembled from WPI and FG may be effective encapsulation systems for FX, which could potentially become a new dietary supplement for cancer patients.