The recovery of rare earth elements (REEs) including neodymium (Nd) and dysprosium (Dy) from NdFeB permanent magnets has become one of the main ways to solve the increased demand for rare earth. Herein, n-dodecyl phosphate (DPPA) was used for the first time as the adsorption functional group donor, sodium alginate as the substrate, and calcium chloride solution as the reactive solvent, a hybrid hydrogel adsorbent DPPA/CaALG was synthesized by sol-gel method for application in the adsorption and separation of Nd and Dy from the Co-Nd-Dy ternary system. SEM-EDS, and N2 adsorption-desorption analysis showed the successful preparation of DDPA/CaALG with mesoporous structure. Batch experiments showed the superiority of the hybrid hydrogel for the good selective adsorption of Nd and Dy, such as large adsorption capacity (Nd: 162.5 mg/g, Dy: 183.5 mg/g), and no adsorption for Co. FT-IR, XPS showed that PO and P-O groups are involved in the adsorption process of Nd and Dy as electron acceptors, where the ion exchange of P-OH is dominant. Furthermore, the chemical properties of ligands and complexes were analyzed by Density Functional Theory (DFT) calculations and revealed their adsorption behaviors as well as the competition between different metal ions.