Poor intrinsic conductivity and huge volume expansion during charge/discharge process greatly limit the development of Ge-based ternary oxide as anode material for both lithium-ion batteries and sodium-ion batteries. To alleviate these issues, an ideal strategy is developed to achieve active particle nanocrystallization and composite with conductive carbon materials, simultaneously. Therefore, ultrasmall Fe2GeO4 nanodots (∼4.6 nm) uniformly and tightly anchored on 3D interconnected N-doped ultrathin carbon nanosheets (3D Fe2GeO4/N-CNSs) were constructed via one-step high temperature calcination process. This unique hybrid nanostructure can not only effectively enhance electron conductivity but also restrict the aggregation and volume fluctuation of Fe2GeO4 during the charge/discharge process. As a result, the 3D Fe2GeO4/N-CNSs electrode exhibited excellent electrochemical performances for both lithium-ion and sodium-ion battery anodes. When utilized for lithium-ion battery anode, the electrode delivered a highly reversible specific capacity (1280 mA h g−1 at 0.4 A g−1 after 180 cycles). It is the first time that Fe2GeO4 was applied for sodium-ion battery anode, which showed a remarkable rate capability (350 mA h g−1 at 0.1 A g−1 and 180 mA h g−1 at 22.8 A g−1), and ultralong cycling stability (∼86% reversible capacity retention after 6000 cycles).