Bottom depth ( H ) of optically shallow waters can be retrieved from multiband imagery, where remote sensing reflectance ( Rrs ) are commonly used as the input. Because of the difficulties of removing the atmospheric effects in coastal areas, quite often, there are no valid Rrs from satellites for the retrieval of H . More importantly, the empirical algorithms for H are hardly portable to new measurements. In this study, using data from Landsat-8 and ICESat-2 as examples, we present an approach to retrieve H directly from the top-of-atmosphere (TOA) data. It not only bypasses the requirement to correct the effects of aerosols but also shows promising portability to areas not included in algorithm development. Specifically, we use Rayleigh-corrected TOA reflectance ( ρrc ) in the 443–2300 nm range as input, along with a multilayer perceptron ( MLPHρrc ), for the retrieval of H . More than 78,000 matchup points of ρrc and H (0–25 m) were used to train MLPHρrc , which resulted in a Mean Absolute Percentage Difference (MARD) of 8.8% and a coefficient of determination ( R2 ) of 0.96. This MLPHρrc was further applied to Landsat-8 data of six regions not included in the training phase, generating MARD and R2 values of 8.3% and 0.98, respectively. In contrast, a conventional two-band ratio algorithm with Rrs as the input generated MARD and R2 values of 31.6% and 0.68 and significantly fewer H retrievals due to failures in atmospheric correction. These results indicate a breakthrough of algorithm portability of MLPHρrc in sensing H of optically shallow waters.