In this work, a hybrid approach for wave spectrum estimation is proposed. Fundamentally, the approach is based on the wave buoy analogy, processing ship response measurements, via a framework combining machine learning and a physics-based method dependent on available transfer functions. Specifically, a non-parametric (Bayesian) estimate is obtained of the directional wave spectrum conditioned on integral wave parameters established by a convolutional neural network. The developed method is assessed in a case study considering about two years of data obtained from an in-service container ship. The method produces good results, significantly improved when compared to the initial estimate made without constraints.