The noise conversion relationships of relative intensity noise (RIN) and shot noise in a broadband source sensing system using a 3×3 coupler are studied. RIN and shot noise are converted into the phase noise floor through the arctangent demodulation scheme and the performances of the 3×3 coupler. To quantify the contribution of RIN and shot noise, we analyzed the equivalent phase noise induced by RIN and shot noise. The theory and experiment demonstrate that the equivalent phase noise of RIN is related to the correlation of three RINs and the initial phase, and the shot noise is related to the initial phase.