Telecentric cameras are widely used in the field of microscopic imaging because of their constant magnification and tiny distortion in the depth of field. Camera calibration has always been a key step in the field of computer vision. Usually, the precise parameters of the telecentric camera are obtained by nonlinear optimization; however, the randomness of the optimization algorithm without proper constraints will cause the results to be inconsistent with reality. Existing studies paid little attention to this issue; therefore, we show a reliable optimization approach for the bi-telecentric camera in a structured illumination three-dimensional microtopography measurement system. In this method, the distortion-free camera parameters are solved through a closed-form solution. Then a nonlinear optimization algorithm with constraining the world coordinates of the precise calibration target is proposed to refine the global parameters, leading to the calibration results being more accurate and authentic. The real experiments are conducted to verify the feasibility of the proposed method. The comparative experiments with the exiting approach are then carried out, manifesting that the proposed method enjoys advantages in terms of both reprojection error and operating efficiency. Additionally, the average offset of the world coordinates on the calibration target derived from the proposed method verifies its effectiveness and reasonability.