Unmanned aerial vehicle (UAV)-borne light detection and ranging (LiDAR) scanners have been adopted as a promising instrument for plant parameter estimation in agricultural studies recently. However, accurate LiDAR data registration typically requires expensive external navigation devices such as survey-grade global navigation satellite systems (GNSSs) and tactical-grade inertial measurement units (IMUs). Although algorithmic point cloud registration can be an alternative method, the lack of unique landmarks in agricultural fields might bring much difficulty to accurate aerial LiDAR data alignment. In this study, we developed a UAV-LiDAR system employing UAV’s built-in navigation units, and proposed a novel approach for registering UAV-LiDAR data of level agricultural fields utilizing a colored iterative closest point (ICP) algorithm and GNSS location and IMU orientation information from the UAV. The proposed algorithm was tested in a peach tree parameter estimation experiment in comparison to GNSS and IMU-based georeferencing. Using manually measured crown widths in two perpendicular dimensions and heights of 11 trees as evaluation metrics, our proposed algorithm achieved a root mean square error (RMSE) range of 0.05 to 0.2 m depending on the tree parameter and flight altitude, and it was able to register tree point clouds up to 67% more accurately in terms of the extracted tree parameters than the georeferencing method. The results demonstrated the potential of the proposed algorithm being a low-cost solution to crop inspection using single-pass aerial LiDAR point clouds from straight-pathed flights, yet future work is still needed to improve the algorithm’s adaptability to multi-pass LiDAR data of complex landscapes from flights with curved paths.