A simple method is proposed to calibrate the fisheye distortion center, which synthesizes the algorithms of the ellipse fitting and gaussian surface fitting. Through analyzing the imaging projection of the equidistant fisheye lens, the calibration principle is introduced in detail. From the experimental results, we find that ellipse fitting error and the relative error of gaussian surface fitting are less than 0.06 pixels and 1.5%, respectively, and the proposed method has a higher calibration precision and stability compared with the methods in the inferences. Besides, the proposed method does not rely on any particular distortion model and other fisheye parameters, and only the image information of the calibration board is utilized, which reduces the calibration complexity and is suitable for the center calibration of any equidistant fisheye camera.