In this paper, a binary geometrical pattern is used for the calibration of structured light system instead of conventional checkboard pattern. The pattern is designed based on pseudo-random coding theory, and the calibration procedure contains following steps. First, an accurate feature detector is proposed by utilizing the geometric property of the pattern elements. It is show that, with the proposed feature detector, feature points can be robustly localized with sub-pixel precision. Based on the extracted feature points, a topological structure is constructed to separate all the pattern elements. The pattern elements are extracted with affine transformation theory and bilinear interpolation. Secondly, to identify the pattern elements, the convolutional neural network technique is adopted, which is trained by collecting a large number of pattern element samples. After the decoding stage, code words of the feature points can be computed. According to the projective transformation principle, pattern feature points in the camera image plane can be transformed to the projector image plane with the corresponding code word. Finally, both intrinsic and extrinsic parameters of camera and projector can be calculated. Experimental results show that the re-projector error of projector calibration results can be controlled within 0.3 pixels. In comparison with conventional calibration methods, both calibration and 3D reconstruction precision can be improved by the proposed approach.