The existing helmet detection system mainly uses a fixed camera, it cannot achieve full-area detection, and the previous detection algorithms based on deep learning have complex structures and high computational costs, which cannot meet the requirements of using mobile vehicles and embedded devices. In this paper, a lightweight helmet detection algorithm scheme based on unmanned aerial vehicle is proposed. The drone is loaded with camera to collect images of the construction site, and the image data is transferred to the computer via wireless communication. Based on the YOLOv5s target detection algorithm, a lightweight detection algorithm is investigated. To improve the detection more efficient, the YOLOv5s target detection algorithm is improved in terms of multi-scale detection, image preprocessing, unbalanced positive and negative samples, and inference speed. This design scheme combines deep learning and unmanned aerial vehicle technology, not only to realize real-time automatic detection of helmet wearing, but also can realize the full-area helmet detection of the construction site. Real experiments show that, the lightweight target detection model is only 1.72% lower than the mean average precision of the original model. The inference speed on the same CPU can be doubled, and the floating-point calculation is reduced from 16.5 billion to 3.4 billion times per second. The model size is almost 1/10 of the original size.