Integrating Computer Vision and Audio Signals for Concrete Vibration Activity Recognition and Assessment

作者：Jiaqi Li, Zhuo Chen, Zhaobo Li, Lingjie Kong, He Zhang · 发表于：Journal of Construction Engineering and Management · 年份：2026 · DOI：10.1061/jcemd4.coeng-17603 · 被引用次数：2 · 研究领域：Occupational Health and Safety Research、BIM and Construction Integration、Tunneling and Rock Mechanics

Concrete vibration is a crucial step in concrete construction, and the adequacy of vibration time affects the quality of the work to a certain extent. Monitoring vibration time helps identify potential cases of inadequate vibration, excessive vibration, or task negligence. However, existing methods generally rely on a single modality: visual-based methods are unable to identify whether the vibrator has actually been started, while audio-based methods can capture vibration sounds but cannot distinguish between specific operators. These limitations reduce the ability to accurately assess vibration behavior. To more accurately evaluate the concrete vibration process, this paper proposes a method that integrates results from both computer vision and audio signal classification. First, we trained a You Only Look Once (YOLO) v8-based object detection model to identify workers, vibrating activities, and vibrators. A worker reidentification (ReID) model based on Transformer was developed using a custom worker reidentification dataset. After inputting video into the object detection model, the presence of vibration activities was recognized by analyzing the spatial relationships between the operator, vibration activities, and the vibrator. Subsequently, the ReID algorithm was used to retrieve and match the operator, linking the recognized vibration activity to its executor. Additionally, Mel spectrograms were extracted from 767 audio segments, and a convolutional neural network (CNN)-...