Abstract:
Action recognition is a critical task in video understanding, where fine-grained action recognition poses greater challenges compared to conventional action recognition. While existing multimodal large models have achieved notable success in video understanding tasks, they still exhibit limitations in capturing fine-grained spatial and temporal information, leading to suboptimal performance. To address this issue, we propose an enhanced fine-grained information alignment and feature fusion method based on the video foundation model InternVideo2’s visual encoder. The approach consists of three key steps: First, region-of-interest feature alignment, extracting and aligning fine-grained features from localized regions of interest; Second, auxiliary encoding branch, introducing a dedicated visual encoding branch, fine-tuned on the extracted fine-grained features; Third, feature fusion for classification, incorporating the encoded fine-grained features as supplementary cues to improve action classification scoring. Through the above steps, the proposed method effectively improves the performance of multimodal large models on fine-grained video action recognition. Extensive experiments on fine-grained action recognition benchmarks demonstrate the effectiveness of our method, achieving 68.2% (Something-Something V1) and 73.1% (Something-Something V2) Top-1 accuracy, achieving an effective improvement over the baseline model.