Dataset Open Access
Nannan Yu, Tao Ma, Jiqing Zhang, Yuji Zhang, Qirui Bao, Xiaopeng Wei, Xin Yang*
Adaptive Vision Transformer for Event-Based Human Pose Estimation
The 32th ACM International Conference on Multimedia (ACM MM2024). (CCF A)
Introduction:
Event-based human pose estimation has gained popularity due to the benefits of high temporal resolution and high dynamic range offered by event cameras. The inherent spatial sparsity of event data makes discarding less significant regions a straightforward and effective way to decrease the computation. However, implementing this operation in CNNs poses a challenge, as it disrupts the regularity of dense convolutional workload. In this paper, we propose an adaptive vision transformer, a novel efficient backbone for human pose estimation with event cameras. Specifically, we present two adaptive patch and token sampling approaches based on the characteristics of events, thereby reducing the computational load while still achieving comparable performance. Firstly, we design an adaptive patch sampling scheme to eliminate inactivity patches by assessing the entropy of the events before they are inputted into the transformer. Secondly, we further propose an adaptive token reduction strategy to selectively remove less informative tokens in transformer layers through a dynamic token pruning algorithm. To exploit event-based visual cues in human pose estimation tasks, we construct a large-scale frame-event-based dataset, dubbed Event Multi Movement HPE (EventMM HPE). Our dataset collects 76 human pose estimation sequences con- sisting of 21 different actions of 7 subjects. Each subject is labeled with 17 joint points. The annotation frequency is up to 240Hz. To encourage community research on multi-modal fusion, we have also gathered synchronous high-frame-rate RGB images with 120FPS. Extensive experiments demonstrate that our proposed approach outperforms existing state- of-the-art methods in estimation accuracy.

Figure 2: a-c) Prophesee camera, High frame-rate camera, and Vicon IR Camera. d) Vicon key positions on the subject and skeleton reputation. e) Schematic of the setup, with Prophesee master camera (position 1), High frame-rate camera (position 2), and Vicon origins.

Figure 7: Event-based human pose estimation results of our method.
Please enter your application information for dataset