Event Tubelet Compressor: Generating Compact Representations for Event-Based Action Recognition

Event cameras asynchronously capture pixel-level intensity changes in scenes and output a stream of events. Compared with traditional frame-based cameras, they can offer competitive imaging characteristics: low latency, high dynamic range, and low power consumption. It means that event cameras are ideal for vision tasks in dynamic scenarios, such as human action recognition. The best-performing event-based algorithms convert events into frame-based representations and feed them into existing learning models. However, generating informative frames for long-duration event streams is still a challenge since event cameras work asynchronously without a fixed frame rate. In this work, we propose a novel frame-based representation named Compact Event Image (CEI) for action recognition. This representation is generated by a self-attention based module named Event Tubelet Compressor (EVTC) in a learnable way. The EVTC module adaptively summarizes the long-term dynamics and temporal patterns of events into a CEI frame set. We can combine EVTC with conventional video backbones for end-to-end event-based action recognition. We evaluate our approach on three benchmark datasets, and experimental results show it outperforms state-of-the-art methods by a large margin.