Learning to Augment Poses for 3D Human Pose Estimation in Images and Videos

Existing 3D human pose estimation methods often suffer inferior generalization performance to new datasets, largely due to the limited diversity of 2D-3D pose pairs in the training data. To address this problem, we present PoseAug, a novel auto-augmentation framework that learns to augment the available training poses towards greater diversity and thus enhances the generalization power of the trained 2D-to-3D pose estimator. Specifically, PoseAug introduces a novel pose augmentor that learns to adjust various geometry factors of a pose through differentiable operations. With such differentiable capacity, the augmentor can be jointly optimized with the 3D pose estimator and take the estimation error as feedback to generate more diverse and harder poses in an online manner. PoseAug is generic and handy to be applied to various 3D pose estimation models. It is also extendable to aid pose estimation from video frames. To demonstrate this, we introduce PoseAug-V, a simple yet effective method that decomposes video pose augmentation into end pose augmentation and conditioned intermediate pose generation. Extensive experiments demonstrate that PoseAug and its extension PoseAug-V bring clear improvements for frame-based and video-based 3D pose estimation on several out-of-domain 3D human pose benchmarks.