Ray-based multi-view 2D to 3D pose lifter
(2025)
Files
thesis.pdf
Open Access - Adobe PDF
- 18.76 MB
Details
- Authors
- Supervisors
- De Vleeschouwer, Christophe
- Abstract
- Estimating 3D human poses from 2D images is a challenging problem in computer vision, with many applications in different fields such as sports analytics, medical diagnostics, virtual reality, and robotics. Although deep learning methods have advanced 2D pose estimation, estimating 3D poses from monocular 2D images remains challenging, since it is an ill-posed problem due to the ambiguity caused by projective geometry. Multi-view setups offer a potential solution, as they increase the availability of spatial information; however, state-of-the-art methods for multi-view 3D pose estimation are heavily reliant on datasets of multi-view images paired with 3D pose annotations. These datasets are often confined to controlled, laboratory environments, limiting the adaptability of existing models to the diverse and dynamic conditions encountered in real-world scenarios. This thesis addresses these limitations by presenting a novel, generalizable framework for 3D human pose estimation, which decouples the tasks of 2D pose detection and 3D pose lifting. In the first stage, off-the-shelf 2D pose estimation models are used to independently detect anatomical keypoints in each view. In the second stage, a transformer-based architecture, called Ray-based Multi-view 3D Pose Lifter (RMPL), fuses these 2D keypoints into a unified 3D pose skeleton. Unlike previous methods, RMPL operates entirely in the pose space rather than the image space, and it does not require paired images and 3D pose datasets for training. Instead, the proposed framework leverages synthetic 2D-3D pose pairs, generated using a novel pipeline that simulates the noise characteristics of 2D pose estimators. Moreover, a novel ray-based representation of 2D keypoints is introduced, which enables the model to be invariant to camera calibration parameters. This innovative approach makes it possible to train the RMPL for arbitrary camera setups, ensuring adaptability across diverse deployment scenarios. Experimental validation demonstrates the effectiveness and robustness of the proposed framework. Evaluations on widely used benchmarks, including the Human3.6M, CMU Panoptic, and RICH datasets, reveal that the RMPL model achieves significant improvements in 3D pose accuracy, with reductions in Mean Per Joint Position Error (MPJPE) of up to 53% compared to triangulation-based baselines. This reduction is above 60% when comparing to transformer-based solutions using image-based representation of 2D pose. Further, the study includes comprehensive analyses of the impact of visibility, camera configurations, and synthetic data quality on the performance of the proposed method. By eliminating the dependency on the costly and limited 3D human pose datasets, this research not only advances the state-of-the-art in multi-view 3D human pose estimation but also establishes a scalable and practical framework for deployment in real-world environments. The modular design of the proposed system ensures compatibility with a wide range of 2D pose detectors and camera setups, paving the way for future developments in human motion analysis and its applications.
- Affiliations
Citations
Ghasemzadeh, S. A. (2025). Ray-based multi-view 2D to 3D pose lifter. https://hdl.handle.net/2078.5/246516