Overcoming occlusions in AR, via multi-view, real-time 3D human pose estimation

We are excited to share that CERTH article, “Overcoming occlusions in AR, via multi-view, real-time 3D human pose estimation,” has been published in Machine Vision and Applications, Springer. 

The developed system gives users “X-Ray vision” to enhance situational awareness in in dynamic environments where visibility is compromised, potentially benefiting various applications, from industrial inspection and safety, first response scenarios to security and surveillance. The framework enables the projection of accurate 3D human skeletal representations onto AR glasses , allowing users to “see” concealed people behind obstacles in real-time.

Key technical highlights include:

Multi-View Fusion: The method utilizes clusters of calibrated cameras to estimate 3D poses, handling complex occlusions that standard single-camera (monocular) setups cannot.

Robust Occupant Tracking: The system integrates a lightweight 3D tracker and Bird’s-Eye View (BEV) tracking to ensure AR users can distinguish hidden individuals by unique IDs , maintaining consistent identity.

Attention-Driven Accuracy: By incorporating a Normalization-based Attention Module (NAM) and end-to-end training, 3D joint error has been reduced compared to baseline methods.

Precise Spatial Alignment: To ensure the virtual skeletons line up perfectly with the real world , AprilTag fiducial markers are fused with IMU sensor data to track the user’s head orientation with high precision .

Real-Time Performance: The entire pipeline, from pose estimation to tracking and AR projection, runs at up to 41 FPS on a single commercial GPU.

Check out the full paper here:  https://doi.org/10.1007/s00138-025-01783-9