Cross-Modal AI for Wearables: When One Sensor Learns from Another

Wearables demand more accurate health monitoring, but PPG is vulnerable to motion artifacts and power burden, while ACC is stable but does not directly capture cardiovascular information. This study, presented at ICASSP 2026, sought to bridge this gap with cross-modal virtual sensing, where one sensor learns the representation of another.

The core architecture has a single lightweight temporal encoder learn a shared latent representation from synchronized sensor streams.

• ACC → virtual PPG reconstruction
• PPG → pseudo-motion embedding generation
• modality-aware denoising
• single-modality real-time inference

Experiments were conducted using 132 HIIT workout logs, 3-axis ACC, 4-channel PPG, and heart rate ground truth from an ECG-grade chest device. In an environment with rapid sprint-recovery transitions and large wrist movement, estimating heart rate from ACC alone was particularly difficult.

Attention-based refinement and VAE-based calibration transformed the noisy motion representation into a more physiologically meaningful latent structure. The goal was not to reconstruct the raw optical signal as-is, but to learn a heart-rate-related latent physiological representation, and the results showed substantial improvement over ACC-only estimation and performance approaching multimodal fusion in several settings.