Multimodal Human Activity Recognition with Video & IMU Sensor Fusion

Overview

Developed an end-to-end deep learning pipeline for multimodal Human Activity Recognition (HAR) as part of the 3rd WEAR Dataset Challenge @ HASCA (organized in conjunction with ACM UbiComp / ISWC). The challenge requires classifying complex outdoor workout activities and distinguishing them from a dominant background "null" class under naturalistic conditions.

The pipeline ingests and synchronizes diverse multimodal inputs, integrating high-rate inertial measurement unit (IMU) signals (accelerometer and gyroscope time-series windows) with egocentric video representations (pre-extracted deep visual features such as VideoMAEv2). Researched and benchmarked a variety of fusion paradigms—evaluating early feature concatenation, cross-modal attention, and late decision-level fusion—while experimenting with specialized loss functions (e.g., Focal Loss and weighted Cross-Entropy) to understand how different fusion mechanisms handle extreme null-class dominance and sensor complementarity.

Key Highlights

• Multimodal Deep Learning Architecture: Designed a unified pipeline bridging continuous temporal IMU sensor windows with egocentric video feature representations.
• Cross-Modal Fusion Strategies: Explored and compared early, intermediate (cross-attention), and late fusion methodologies to examine how inertial kinematics and visual context complement each other in activity classification.
• Loss Function & Imbalance Optimization: Experimented with customized loss formulations including Focal Loss, label smoothing, and class-weighted Cross-Entropy to counter the severe imbalance created by the prevailing null class.
• Competitive Kaggle Benchmark: Evaluated model performance across 18 distinct physical workout exercises plus the null class under subject-disjoint validation protocols.

Resources