Problem Overview
Predicting whether a pedestrian intends to cross the street is crucial for safe autonomous driving in urban environments. Unlike simple detection, crossing intention prediction requires understanding subtle behavioral cues — body language, gaze direction, and contextual relationships with the environment — often seconds before the pedestrian actually steps onto the road.
Method
Our approach combines three complementary information sources:
- Local context (RGB): Pedestrian bounding box cropped from the camera feed, capturing fine-grained body pose and motion
- Global context (Semantic segmentation): Scene-level understanding of road layout, crosswalks, traffic signals, and other agents
- Ego-vehicle speed: Critical contextual cue — a slowing vehicle signals yielding intent to the pedestrian
Spatio-temporal Attention Architecture
The core innovation lies in our two-stage attention mechanism:
- Spatial Attention: Fuses the three input modalities by learning which scene regions and features are most relevant to the pedestrian's decision-making process
- Temporal Attention: Applied over a recurrent neural network (LSTM) that processes video frames sequentially, allowing the model to focus on critical moments in the pedestrian's approach to the curb
Key Results
- Outperformed single-modality baselines by 8-12% in F1 score
- Achieved 92.3% accuracy at 0.5s before crossing — crucial for real-time intervention
- Ablation studies confirmed that ego-vehicle speed provides uniquely valuable context beyond visual features alone
Dataset & Code
Experiments conducted on the JAAD (Joint Attention for Autonomous Driving) dataset, which contains 346 video clips with annotated pedestrian crossing behaviors.