Intelligent Robotics

Supervisor: Prof. Ville Kyrki (ville.kyrki@aalto.fi)
Advisor: Dr. Farzeen Munir (farzeen.munir@aalto.fi)

Keywords: Driver attention prediction, Computer vision, Multimodal learning, Gaze, Autonomous driving, Carla

Project Description

Modern vision foundation models such as DINOv2 and DINOv3 provide powerful general-purpose visual representations, but they are not explicitly trained to identify which parts of a traffic scene are most relevant for driving decisions. Human drivers naturally perform this selection through visual attention, focusing on relevant elements such as pedestrians, vehicles, traffic lights, and potential hazards before taking an action. This thesis investigates whether human gaze can be used as privileged supervision to improve foundation-model representations for autonomous driving. Using the LAIA (Labelled Attention for Intelligent Automobiles) dataset, which provides synchronized driving videos, human gaze measurements, vehicle-control signals, and different environmental conditions, the student will develop a gaze-aware teacher model built on a pretrained vision foundation model. The teacher will use human gaze to emphasize driving-relevant visual features while learning to predict driving actions such as steering, throttle, and braking.

The main research challenge is to transfer this gaze-informed knowledge to an RGB-only student model that does not require gaze information during deployment. The thesis will investigate knowledge-distillation strategies that transfer driving actions, intermediate visual representations, and/or attention patterns from the gaze-aware teacher to the student. The resulting system will be compared with a standard foundation-model-based driving model to determine whether human-attention supervision improves driving-action prediction, learned visual attention, and generalization across changing environmental conditions. Ultimately, the thesis aims to determine whether human visual attention can serve as an effective training signal for teaching autonomous-driving models what information matters for driving, while retaining the practical advantage of camera-only inference.

Deliverables

• Foundation-model driving baseline: Implement an RGB-only driving-action prediction model using a pretrained vision foundation model such as DINOv2/DINOv3 and the LAIA dataset.
• Gaze-aware teacher model: Develop a teacher model that incorporates human gaze as privileged supervision to learn driving-relevant visual representations and predict vehicle controls.
• RGB-only distilled student: Develop and evaluate knowledge-distillation methods for transferring gaze-informed features, attention, and/or driving knowledge from the teacher to an RGB-only student.
• Experimental evaluation and thesis: Conduct systematic experiments and ablations comparing baseline, teacher, and student models, including driving-action prediction, gaze/attention alignment, and generalization across environmental conditions.

Practical Information

Pre-requisites: Python(high), Deep Learning (high)

Tools: PyTorch

Start: Available immediately

References

• Scaling Vision-based End-to-End Autonomous Driving with Multi-View Attention Learning (https://arxiv.org/pdf/2302.03198)
• Guiding Attention in End-to-End Driving Models (https://arxiv.org/pdf/2405.00242)
• The LAIA Dataset: Labelled Attention for Intelligent Automobiles (https://arxiv.org/pdf/2607.25570)
• DINO-MX: A MODULAR & FLEXIBLE FRAMEWORK FOR SELF-SUPERVISED LEARNING (https://arxiv.org/pdf/2511.01610)