Explainable Interactions between Humans and Autonomous Systems
With the growing advancement of robotics research, there is a growing need for people-friendly communication between robots and humans. On one hand, the decisions of the autonomous system need to be understandable to humans, and on the other – humans need to be able to specify commands in a way that is natural to them.
In cases of shared autonomy, where an operator needs to take control in case the autonomous system is uncertain about a situation, our research focuses on automatically generating hypothetical explanations of the model decisions and analysis of which part of the data would be relevant for a human to understand the situation in order to react in a timely manner.
Another direction of the research is allowing humans to specify commands to the autonomous systems in an intuitive way – by using natural language.
People involved
- Tsvetomila Mihaylova (tsvetomila.mihaylova(at)aalto.fi), Postdoctoral researcher.
- Ville Kyrki (ville.kyrki(at)aalto.fi), Professor, group leader.
Project updates
Robotic Navigation with Natural Language Commands
The main goal of this project is to add capabilities for processing commands in natural language to an existing system for robot navigation.
Master Thesis on “Fine-tuning Open-source LLMs for Processing Open-vocabulary Commands for Robotic Navigation”
Most of the systems performing open-vocabulary navigation use closed LLMs accessible via an API. If this kind of systems are to be adopted for practical robotic applications, it is important that the information is not sent to third-party systems. The goal of the project is to evaluate fine-tuning of local open-source models for processing natural language commands for robotic navigation.
Master Thesis on “Natural Language Commands for Robotic Navigation”
Recent breakthroughs in natural language processing are enabling robots to understand human language like never before. The thesis will address the problem of translating human language to executable robot skills.
Master Thesis on “Controlling a Robotic Arm with Instructions in Natural Language”
The goal of this master thesis is to integrate a large vision-language model (VLM) with a manipulation policy in order to control a robotic hand for predefined manipulation tasks, such as grasping or pushing.
Master Thesis on “Explaining Driving Situations with Natural Language”
The goal of this master thesis is to explore existing approaches, datasets and models that provide textual explanations of driving situations, to implement a state-of-the-art model and to validate it on predefined driving conflict situations.