Supervisor: Prof. Ville Kyrki (ville.kyrki(at)aalto.fi)
Advisor: Dr. Almas Shintemirov (almas.shintemirov(at)aalto.fi)
Keywords: dynamic modeling, deep learning, deformable object manipulation.
Figure 1 – Simulation of a plastic bag manipulation by a Franka robotic arm in NVIDIA Isaac Sim.
Manipulation of deformable objects such as textiles is a very challenging task due to the complexity and high dimensionality of object modeling. Latest strategies for dynamic model generation of deformable objects are based on applying a variety of deep learning configurations, e.g. [1-3]. In this thesis the Iterative Residual Policy (IRP), a general learning framework applicable to repeatable tasks with complex dynamics , will be adopted and complex manipulation task and motion planning scenarios such as plastic bag manipulation, e.g. [5, 6]. A state-of-the-art NVIDIA Isaac Sim robotics simulator, offering photorealistic, physically accurate virtual environments to develop, test, and manage AI-based robots , will be used for synthetic data generation, deep learning model training and action verification through deploying generated motion planning scenarios on a simulated and experimental Franka research robot-manipulator in the Aalto Robot Lab (TUAS building lobby).
- Literature review om dynamic modeling of deformable objects for manipulation tasks;
- Implementation of relevant deep learning and robot simulation scenarios.
- Evaluation of the developed models and algorithms on a real Franka robot
Pre-requisites: Python (high/medium), Machine/Deep Learning (medium), C++ (beginning)
Tools: Nvidia Omniverse Isaac Sim, PyTorch, Robot Operating System (ROS)
Start: Available Immediately
- H. A. Kadi, K. Terzi_Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: present, challenges, and future prospects, Sensors, 2022 https://www.mdpi.com/1424-8220/23/5/2389
- X. Ma, D. Hsu, W-S. Lee, Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects https://arxiv.org/abs/2104.12149
- B. Shen, et al. ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation, https://arxiv.org/abs/2203.06856
- C. Chi, et al. Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects. https://arxiv.org/abs/2203.00663
- Z. Weng, et al. Graph-based Task-specific Prediction Models for Interactions Between Deformable and Rigid Objects, https://arxiv.org/abs/2103.02932
- L. Chen, et al. AutoBag: Learning to Open Plastic Bags and Insert Objects https://arxiv.org/pdf/2210.17217.pdf