Master Thesis on “Learning Based Dynamic Manipulation of Deformable Objects”

21/08/2023

Supervisor: Prof. Ville Kyrki (ville.kyrki(at)aalto.fi)

Advisor: Dr. Almas Shintemirov (almas.shintemirov(at)aalto.fi)

Keywords: dynamic modeling, deep learning, deformable object manipulation.

Figure 1 – Simulation of a plastic bag manipulation by a Franka robotic arm in NVIDIA Isaac Sim.

Project Description

Manipulation of deformable objects such as textiles is a very challenging task due to the complexity and high dimensionality of object modeling. Latest strategies for dynamic model generation of deformable objects are based on applying a variety of deep learning configurations, e.g. [1-3]. In this thesis the Iterative Residual Policy (IRP), a general learning framework applicable to repeatable tasks with complex dynamics [4], will be adopted and complex manipulation task and motion planning scenarios such as plastic bag manipulation, e.g. [5, 6]. A state-of-the-art NVIDIA Isaac Sim robotics simulator, offering photorealistic, physically accurate virtual environments to develop, test, and manage AI-based robots [7], will be used for synthetic data generation, deep learning model training and action verification through deploying generated motion planning scenarios on a simulated and experimental Franka research robot-manipulator in the Aalto Robot Lab (TUAS building lobby). 

Deliverables

  • Literature review om dynamic modeling of deformable objects for manipulation tasks;
  • Implementation of relevant deep learning and robot simulation scenarios. 
  • Evaluation of the developed models and algorithms on a real Franka robot

Practical Information

Pre-requisites: Python (high/medium), Machine/Deep Learning (medium), C++ (beginning)

Tools: Nvidia Omniverse Isaac Sim, PyTorch, Robot Operating System (ROS)

Start: Available Immediately

References

  1. H. A. Kadi, K. Terzi_Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: present, challenges, and future prospects, Sensors, 2022 https://www.mdpi.com/1424-8220/23/5/2389 
  2. X. Ma, D. Hsu, W-S. Lee, Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects  https://arxiv.org/abs/2104.12149 
  3. B. Shen, et al. ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation, https://arxiv.org/abs/2203.06856 
  4. C. Chi, et al. Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects. https://arxiv.org/abs/2203.00663 
  5. Z. Weng, et al. Graph-based Task-specific Prediction Models for Interactions Between Deformable and Rigid Objects, https://arxiv.org/abs/2103.02932 
  6. L. Chen, et al. AutoBag: Learning to Open Plastic Bags and Insert Objects https://arxiv.org/pdf/2210.17217.pdf 
  7. https://developer.nvidia.com/isaac-sim 
Tags: