Tolga-Can Callar: Learning of Inverse Dynamics for Proprioceptive Force Estimation during Irregular Fine-Scale Robot Motion

Supervisors: Sven Böttger, Elmar Rückert

Finished: 21.September 2021

Abstract

The applicability of robotic automation has transcended the industrial domain through the emergence of collaborative robotics and is increasingly entering the realm of applications with high levels of physical human-robot interactions. This is concomitant with a paradigm shift towards higher force control sensitivity to accomplish functional and safety requirements concerning the regulation of contact forces between robots and humans. A fundamental challenge in this regard is the observability and estimation of interaction forces. Utilizing the availability of joint position and torque sensors in recent collaborative robot models that yield a larger perceptive field for interaction forces than local force sensors, a proprioceptive approach is taken in this thesis to develop inverse dynamic models to estimate dynamic disturbances and determine external interaction forces during fine-scale motion. A series of state-of-the-art techniques are implemented and evaluated on the KUKA LBR iiwa 14, including dynamic parameter identification, neural-network based single-step, and time-series models, and a novel hybrid architecture combining a rigid body dynamics model with downstream neural networks and joint rotational displacement encodings. The results indicate that significant improvements in torque and force estimation accuracy can be obtained by the proposed method when compared with conventional rigid body dynamics models or neural networks alone.

Thesis

B.Sc. or M.Sc. Thesis/Project: Running ROS-Mobile on a EV3

Supervisor: Linus Nwankwo, M.Sc.;
Univ.-Prof. Dr Elmar Rückert
Start date: ASAP from October 2021

 

Theoretical difficulty: low
Practical difficulty: mid

Abstract

Nowadays, robots used for survey of indoor and outdoor environments are either teleoperated in fully autonomous mode where the robot makes a decision by itself and have complete control of its actions; semi-autonomous mode where the robot’s decisions and actions are both manually (by a human) and autonomously (by the robot) controlled; and in full manual mode where the robot actions and decisions are manually controlled by humans. In full manual mode, the robot can be operated using a teach pendant, computer keyboard, joystick, mobile device, etc.

Recently, the Robot Operating System (ROS) has provided roboticists easy and efficient tools to visualize and debug robot data; teleoperate or control robots with both hardware and software compatibility on the ROS framework. Unfortunately, the Lego Mindstorms EV3 is not yet strongly supported on the ROS platform since the ROS master is too heavy on the EV3 RAM [2]. This limits our chances of exploring the full possibilities of the bricks.

However, in the context of this project, we aim to get ROS to run on the EV3 Mindstorms to enable us to teleoperate or control it on the ROS platform using a mobile device and leveraging on the framework developed by [1].

Tentative Work Plan

To achieve our aim, the following concrete tasks will be focused on:

  • Configure and run ROS on the Lego EV3 Mindstorms
  • Set up a network connection between the ROS-Mobile device and the EV3 robot
  • Teleoperate the EV3 robot on the ROS-Mobile platform
  • Perform Simultaneous Localization and Mapping (SLAM) for indoor applications with the EV3 robot

References

Meeting Notes Frontiers, Vacation, Cuda

Meeting Details

Date : 18th August 2021

Time : 15:00 – 15:20

Location : Web-ex Meeting

Participants: Univ.-Prof. Dr. Elmar Rueckert, Honghu Xue

Agenda

Vacation and Future Plan

Topic 1: Frontiers Journal

  1. Both reviewers endorsed publication of the paper, waiting for the final response of editor
  2. Print the Invoice from Frontiers, take to Conni and deduct from TRAIN Project.

Topic 2: Update Information on website

  1. Provide some tutorials on using Cuda Pytorch
    •  
  1. Update personal information
    • Add publication with Tuluhan.
    • Integrate it with the transfer learning model. 

Next Steps

  1. Data journal submission.
  2. Benedikt Hein’s Master thesis as conference submission.
  3. Benedikt Hein’s Master Internship as journal submission.

Next Meeting

Honghu will write you an email on when his flexible vacation ends.

GPU Cluster with eight A100

Getting started with Pytorch using Cuda acceleration

This tutorial gives an instruction on installing Cuda and enabling Cuda acceleration using Pytorch in Win10. Installation in Linux or Mac systems are all possible. An additional .py file will verify whether the current computer configuration uses the Cuda or not. The following instruction assumes that you have already installed Python IDE, e.g., Anaconda, Pycharm, Visual Studio…

Step 1: Check which Cuda version is supported by your current GPUs under this website. From the left figure, we can see that A100 supports Cuda 11.0. It is also reported from other blogs/ forums that A100 can support Cuda 11.1. In this post, we install Cuda 11.1.

Step 2: Download Nvidia Cuda Toolkit 11.1 (the same version as Cuda in Step 1) from the website. In Win10, for instance, we follow up the choice as shown right. The size of exe(local) is around 3.1GB. After downloading, run the .exe and perform installation. It may take some minutes to complete installation.

Step 3: On the homepage of Pytorch, choose the appropriate options as shown in the left figure. IMPORTANT: The cuda version must be the same as in Step 1. It is also recommended to use Stable version. After finishing the , copy the command into Anaconda Powershell Prompt or other command prompt where you install packages for Python. Waiting for the installation, which may require larger than 1GB disk space and takes some minutes for installation. You could also find historical version of Pytorch in that homepage.

Verify your installation with .py file

You could download a cuda-test.py file and run it. If the result shows ‘cuda’, then you can enjoy the Cuda acceleration for training neural networks!

Using Multiple GPUs for further acceleration

Running Pytorch with Multiple GPUs can further increase the efficiency. We have 8 GPU cards and can be used parallely for training. Please refer to (1) (2) (3) for details. 

Meeting Notes Template

As PhD you have to attach a meeting note to your personal profile post, e.g., Linus Nwankwo, M.Sc.. The notes have to be uploaded right after the meeting!

 

Clone this template and change the category to ‘notes_yourname’ (you may need to create this category first). You will need this category to display only your notes on your personal post. 

Make sure that the post you create is only accessible by the administrator and PhDs. In special case you may restrict the access only to you and your supervisor which is the administrator. 

 

The title of the meeting notes should contain the date and some keywords of the discussed topics. 

To attach the notes to your personal post, look at the example here. An image of the relevant section in your post is shown on top.

Meeting on the 13th of August 2021

Location: Chair of CPS

Date & Time: 13th Aug. 2021, 1pm to 2pm

Participants: Univ.-Prof. Dr. Elmar Rueckert, Linus Nwankwo, M.Sc.

 

Agenda

  1. Organisatorial progress update by Linus.
  2. Feedback to the research talk presentation by Elmar.
  3. Topics of promising future research direction.
  4. Next steps of Linus.
  5. Date of the next meeting.

Top 1: Organisational Update

add you text here. 

PhD Thesis Roadmap & Goals

Defining concrete measurable goals is of major importance for PhD students. 

These goals should be well crafted based on the research focus and the strengths of the student.

The goals form an agreement between the supervisor and the student. They can be used to evaluate the progress at any time. 

Your Thesis Topic in One Sentence

Every student should be able to explain their research topic and the scientific contribution in one sentence

Start thinking about it now. You will be asked dozens of times during your PhD at meetings or conferences.

What is your research about; and why is that important; and what is new; in one sentence. 

Thesis Topics

Ideally, your thesis consists of three parts or research topics. These parts might be adapted during the progress of the thesis. 

Each part should result an a journal publication and/or two conference publications. A good strategy is to first publish two short stories (= conference papers of 6-8 pages). Thereafter, the short stories are combined in a larger approach and published as journal paper. 

  1. Plan: planning future actions.
    Open research question: How can biologically inspired neural networks plan complex actions in few milliseconds?
  2. Act: Executing the actions. 
    Open research question: How can closed-loop controller be learned or generated from high-level plans?
  3. Sense: Sensing the environment and monitoring the execution. 
    Open research question: How to monitor the execution of neural closed-loop controller and to apply corrective actions in real-time?

Defining a Short Research Story

  1. Start by collecting the relevant related work to your research question. 
  2. Define an open unsolved problem and why it is relevant to the community. 
  3. Create measurable scores or goals how a solution to this open problem could be validated. 
  4. Also define 1-3 baseline algorithms or approaches which you will (re-)implement first. 
  5. Think about a toy task to demonstrate the principles. Typically, toy task simulations of point masses, simple 2D games or robot simulations in V-Rep are useful first case studies.
  6. Develop a relevant experimental setting to evaluate your approach. 
  7. Compute statistically significant statistics. Print Tables and Figures.
  8. Start by writing the results section of you paper. Describe the Tables and Figures first. Is your research question answered? 
  9. Continue with the methods section. Did you define a problem statement? Are your baseline approaches explained?
  10. Structure the collected related work and add it as subsection to the introduction.
  11. For the introduction, first write 1-3 sentences on the topic and the relevance. Next talk about the open problem and discuss the related work. Explicitly discuss your idea, own prior work and your contribution. 
  12. Write an abstract and a conclusion. 
 

PhD Thesis Goals and Achievements

.tg {border-collapse:collapse;border-color:#9ABAD9;border-spacing:0;} .tg td{background-color:#EBF5FF;border-color:#9ABAD9;border-style:solid;border-width:1px;color:#444; font-family:Arial, sans-serif;font-size:14px;overflow:hidden;padding:10px 5px;word-break:normal;} .tg th{background-color:#409cff;border-color:#9ABAD9;border-style:solid;border-width:1px;color:#fff; font-family:Arial, sans-serif;font-size:14px;font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;} .tg .tg-0lax{text-align:left;vertical-align:top}
Item Goal Accepted or
Achieved
Under review or
in Progress
Note
Journal Paper 2 - 3 min 1 accepted & 1 submitted
Conference Paper 4 - 8 min 2 accepted & 2 submitted
Workshop Paper or
Abstracts
2 - 4
Live Demo 1 - 2 min 1 own demo
Patents 0 - 2 a patent can replace a journal paper!
Conference Paper Review 8 - 24 number of reviewed papers, not conferences.
Journal Paper Review 2 - 6
B.Sc. Thesis 4 - 12
B.Sc. Project 4 - 12
M.Sc. Thesis / M.Sc. Project 3 - 6
Teaching assistant of exercises 2 semester. It can be the same exercise.
Guest lectures 0 - 4
Conference and Lab Talks 6 - 12
Research or
Industrial Collaborations
1 - 3
Summer Schools 1 - 2
Lab Stays at other Universities
for 1 - 3 weeks
0 - 2
Internship of 2 - 3 months 0 - 1
Soft Skill Training 2 - 6
Outreach (lab demos,
school events, fairs, etc.)
4 - 12
Contribution to a research
grant proposal
0 - 3 typically few paragraphs only.
Student or hiring committee
member
0 - 2

Checklist New Employee

Congratulations! You are now part of the CPS team. 

This post will help you setting up your work environment, getting ready to start as scientist and provides helpful instructions at the beginning. 

Organisation Things to Do

If you have not looked at our organisational instructions, do it right now! 

These steps are of utter importance. Your official start and when you get your first salary depend on these steps.

First Technical Steps

First Scientific Steps

Clone the PhD Roadmap post and add it as link to your people page post. Follow the instructions there, define topics and research questions. Remove unnecessary parts like the section on ‘Defining a Short Research Story’.

Checklist New Employee for Admins

When a new employee starts at the Chair of CPS, a number of things needs to be done by the Chair, the secretary and the technician. 

Just before the new employee starts

Items to be done at the first day at work

Items to be done after the Work Contract was signed

Welcome to Leoben for Researcher

Organisational Steps after your Arrival

There are a few very important things you need to do after you arrived. Please update the information if it is outdated. 

1. Get a Certificate of Registration from your Landlord

Once you signed the contract for your room or flat in Austria, you get a Certificate of Registration (germ. Meldebestätigung). An example is shown in the Figure on the left. If you did not get such a certificate, just print the PDF and get it signed by your landlord. 

 

2. Get a Registration Form from the City Hall

You need to go to the city hall (germ. Stadtgemeinde Leoben) to register your new main residence.  You will receive a Registration form (germ. Meldebestätigung). An example is shown in the left panel.

3. VISA or Residence Permit for Researcher

As researcher you can apply for a Residence Permit for Researcher. Sadly, the government’s information page is only in german. 

Below we outline the necessary steps for Non-Europeans. 

4. Get in contact via Email with the District Headquarters

To get a residence permit, you need to get in contact with the district headquarters Leoben (DH). Send them an Email or call them. The district headquarters will need from you: 

  1. A copy of your passport.
  2. Your Work contract.
  3. Your Registration form.

Once you send all your files to the DH, they will send you an appointment date via email. 

5. Documents required from Non-EU applicants

Find all important information on the page of the government

In order to obtain a residence permit, you need to contact the Bezirkshauptmannschaft Leoben. Send them an email or call them. The best way is to call them and inform them about your visa status. They will send you and appointment and you will need to submit the following documents:

  1. Completed and signed application form
  2. Copy of valid travel document
  3. Birth certificate (with translation)
  4. Proof of criminal record from the place of your residence (with translation). Note in some cases also records from your home country are needed!
  5. Current passport photo in the size of 3,5 x 4,5 cm (not older than 6 months)
  6. City Registration certificate
  7. Proof of legal entitlement to accommodation in Austria: e.g. rental contract, housing rights agreement.
  8. Proof of health insurance valid in Austria (probably only possible after starting work)
  9. Original admission agreement with the University of Leoben
  10. Original employment contract with the University of Leoben

6. Send all your Files to Mrs. Hotter

Mrs. Hotter of the Montanuniversität Leoben is responsible for new employees. Send her your files asap.