1

Deep Reinforcement Learning for Navigation in Warehouses

|

Deep Reinforcement Learning (DRL) has demonstrated great success in learning single or multiple tasks from scratch. Various DRL algorithms have been proposed and were applied in a broad class of tasks including chess and video games, robot navigation or robot manipulation. 

In this work, we investigate the potential of DRL in a mapless navigation task within a warehouse. The challenges of the task are the partial observability of the space and the need of effective exploration strategies for fast learning of navigation strategies. 

We trained a mobile robot (the agent) from scratch and compared how different sensor observations could influence the navigation performance. The evaluated sensors are a 360-degree Lidar sensor, only depth image and only RGB image. For Lidar and RGB inputs, we evaluated partial and full observability of the state space. We successfully trained the agent to navigate to a goal with a reward setting that is also applicable to the real world.

Currently, we are extending the work to multi-modal sensor inputs with both Lidar and RGB inputs (RGB image only frontal view) and incorporated self-curriculum learning on a more challenging navigation task in a warehouse and obtained promising initial outcomes.

https://cps.unileoben.ac.at/wp/RGB_only_front_snail_140k.mp4https://cps.unileoben.ac.at/wp/RGB_only_front_snail_85k.mp4

The video shows learned navigation strategies using a single RGB-D camera mounted at the front of the robot.  The results were obtained after 85.000 interactions (single action excution, e.g. wheel velocity commands). 

This video shows the learnded strategy after 140.000 interactions with the environment. 



Intrinsic Motivation Learning in Stochastic Neural Networks

|

Video

Link to the file

You may use this video for research and teaching purposes. Please cite the Chair of Cyber-Physical-Systems or the corresponding research paper. 

Publications

2019

Tanneberg, Daniel; Peters, Jan; Rueckert, Elmar

Intrinsic Motivation and Mental Replay enable Efficient Online Adaptation in Stochastic Recurrent Networks Journal Article

In: Neural Networks - Elsevier, vol. 109, pp. 67-80, 2019, ISBN: 0893-6080, (Impact Factor of 7.197 (2017)).

Links | BibTeX

Intrinsic Motivation and Mental Replay enable Efficient Online Adaptation in Stochastic Recurrent Networks

2017

Tanneberg, Daniel; Peters, Jan; Rueckert, Elmar

Efficient Online Adaptation with Stochastic Recurrent Neural Networks Inproceedings

In: Proceedings of the International Conference on Humanoid Robots (HUMANOIDS), 2017.

Links | BibTeX

Efficient Online Adaptation with Stochastic Recurrent Neural Networks

Tanneberg, Daniel; Peters, Jan; Rueckert, Elmar

Online Learning with Stochastic Recurrent Neural Networks using Intrinsic Motivation Signals Inproceedings

In: Proceedings of the Conference on Robot Learning (CoRL), 2017.

Links | BibTeX

Online Learning with Stochastic Recurrent Neural Networks using Intrinsic Motivation Signals



Teaching the Humanoid Robot ICub Manipulation Skills

|

Video

Link to the file

You may use this video for research and teaching purposes. Please cite the Chair of Cyber-Physical-Systems or the corresponding research paper. 



150.570/571 Seminar Bachelorarbeit Industrial Data Science (5SH SE, WS21/22, SS 22)

|

You are interested in working with modern robots or want to understand how such machines ‘learn’?

If so, this bachelor thesis will enable you to dig into the fascinating world of robot learning. You will implement and apply modern machine learning algorithms in Python, Matlab or C++/ROS. 

Your learning or control algorithm will be evaluated in cyber-physical-systems. Find out which theses are currently supervised and offered. 

 

Links and Resources

Location & Time

Learning objectives / qualifications

  • Students will work on controlling, modeling and simulating Cyber-Physical-Systems and autonomously learning systems.
  • Students understand and can apply advanced model learning and reinforcement  techniques to real world problems.
  • Students learn how to write scientific reports.

Literature

  • The Probabilistic Machine Learning book by Univ.-Prof. Dr. Elmar Rueckert. 
  • Bishop 2006. Pattern Recognition and Machine Learning, Springer. 
  • Barber 2007. Bayesian Reasoning and Machine Learning, Cambridge University Press. 
  • Murray, Li and Sastry 1994. A mathematical introduction to robotic manipulation, CRC Press. 
  • B. Siciliano, L. Sciavicco 2009. Robotics: Modelling,Planning and Control, Springer.
  • Kevin M. Lynch and Frank C. Park 2017. MODERN ROBOTICS, MECHANICS, PLANNING, AND CONTROL, Cambridge University Press.


150.510 Industrial Data Science Projekt (8SH SE, SS 22)

|

You are interested in working with modern robots or want to understand how such machines ‘learn’?

If so, this project will enable you to dig into the fascinating world of robot learning.

The course provides a structured and well motivated overview over modern techniques and tools which enable the students to define learning problems in Cyber-Physical-Systems. 

Links and Resources

Location & Time

Learning objectives / qualifications

  • Students get a practical experience in working, modeling and simulating Cyber-Physical-Systems.
  • Students understand and can apply advanced model learning and reinforcement  techniques to real world problems.
  • Students learn how to write scientific reports.

Literature

  • The Probabilistic Machine Learning book by Univ.-Prof. Dr. Elmar Rueckert. 
  • Bishop 2006. Pattern Recognition and Machine Learning, Springer. 
  • Barber 2007. Bayesian Reasoning and Machine Learning, Cambridge University Press. 
  • Murray, Li and Sastry 1994. A mathematical introduction to robotic manipulation, CRC Press. 
  • B. Siciliano, L. Sciavicco 2009. Robotics: Modelling,Planning and Control, Springer.
  • Kevin M. Lynch and Frank C. Park 2017. MODERN ROBOTICS, MECHANICS, PLANNING, AND CONTROL, Cambridge University Press.


Responsibilities & Contacts

|

This post provides information on whom to contact depending on your purpose.

Note that this post is continuously updated to keep the contact persons up-to-date. 

If you discover out-dated information, please contact our secretary. 

 

Sabine Fluch

  • CISCO telephone stations for new employees. 

sabine.fluch@unileoben.ac.at

 

Helga Winklmayr

  • Transponder. 

helga.winklmayr@unileoben.ac.at

Julia Schmidbauer

  • Knowledge- and Technology Transfer & Business Partnerships

julia.schmidbauer@unileoben.ac.at

Moodle Courses

SAP GUI MAC



Adding a New Thesis

|

As Ph.D. student and as senior research you will supervise undergraduate and graduate students. 

Often students contact the chair and ask for thesis topics or even approach us with their own ideas. That’s great and the chair usually forwards these requests to Ph.Ds. and senior research to do research in that domain. 

Once you agreed on a topic with a student, you will need to define a roadmap or thesis concept with the student. 

The thesis topic will be published on our web page. Below are the instructions on how to do that. 

Clone an Existing Thesis Post

The best strategy is to clone an existing post. Is this is new to you, follow our instructions in the linked post. 

Select the Correct Category

Three categories are used for student theses:

  1. teaching_currentthesestopics
  2. teaching_openthesestopics
  3. teaching_pastthesestopics

For current ongoing theses use the first category, for open topics use the second, and for completed theses use the last category. 

Mandatory Information of your Thesis Post

  • The supervisors. Usually that are the chair and you. 
  • The start date. 
  • The end date for completed theses. 
  • An abstract.
  • A Roadmap with mandatory and optional tasks.
  • A list of related work. 

Add the Student’s Contact Details

Add the name and email of the student to our CPS list for informing the student about upcoming talks and important internal issues. 



Adding a New Publication

|

File Naming Conventions

First you need to give your pdf file a proper name, e.g., IROS2021Rueckert.pdf. Use the following naming convention

  • ConferenceAcronymYYYYFirstAuthorLastName

The ConferenceAcronym is for example ICRA, Humanoids, NeurIPS or for journals JMLR, RAL, etc.

YYYY denotes the year like 2021.

The FirstAuthorLastName might be followed by a keyword, if you have multiple papers at the same conference or journal. 

Featured Image

Create also a Featured Image from taking a screenshot of your paper. Use the same filename as above with the image file type, e.g. IROS2021Rueckert.png.

Upload your files to the

Upload both files to the media library.

Add a new publication

  • use the “Publications/Add new” plugin
  • enter the title, Author, Date of publishing
  • for type=Journal Article, enter volumne, number, doi
  • for type=Inproceedings, enter the booktitle, address of the conference
  • create the bibkey using the button
  • URL/File=https://cps.unileoben.ac.at/
    wp/    AR2018Paraschos.pdf, Article File (replace the author name in the url)

Edit the Post in Elementor

  • bookmarks: select add to your own list
  • select at least one tag
  • image URL=https://cps.unileoben.ac.at/
    wp/AR2018Paraschos    .png (replace the author name in the url)
  • press the create button and you are DONE!


How to use our FRANKA EMIKA Panda

|

Franka World Account

 

Franka Emika Webinterface

To connect to the web interface, use the robot PC. Open the browser and visit the website: https://robot.franka.de/ or via IP 172.16.0.2 . The access data for the web interface are as follows:

Account name: CPSFranka
PW: bs6m6BYftuQjd8p

Getting started with the Guiding Mode

Before you start with the guiding mode you should get to know the status colors of the Franka Panda. You can find them in the picture below. In case of difficulties, the user manual on page 137 provides you with initial assistance.


In guiding mode, motion of the arm follows the corresponding guiding configuartion, which is displayed in the sidebar. The guiding configuration can be changed by pressing the guiding mode button on top of the grip. You can also select the guiding mode in the web interface.

  • Translation: In this guiding mode, the arm can only be moved to change the Cartesian orientation of the hand; its orientation remains as it was before entering the guiding mode.
  • Rotation: In this guiding mode, the arm can only be moved to change the Cartesian orientation of the end effector; its position remains as it was before entering guiding mode. The reference coordinate system for this rotation is the predefined coordinate system of the end effector.
  • Free: The arm can be moved freely, all 7 joints can be moved.
  • User: The user can define the guiding behavior in this guiding mode, meaning that it is possible to define for each Cartesian translation and rotation axis if the arm is moveable or immovable.

Teaching the Robot to Pass Through a Sequence of States

You can register a sequence of robot configurations as target states. These points can be autonomously approached, one after another. 

Activating the Mode ‘Guide Task’

To set up a guide task, press the “+” button next to the tasks in the lower left part of the web interface. Next, you can add either a Cart Motion App or a Joint Motion App. After you have set up the task, you can begin the guidance process.

To start the guidance program, unlock the robot’s joints in the web interface and check that the two switches are off (on the right of the window). Next, open the first app of your task. Now you need to go to the robot and press the two buttons at the end of the end effector. Then move the robot arm to the first control point and press the “o” button on the control panel, repeat this process until you have completed your guidance program. Finally, press the green check button. Now you only have to enter the maximum speed and acceleration for your task and you are done with the first app. Use the same procedure for all apps. You can then return to the computer and start your program after unlocking the safety switch.

ATTENTION: Do not be in the robot’s danger zone at any time while your guidance program is running!



Robot How to Build a USB Controlled Treadmill

|


This post discusses how to develop a low cost treadmill with a closed-loop feedback controller for reinforcement learning experiments.

MATLAB and JAVA code is linked.

Code & Links

The Treadmill

  • Get a standard household treadmill Samples
  • Note: It should work with a DC-Motor, otherwise a different controller is needed!

 

The Controller and the Distance Sensor

  • Pololu Jrk 21v3 USB Motor Controller with Feedback or stronger (max. 28V, 3A)
  • Comes with a Windows Gui to specify the control gains
  • Sharp distance sensor GP2Y0A21, 10 cm – 80 cm or similar
  • USB cable
  • Cable for the distance sensor
  • Power cables for the treadmill
  • Contorller User Guide by Polo

The Matlab Interface

  • Get the java library  build or the developer version, both from Sept 2015 created by E. Rueckert.
  • Run the install script installFTSensor.m (which add the jar to your classpath.txt)
  • Check the testFTSensor.m script which builds on the wrapper class MatlabFTCL5040Sensor (you need to add this file to your path)