Meeting on the 23th, November 2022

Location: Chair of CPS

Date & Time: 23rd Nov 2022

Participants: Univ.-Prof. Dr. Elmar Rueckert, DI Nikolaus Feith, BSc



  1. Update
  2. Future Steps

Top 1: Update

  • RH8D:
    • Finished the hardware interface, no more communication issues with the left hand.
    • Sample Position controller
  •  Webserver:
    • basics in websockets and js/css/html try outs
    • established connection with ROS2 via rosbridge
    • literature research on related work (shared control, webservers in robotics)
    • search for libraries to display dynamic plots (flot.js or dc.js)

Top 2: Future Steps


Meeting on the 16th, November 2022

Location: Chair of CPS

Date & Time: 5th Oct 2022

Participants: Univ.-Prof. Dr. Elmar Rueckert, DI Nikolaus Feith, BSc



  1. Update
  2. Future Steps

Top 1: Update

  • Literature search on Related work to Online Reinforcement Learning and Preference Learning was started
  •  Hardware interface for RH8D was implemented and needs to be tested now. Some issues with exceeding cycle time of the mainloop were found.

Top 2: Future Steps

  • Start working on webservices and ROS2
  • MP:
    • What do we need for kinestetic teaching, shared control etc.
    • Whiech Parameters are changed and how to display them
  • Start with simulation and not a real robot
  • GNN: for the future reimplementation of toy tasks and research whats new in this field or what is still left out
  •  RH8D: Try fixing the communication error with Vedant’s SDK
  • Webserver/Websockets:
    • What are requirements for the webserver/websocket to use it with ROS2 and shared control

Meeting on the 19th, October 2022

Location: Chair of CPS

Date & Time: 5th Oct 2022

Participants: Univ.-Prof. Dr. Elmar Rueckert, DI Nikolaus Feith, BSc



  1. Update
  2. Future Steps/News

Top 1: Update

  • Since October 12, two presentations were prepared and one was given to the chair on the ROS2 tutorial. The second one deals with the paper “VisuoSpatial Foresigt for Multi-Step, Multi-Task Fabric Manipulation” and will be given next Thursday.
  • Furthermore, a concept for the CPS HUB for the ROS2 repositories was developed and finalized with the other PhD students. 
  • Literature search on Related work to GNN in Motion Planning has been started.
  •  First ROS2 submodule of RH8D robot arm was completed. Data of force sensors at fingertips can be published.

Top 2: Future Steps/News

  • Usage of the desk app in nexttcloud was discussed
  • ROS2: Further discussion on the shared control project, usage of the tablet with android app. Continue working on the RH8D hand.
  • Topics for the Integrated Project, Bachelor’s theses and Master’s theses should be considered. What should be investigated in more detail?
  • Not only on shared control should be further investigated but also GNN.

ROS2 Tutorial

Robot Operating System 2 (ROS2) is an open source middleware for the development of robot applications. This tutorial describes the architecture and the basic components. Furthermore, a comparison to ROS1 is drawn and the application with Python is explained in more detail.

Hier finden sie den Foliensatz zum Vortrag.

Introduction to Python

Why should you consider learning Python

  • Python is the most popular programming language in the world with a popularity of 28%.
  • It is easier to learn than many other programming languages.
  • Python is very readable due to its structure. Thus, bugs can usually be found and fixed quickly.
  • Python can be both procedural and object-oriented, which makes it very versatile.
  • There is a large number of software libraries. On python package index, PyPI, over 400,000 packages can be found. And the most important ones can be downloaded quickly and easily using pip.
  • Python and almost all Python software libraries are available for free on the Internet.

What Python can be used for

  • Data Science is one of the most popular application areas of Python. Here, not only the complete data analysis but also the visualization will be implemented in Python. Generally, software libraries such as Numpy, Matplotlib and Pandas are used for this purpose.
  • Machine learning is another very popular area. Python can be used for supervised, unsupervised and reinforcement learning. Libraries such as TensorFlow, Keras, PyTorch or scikit-learn are used for this purpose.

Furthermore, Python can be used for all kinds of general purpose programs, such as app development, GUI design, web development and in many other areas. In addition, programs with Python interfaces can be automated through a Python program, for example simulation programs.

Practical example of Python in academia

Especially during the time as a student it is good to have a tool with which you can validate your hypotheses or perform the calculations by the computer. Thus, programs can be written with which problems in physics, chemistry or other tasks can be simulated and subsequently the simulation data can be plotted or even videos can be generated.

Important links

Meeting on the 5th, October 2022

Location: Chair of CPS

Date & Time: 5th Oct 2022

Participants: Univ.-Prof. Dr. Elmar Rueckert, DI Nikolaus Feith, BSc



  1.  Update and more detailed definition of the the project
  2. Master thesis GNN for Motion Planning
  3. Next steps

Top 1: Update and more detailed definition of the the project

  • Last week, work continued on the ROS framework. There were problems with the inverse kinematics solver, because the Jaccobian matrix is generated incorrectly by the used Python packages. For the time being we will only work with the Impedance Effort Controller from Franka.
  • Furthermore, the CPS Lecture Repository was created and the CoppeliaSim simulation file and the Python script to operate it were programmed. In addition, a description of how to work with the simulation was written.
  • Further details were discussed in the ROS framework. The framework should work with different simulation programs (CoppeliaSim, IssacSim). The goal is to implement an RL environment, so that an initial solution can be given with the pen on the tablet. This initial solution (trajectory) should be used as motion primitives for the RL. The weights of the MPs are to be learned. Furthermore, it should be possible to manipulate the MP in real time, so that the user can further refine the trajectory in the learning process. The Gym library will be used for the evaluation. For the experiment, the lathe and the milling machine will be used (button press, lever operation, etc.). 
  • A research on related work shall be done, especially if such systems/frameworks have already been developed with ROS2. 
  • Not all sensors, HW devices, robots, etc. need to be implemented by myself, but there need to be a template for the different parts, so every one can extend the frame work.
  • XBox Controller: The Xbox controller is to be used for teleoperation for the robots, as well as the possibility for recording experiments should be implemented. E.g. fixation of a configuration, so that the same experiment can be recorded again and again from the same configuration or the possibility to separate a recording by pressing a button, so that the experiment has to be started only once and not for every run again.

Top 2: Master thesis GNN for Motion Planning

  1. Definition of a master thesis with the topic: GNN for Motion Planning

Top 3: Next Steps

  1. ROS
  2. ROS2
  3. Tablett for shared control (user correction)
  4. Hardware: which hardware is needed to work with the Framework

Meeting on the 28th, September 2022

Meeting on the 13th of August 2021

Location: Chair of CPS

Date & Time: 13th Aug. 2021, 9 am to 9:30 am

Participants: Univ.-Prof. Dr. Elmar Rueckert, Nikolaus Feith, BSc



  1.  Project definition
  2. Update and next steps

Top 1: Project definition

Three main projects:

  1. ROS CPS Framework
  2. Reimplementation GNN for Motion Planning
  3. Literature research

The development of a ROS/ROS2 framework to control the CPS robots.
Furthermore, an interface for the use of a tablet is to be programmed in ROS2. On the tablet trajectories can be designed (movement primitives) which can be changed directly with the pen. For this a simple GUI is needed for the display and manipulation of trajectories as well as the display of the robot in 3D in real time. As well as the use of the CPS glove.
This program is intended to be an application of active learning methods, especially for shared and preference learning.
Finally, a connection will be made to the methods of Probabilistic Inference, see GNN for Motion Planning.
If possible, a comprehensive platform including ROS Mobile should be developed.

Top 2: Update and next steps

  1. Update: 
    1. Last week a Joint Effort Controller was implemented and tested in Gazebo. The application in combination with the inverse solvers of the robotic toolbox was tested, but still leads to errors in the control.
    2. Literature research: Bishop
  2. Next steps:
    1. In the coming week, these errors will be corrected and the tests on the Franka robot arm will start.
    2. Literature research: finish Bishop

How to install Python and PyCharm

This tutorial explains how to install the basic Python environment for the Lecture Maschine Learning. This course requires a Python version >= 3.8 and PyCharm as IDE. In case you are using another operating system you can find some links at the end of this wiki post.

Download and Installation of Python

To program with Python you need a current Python version, which can be downloaded from the following website: Python. Basically every version after 3.8 can be used, quite new versions can still have bugs now and then, furthermore some packages are not yet transferred to the newest version, therefore 3.9 is recommended.

After the download, the program must now be installed. It is important that Python is added to the PATH, otherwise this step must be done manually. This is accomplished by selecting the “Add Python 3.9 to PATH” checkbox. Now Python is installed and the IDE “PyCharm” can be downloaded. 

Installation of PyCharm

To install PyCharm visit the jetbrains website and follow the instructions. We recommend the Professional version, to get the license for it you need to create an account on Jetbrains and log in to the program after the installation. 


How To fix the Robot Hand (Overheat Error)

Signs of an Overheat Error

Two signs occur when the overheating error is present:

1) Only the red lamp on top of the robot hand is light up continuously.
2) The ROS Terminal (running the roslaunch rh8d start_rh8d.launch programm) displays “Overheat Error”. In the picture below the error message for servo #38 is displayed.

Needed Resources

  1.  Download the TSBAdvanced Loader binary from github.
  2.  Download the RH8D Fix folder (so far we have only the fixes for Servo #33, #35 and #38, if you need a different one please talk to Prof. Rückert).

How the fix works

  1. Connect the robot hand via the micro USB connector on the side of the hand to a Windows computer.
  2. Plug in the power supply of the robot hand.
  3. Open a Terminal on the Windows computer.
  4. Move to the directory were you unziped the “RH8D” files.
  5. Check the COM connection to the robot hand in the devices manager (i.e. COM4).
  6. Now enter the bridge mode with the terminal command : “tsbloader_adv.exe -port=[the Virtual Serial port number/id you determined in point 1] -seederos=bron” (replace the [the Virtual Serial port number/id you determined in point 1] with i.e. COM4.
  7.  Depending on the servo motor, the corresponding pwd must now be used. Servo #31 = A; Servo #32 = B; … Servo #38 = H.
  8. To reset the faulty temperature sensor of the corresponding servo motor use the following command: “tsbloader_adv.exe -port=[YOUR COMM PORT] -prewait=2000 -pwd=[SERVO-PWD] -eop=ewv -efile=[RESET FILE FOR THE SERVO] -baud=33333”
  9. If there are more than one errors, fix only one at a time!
  10.  After completion use the command: “tsbloader_adv.exe -port=[the Virtual Serial port number/id you determined in point 1] -seederos=broff” to leave the bridge mode.
  11. Afterwards reboot the hand (replug the power supply and disconnect the micro USB cable). After booting all three lamps should light again. Otherwise contact the support of seed robotics via email:


CoppeliaSim Tutorial

This tutorial describes the usage of the program CoppeliaSim. In particular, the use of the software in the context of the course “Cyber-Physical-Systems” is discussed.

Download and setup of the program

CoppeliaSim can be downloaded from this website. Furthermore, the Python package “msgpack” must be downloaded via pip. Depending on your operating system, different steps are required after the installation.

Windows 10

For Windows 10, no further installation steps are necessary. However, to use the B0-based remote API, some .dll files must be available in the working directory. These can be found in the installation folder of CoppeliaSim. To shorten the search for the files, you can find all the required files in the GitHub project linked below.

Ubuntu 20.04

Before CoppeliaSim can be started, dependencies for the BlueZero API have to be installed. To accomplish that follow the instructions bellow.

Bildschirmfoto von 2021-11-16 16-53-35

In contrast to Windows, CoppeliaSim must be started from the terminal on Ubuntu. To do this, right-click on the unpacked folder and select the option “Open in Terminal”. Then enter the following command “./”. After confirming with the Enter key, CoppeliaSim starts. To use the B0-based remote API, the file “” must be available in the working directory. For some simulations, additional files must be added in the same directory, these can also be found in the GitHub project linked below.

CoppeliaSim and B0-based remote API - Python Client

As described above, depending on the operating system, .dll and .so files need to be added to the workspace. Besides the operating system specific files, the Python scripts “” and “” must be present in the working directory. These files can be found in the installation folder or in the GitHub project. For the task of the course, the following two applications are most important: actuation, sensing. Sample code for actuation and sensing can also be found in the GitHub project. In the following section, their application is briefly discussed. 


Using the API, Two movement modes are implemented in the provided scene “scene_with_pandas.ttt”. These are used with the method “simxCallScriptFunction” in Python because they are programmed as a function in the simulation file. The following modes are available:

  • – pts: In this mode, the angular positions of the joints and the corresponding time are passed to the simulation as a list (all intermediate points are interpolated). This mode is important for control tasks and if the inverse and forward kinematics have been developed by the user.
  • mov: In the “mov”-mode, the positions and speeds of intermediate points are transferred to the simulation. The inverse kinematics of Reflexxes Motion Library type II or IV is used in this mode.


At the beginning, the object handles of the observed objects must be determined, this is done with the help of the method “simxGetObjectHandle”. To execute this method you need a so-called topic, more about this down below. Since “simxGetObjectHandle” only needs to be executed once, and only at the beginning or before the simulation, the topic “simxServiceCall” is used.

Afterwards, the joint angles or joint positions can be streamed with the help of the method “simxGetJointPosition” and the position of the end effector with “simxGetObjectPosition”. To achieve this, a callback function is needed for each angle or for the coordinates (one for each xyz triplet). These callback functions are called cyclically and can be used, for example, to store the angles in an array. Finally, it must be noted that the sensor data should always be saved with their time, otherwise no meaningful calculations or diagrams can be made.

As with the actuation, sample codes are available in the GitHub project.