LEGO EV3 for Robotic Tasks

We have five EV3 sets and use them for studying robot control, motion planning and visual navigation from depth images. 

We use our GitHup LEGO Python project for our developments. 

Tactile Sensing

Several special purpose sensors including depth image cameras (shown in the center in the image), IMUs, accelerometers, gyroscopes, sonic sensors (two are shown in the image), etc. can be connected to the EV3 brick. 

The EV3 systems can be used to explore neural sensor fusion approaches, embedded computing implementations and classical mobile robotics tasks.  

• Details to our LEGO Python GitHub project

Videos

https://vimeo.com/501651310https://vimeo.com/374166607

Human-inspired, Adult-size Dexterous Robot Hand

We use a adult-sized robot hand for learning grasping and object manipulation skills. The hand is mounted on our FRANKA EMIKA Panda robot. 

The hand has 19 degrees-of-freedom and uses 8 smart actuators for precise control (actuators contained inside the unit).

Under actuated design aims to provide the right balance between fine control and conformance to the shape of the objects.

Tactile Sensing

All actuators provide real time control and feedback of position, speed and current measurement (with direction), enabling inference of applied force.

Additional data including actuator temperature, (over)load status and PWM, a Palm ToF Distance sensor and optional Capacitive pads at the back of the palm complete the sensor array.

We also have five 3-axis force-torque sensors (FTS) (shown in the image) attached to each finger tip. The FTS measure contact force and shear forces with a resolution of 1mN / 0.1g.

• E-Manual of the RH8 hand.
• ROS Interface description of the RH8 hand.
• E-Manual of the 3Axis F/T Sensors 
• Details to our FRANKA ROS GitHub project.
• Details on the robot hardware can be found on the SEED Robotics page.

Videos

• Research videos using the robot will be presented here. 

Publications

2020

Xue, H.; Boettger, S.; Rottmann, N.; Pandya, H.; Bruder, R.; Neumann, G.; Schweikard, A.; Rueckert, E. Sample-Efficient Covariance Matrix Adaptation Evolutional Strategy via Simulated Rollouts in Neural Networks Proceedings Article In: International Conference on Advances in Signal Processing and Artificial Intelligence (ASPAI’ 2020), 2020. Links | BibTeX @inproceedings{Xue2020, title = {Sample-Efficient Covariance Matrix Adaptation Evolutional Strategy via Simulated Rollouts in Neural Networks}, author = {H. Xue and S. Boettger and N. Rottmann and H. Pandya and R. Bruder and G. Neumann and A. Schweikard and E. Rueckert}, url = {https://cps.unileoben.ac.at/wp/ASPAI2020Xue.pdf, Article File}, year = {2020}, date = {2020-06-30}, booktitle = {International Conference on Advances in Signal Processing and Artificial Intelligence (ASPAI’ 2020)}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Close Article File Close

We are developing a repository for real-time control of the FRANKA EMIKA Panda 7-dof robot arm.

Our project is based on ROS and allows to teleoperate the robot arm in real-time using motion tracking data provided by OptiTrack’s Motive software. 

GitHub Project and Links

• https://github.com/ai-lab-science/Franka-ROS-TRAIN
• Details on the robot hardware
• Or on the developer site on world.franka.de
• ROS Gazebo Tutorial

Videos

• Research videos using the robot will be presented here. 

Publications

2020

Xue, H.; Boettger, S.; Rottmann, N.; Pandya, H.; Bruder, R.; Neumann, G.; Schweikard, A.; Rueckert, E. Sample-Efficient Covariance Matrix Adaptation Evolutional Strategy via Simulated Rollouts in Neural Networks Proceedings Article In: International Conference on Advances in Signal Processing and Artificial Intelligence (ASPAI’ 2020), 2020. Links | BibTeX @inproceedings{Xue2020, title = {Sample-Efficient Covariance Matrix Adaptation Evolutional Strategy via Simulated Rollouts in Neural Networks}, author = {H. Xue and S. Boettger and N. Rottmann and H. Pandya and R. Bruder and G. Neumann and A. Schweikard and E. Rueckert}, url = {https://cps.unileoben.ac.at/wp/ASPAI2020Xue.pdf, Article File}, year = {2020}, date = {2020-06-30}, booktitle = {International Conference on Advances in Signal Processing and Artificial Intelligence (ASPAI’ 2020)}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Close Article File Close

FRANKA EMIKA’s Panda robot arm is a complient, light-weight robot arm with seven degrees-of-freedom. 

We use the C++ libfranka library in our own ROS project for learning complex manipulation skills. 

Links

• Details to our FRANKA ROS GitHub project
• Details on the robot hardware can be found on the FRANKA EMIKA page
• Or on the developer site on world.franka.de

Videos

• Research videos using the robot will be presented here. 

Publications

2020

Xue, H.; Boettger, S.; Rottmann, N.; Pandya, H.; Bruder, R.; Neumann, G.; Schweikard, A.; Rueckert, E. Sample-Efficient Covariance Matrix Adaptation Evolutional Strategy via Simulated Rollouts in Neural Networks Proceedings Article In: International Conference on Advances in Signal Processing and Artificial Intelligence (ASPAI’ 2020), 2020. Links | BibTeX @inproceedings{Xue2020, title = {Sample-Efficient Covariance Matrix Adaptation Evolutional Strategy via Simulated Rollouts in Neural Networks}, author = {H. Xue and S. Boettger and N. Rottmann and H. Pandya and R. Bruder and G. Neumann and A. Schweikard and E. Rueckert}, url = {https://cps.unileoben.ac.at/wp/ASPAI2020Xue.pdf, Article File}, year = {2020}, date = {2020-06-30}, booktitle = {International Conference on Advances in Signal Processing and Artificial Intelligence (ASPAI’ 2020)}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Close Article File Close

Nils Rottmann, M.Sc. has developed a tutorial on using ROS and Gazebo. 

This tutorial was used in our humanoid robotics and machine learning lectures. 

GitHub Code & Links

• https://github.com/NRottmann/ROS_Gazebo_Tutorial

Weitere Links und Tutorials

• Python Videolectures
• Python Online Tutorials

Dieses open-source Projekt enthält Tools und Demos für die Python-Entwicklung mit den Lego Mindstorms EV3 und EV3Dev Bricks. 

Die Inhalte sind verständlich aufbereitet und wir haben zahlreiche Tutorials und Aufgaben für Schüler*innen erstellt. 

GitHub Code & Links

• https://github.com/ai-lab-science/LEGORoboticsPython
• Dieses Repository ist Teil unseres Lego Robotik Schüler*innen Projekts.  

Details to the Software Development

Dieser Einführungsvortrag beschreibt die grundlegenden Schritte um einen LEGO Roboter zu bauen und mit Python zu programmieren. 

Weitere Links und Tutorials

• Wiki: https://github.com/rueckert/LEGORoboticsPython/wiki
• Installation und Dokumentation von Micropython.

Sensor gloves are gaining importance in tracking hand and finger movements in virtual reality applications as well as in scientific research. In this project, we developed  a low-budget, yet accurate sensor glove system that uses flex sensors for fast and efficient motion tracking. 

The contributions are ROS Interfaces, simulation models as well as motion modeling approaches. 

GitHub Code & Links

• https://github.com/ai-lab-science/SensorGloves
• The software development is part of the TRAIN project. 
• The development is based on prior sensor glove designs.

Details to the Software Development

The figure shows a simplified schematic diagram of the system architecture for our sensor glove design:

(a) Glove layout with sensor placements, the orange fields denote the flex sensors, while the IMU is marked as a green rectangle,

(b) Circuit board which is wired with the sensor glove, has 10 voltage dividers for reading each flex sensor connected to ADC pins of the microcontoller ESP32-S2 and the IMU is connected to I2C pins,

(c) The ESP32-S2 sends the raw data via WiFi as ROS messages to the computer, which allows a real-time visualization in Unity or Gazebo,

(d) Post-processing of the recorded data, e.g. learning probabilistic movement models and searching for similarities.

Publications

A research publication by Robin Denz, Rabia Demirci, M. Ege Cansev, Adna Bliek, Philipp Beckerle, Elmar Rueckert and Nils Rottmann is currently under review. 

Univ.-Prof. Dr. Elmar Rueckert is organizing this research seminar. Topics include research in AI, machine and deep learning, robotics, cyber-physical-systems and process informatics. 

Language:
English only

Are you an undergraduate,  graduate, or doctoral student and want to learn more about AI?

This course will give you the opportunity to listen to research presentations of latest achievements. The target audience are non-experts. Thus no prior knowledge in AI is required.

To get the ECTS credits, you will select a research paper, read it and present it within the research seminar (10-15 min presentation). Instead of selecting a paper of our list, you can also suggest a paper. This suggestion has to be discussed with Univ.-Prof. Dr. Elmar Rueckert first.

After the presentation, the paper is discussed for 10-15 min.

Further, external presenters that  are leading researchers in AI will be invited. External speakers will present their research in 30-45 min, followed by a 15 min discussion.

Location & Time

• Location: HS Kuppelwieser 
• Dates: Thursdays 12:00-14:00, with some exceptions, see here the list of  the dates.

List of Talks and Dates

The specified time windows do not include discussions. 

• 26.04.24 11:15-12:45 HS TPT 
  
  • Univ.-Prof. Dr. Elmar Rueckert, Introductory Slides.
    
  • Ph.D. research talk: Raphael Scharf on CFD Simulation of Textured Surfaces in Lubricated Sliding Contacts. Slides.
• 03.05.24 11:15-12:45 HS TPT 
  • Ph.D. research talk: Vedant Dave on Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training (ICRA 2024).
• 17.05.24 11:15-12:45 HS TPT 
  • Ph.D. research talk: Sahar Keshavarz on Real-Time Autonomous Decision-Making in Well Construction.
  • Ph.D. research talk: Marcel Czipin on Data-Driven Approaches for Process Improvement in Wire-Arc Additive Manufacturing.
• 24.05.24 11:15-12:45 HS TPT 
  • CANCELED due to the LANGE Nacht der FORSCHUNG 2024
    
• 07.06.24 11:15-12:45 HS TPT 
  • Ph.D. research talk: Dipl.-Ing. Karin Ungerer on Vision-based Vibration Measurements.
  • Ph.D. research talk: Dipl.-Ing. Nikolaus Feith on Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvement (UR 2024).
• 14.06.24 11:15-12:45 HS TPT 
  •  
• 21.06.24 11:15-12:45 HS TPT 
  • Ph.D. research talk: Dipl.-Ing. Melanie Neubauer on Semi-Autonomous Fast Object Segmentation and Tracking Tool for Industrial Applications (UR 2024).
  • Ph.D. research talk: Dipl.-Ing. Fotios Lygerakis on M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation (UR 2024).
  • Ph.D. research talk: Dipl.-Ing. Nikolaus Feith on Advancing Interactive Robot Learning: A User Interface Leveraging Mixed Reality and Dual Quaternions (UR 2024).
• Template to be reused
  • FREE  
  • M.Sc. thesis: XX on XX.
  • B.Sc. thesis: XX on XX.
  • Internship Project: XX on XX.
  • Ph.D. research talk: XX on XX.

Some Research Paper Candidates

• Predictive Whittle Networks for Time Series by Yu et al. (UAI 2022)
• Natural Gradient Shared Control by Oh, Wu, Toussaint and Mainprice
• Learning to solve sequential physical reasoning problems from a scene image by Driess, Ha and Toussaint (IJRR 2021)
• Deep Visual Constraints: Neural Implicit Models for Manipulation
  Planning from Visual Input by Ha, Driess and  Toussaint
• AW-Opt: Learning Robotic Skills with Imitation and
  Reinforcement at Scale by Lu et al. (CoRL 2021)
• Decision Transformer: Reinforcement Learning via Sequence Modeling (NeurIPS 2021) by L. Chen et al
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning (ICML 2020) by A. Srinivas et al
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble (CoRL 2021) by  S. Lee et al
• Offline Meta-Reinforcement Learning with Online Self-Supervision by Pong et al. (ICML 2022)
• Information is Power: Intrinsic Control via Information Capture by Rhinehart et al. (NeurIPS 2021)
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks by Finn et al.
• Event-based Asynchronous Sparse Convolutional Networks by Messikommer et al.
• Gaussian-binary Restricted Boltzmann Machines on Modeling Natural Image Statistics by Wang et al.
• A Spiking Neural Network Model of Depth from Defocus for Event-based Neuromorphic Vision by Haessig et al.
• Dream to Control: Learning Behaviors by Latent Imagination by Hafner et al.

Univ.-Prof. Dr. Elmar Rueckert was teaching this course at the University of Luebeck in 2018, 2019 and 2020.

Teaching Assistant:
Nils Rottmann, M.Sc.

Language:
English and German

Course Details

On this page you can find short videos which explain the exercise 02 given in the lecture Humanoid Robotics.

https://youtu.be/XMdS8-NnqbMhttps://youtu.be/HSe6kLG5Aywhttps://youtu.be/Pm7O7p7UZwMhttps://youtu.be/x88o7tUJNfo

Univ.-Prof. Dr. Elmar Rueckert was teaching this course at the Technical University Graz in the winter semester in 2012/13 and in 2013/14.

Language:
German only

Link to the university’s course page

Link to the course in the TUG online system.

Course Details

• Elementare Datenstrukturen (Felder, Stapel, Schlange).
• Asymptotische Laufzeitanalyse von Programmen (O-Notation).
• Sortierverfahren (Einfügen, Auswahl, Quicksort, Mergesort, Heapsort, Fachverteilung, i-größte Zahl, Randomisierung, untere Laufzeitschranken).
• Gestreute Speicherung (Hashing; Überläuferlisten, offene Adressierung, Hashfunktionen).
• Suchmethoden (sequentiell, binär, interpolativ, quadratische Binärsuche).
• Baumstrukturen (Binärbäume, (a-b)-Bäume, amortisierte Umstrukturierungskosten, optimale Suchbäume).
• Dynamische Datenverwaltung (Wörterbuchproblem, Warteschlangenproblem, Union-Find Problem).
• Algorithmische Techniken (Inkrementelles Einfügen, Elimination, Divide & Conquer, dynamisches Programmieren, Randomisierung).

Mit bis zu 390 Teilnehmern*innen pro Vorlesung.

Literature

• Cormen, Leiserson, Rivest: Introduction to Algorithms, MIT Press, London, 1990.