The applied machine and deep learning course start on 02.10.2023.
Study the publication [1] for the next work.
Do next
3D complete coverage on quadruped robot for mine inspection tasks.
Prior in SLAM from architectural floor plans.
Reference
[1] Song, Soohwan & Kim, Daekyum & Jo, Sungho. (2020). Online coverage and inspection planning for 3D modelling. Autonomous Robots. 44. 10.1007/s10514-020-09936-7.
190.015 Applied Machine and Deep Learning (5SH IL, WS) [2023 WS]
|
Course Content
In the first week, advanced machine and deep learning methods like multi-layer-perceptrons, convolutional neural networks, variational autoencoder, transformers, simultaneous navigation and mapping approaches, and more will be presented.
These methods can be tested using interactive tools like for example using https://playground.tensorflow.org. To deepen the knowledge, students will answer well-crafted scientific questions using latex handouts alone or in teams of two students in the lecture room.
Additionally, Jupyter notebook files were prepared to implement advanced machine and deep learning approaches without installing any software. For all participants of the course user accounts will be created using our JupyterHub at https://jupyter.cps.unileoben.ac.at. The accounts will remain active till the end of the semester.
MON
TUE
WED
THUR
FRI
02.10.2023
03.10.2023
04.10.2023
05.10.2023
06.10.2023
Topic
Intro to ML Organisation
Neural Networks
Representation Learning
Robot Learning
AML Projects
10
am
:15
Quizz on ML
Quizz on Neural Nets
Introduction to Deep Representation Learning: Core Methods & Coding Examples
Quizz on AML
:30
Introduction to ML
Introduction to Multi-Layer-Perceptrons
Project Topic Presentations
:45
11
am
15 min Break
15 min Break
:15
Statistics, Model Validation, Figures & Evaluations
Handout on Neural Networks using playground.tensorflow
Team Ass., Git Repos & Wiki Instructions
:30
30 min Break
:45
AML Summary
12
pm
30 min Lunch Break
30 min Lunch Break
Curiosity (MLPs), Imagination (Dreamer) and Information (Empowerment)
:15
Quizz on Robotics
:30
Course Organisation & Grading
Intro to Timeseries & Databases
Introduction to Robot Learning
:45
1
pm
15 min Break
15 min Break
Quizz Summary
:15
Python Programming with our JupyterHub
JupyterHub NB on MLPs & Databases
15 min Break
:30
Handout on Robot Learning (Model Learning & RL)
:45
Quizz Summary
Quizz Summary
2
pm
:15
15 min Break
:30
Introduction to Mobile Robotics & SLAM
:45
3
pm
JupyterHub NB on Path Planning
:15
:30
Quizz Summary
:45
Legend
Quizz on ML
Online Quizz using https://tweedback.de
Course Content Presentation
Using google slides, etc.
15 min Break
Breaks to recover or to continue programming
Organisation & Instructions
Using google slides, etc.
Practical Exercise
Using online tools, our JupyterHub, etc.
Latest Research
State-of-the-art research
Prerequisites & If you Miss Course Contents
During the first week, a laptop or tablet will be needed to use the interactive tools and the Jupyter notebooks.
Modern machine learning methods and in particular deep learning methods are entering almost all areas of engineering.
The integrated course enables the students to apply these methods in the application domains of their study.
For this purpose, current problems from the industry are investigated and the possibilities of machine and deep learning methods are tested.
Students gain a deep understanding of method implementations, how data must be prepared, which criteria are relevant for selecting learning methods, and how evaluations must be performed in order to interpret the results in a meaningful way.
Initially, the basics of learning methods are developed in 5-6 lecture units. Then, students select one of the listed industrial problems and work on it alone or as a team (with extended assignments). The project work is accompanied by weekly tutorials with tips and tricks. Finally, the project results are discussed in a written report and presented for a final 10-15min.
Grading is based on the quality of the code, the report, and the final short presentation.
Among others, one of the following industry problems can be chosen:
1. Application and comparison of deep neural networks for steel quality prediction in continuous casting plants with data from the ‘Stahl- und Walzwerk Marienhütte GmbH Graz’.
2. Predictive maintenance of bearing shells using frequency analysis in decision trees and deep neural networks based on acoustic measurement data.
3. Motion analysis and path planning for human-machine interaction in logistics tasks with mobile robots of the Chair of CPS.
4. Autonomous navigation and mapping with RGB-D cameras of the four-legged robot Unitree Go1 for excavation inspection in mining.
31.01.2024 Project Results & Best Practices – Evaluation Results
Learning objectives / qualifications
Implement or independently adapt modern machine learning methods and in particular deep learning methods in Python.
Analyze data of complex industrial problems, process (filter) the data, and divide it into training- and test data sets such that a meaningful interpretation is possible.
Define criteria and metrics to evaluate evaluations and predictions and generate statistics.
Develop, evaluate, and discuss meaningful experiments and evaluations.
Identify and describe assumptions, problems, and ideas for improvement of practical learning problems.
Grading
Continuous assessment: During the lectures and the tutorials 0-20 bonus points can be collected through active participation.
Project assignments: Alone or in small groups (2-3 students) one of the listed projects has to be implemented. A written report in form of a git repository wiki page have to be submitted. – For the implementation (Python Code) 0-40 Points can be obtained. – For the wiki page report, 0-60 Points will be given.
With an overall score of up to 79%, an additional oral performance review MAY (!) also be required if the positive performance record is not clear. In this case, you will be informed as soon as the grades are released. If you have already received a grade via MU online, you will not be invited to another oral performance review.
Bachelor Thesis Student at the Montanuniversität Leoben
Short bio: Gabriel is a Bachelor Student in Mechanical Engineering at Montanuniversität Leoben and, as of March 2023, is writing his Bachelors thesis at the Chair of Cyber-Physical Systems.
When observing animals in nature, navigation and walking seem like medial-side tasks. However, training robots to effectively achieve the same objective is still a challenging problem for roboticists and researchers. We aim to autonomously perform tasks like navigating traffic, avoiding obstacles, finding optimal routes, surveying human hazardous areas, etc with a quadrupedal robot. These tasks are useful in commercial, industrial, and military settings, including self-driving cars, warehouse stacking robots, container transport vehicles in ports, and load-bearing companions for military operations.
For over 20 years today, the SLAM approach has been widely used to achieve autonomous navigation, obstacle avoidance, and path planning objectives. SLAM is a crucial problem in robotics, where a robot navigates through an unknown environment while simultaneously creating a map of it. The SLAM problem is challenging as it requires the robot to estimate its pose (position and orientation) relative to the environment and simultaneously estimate the location of landmarks in the environment.
Some of the most common challenges with SLAM are the accumulation of localization errors over time, inaccurate pose estimation on a map, loop closure, etc. These problems have been partly overcome by using Pose Graphs for localization errors, Extended Kalman filters and Monte Carlos localization for pose estimation.
Quadrupedal robots are well-suited for challenging environments, where the surface conditions are non-uniform, e.g. in off-road environments or in warehouses where stairs or obstacles have to be overcome but have the difficulty of non-uniform dynamic movement which poses additional difficulty for SLAM.
In the context of this thesis, we propose to study the concept of SLAM with its associated algorithms and apply it to a quadrupedal robot (Unitree Go1). Our goal is to provide the robot with certain tasks and commands that it will then have to autonomously execute. For example, navigate rooms, avoid slow-moving objects, follow an object (person), etc.
Tentative Work Plan
To achieve our objective, the following concrete tasks will be focused on:
Study the concept of SLAM as well as its application in quadrupedal robots.
Setup and familiarize with the simulation environment:
Build the robot model (URDF) for the simulation (optional if you wish to use the existing one)
Setup the ROS framework for the simulation (Gazebo, Rviz)
Develop a novel SLAM system for a quadrupedal robot to navigate and map challenging real-world environments:
2D/3D mapping in complex indoor/outdoor environments
Localization using either Monte Carlo or extended Kalman filter
Establish a path-planning algorithm
Intermediate presentation:
Presenting the results of the literature study
Possibility to ask questions about the theoretical background
Detailed planning of the next steps
Implementation:
Simulate the achieved results in a virtual environment (Gazebo, Rviz, etc.)
Real-time testing on Unitree Go1 quadrupedal robot.
Evaluate the performance in various challenging real-world environments, including outdoor terrains, urban environments, and indoor environments with complex structures.
B.Sc. thesis writing.
Research paper writing (optional)
Related Work
[1] Wolfram Burgard, Cyrill Stachniss, Kai Arras, and Maren Bennewitz , ‘SLAM: Simultaneous Localization and Mapping’, http://ais.informatik.uni-freiburg.de/teaching/ss12/robotics/slides/12-slam.pdf
[2] V.Barrile, G. Candela, A. Fotia, ‘Point cloud segmentation using image processing techniques for structural analysis’, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W11, 2019
[3] Łukasz Sobczak , Katarzyna Filus , Adam Domanski and Joanna Domanska, ‘LiDAR Point Cloud Generation for SLAM Algorithm Evaluation’, Sensors 2021, 21, 3313. https://doi.org/10.3390/ s21103313.
Diese Beschreibung ist die Grundlage für die Dateneingabe im MUOnline.
Inhalte in DE und EN.
Wichtig sind die Zuordnungen zu Pflichtfächern, Wahlpflichtfächern.
Wichtig sind die formalen Voraussetzungen.
Die LV Beschreibung wird bei Bedarf an andere Lehrstühle verschickt und muss fehlerfrei (Wochenstunden, ECTS, LV-Typ, Titel, etc.) sein!
Prüfen von Konflikten nach Studienplänen
Terminkonflikte mit anderen Pflicht- und Wahlpflichtfächern aus den zugeordneten Studiengängen (siehe LV Beschreibung oben) müssen unbedingt vermieden werden. Alle Studierenden sollen ihren Studienplan ohne Terminkollisionen umsetzen können.
Ausdrucken der Wochenstundenpläne nach Studiengang im MUOnline
Im MUOnline geht man wie folgt vor:
Nach der Anmeldung klickt man auf den Punkt Studies / Course Offer
Danach wählt man den relevanten Studiengang aus (hier im Bild ist es das Bachelor IDS Studium).
3. Als nächstes wählt man das Studienjahr und klickt auf Semesterplan.
4. Dann wählt man das zugehörige Semester aus und klickt auf das Kalendersymbol. Wichtig LVs die nur im Wintersemester stattfinden haben ungerade Semesterzahlen (1, 3, 5, …).
5. Als nächstes müssen relevante Wochen ausgewählt werden. Dazu zieht man das Semesterstartdatum heran.
Wintersemester: 1.Oktober
Sommersemester: 1.März
Wenn keine Einträge vorhanden sind, muss das Vorjahr ausgewählt werden.
Wichtig ist, mehrere Wochen zu betrachten um möglichst alle Terminkonflikte auszuschließen. Unten sind 2 Beispiele für den Master Maschinenbau im Wintersemester.
Prioritäten und Wahl eines geeigneten Termins
Aus den Wochenplänen erkennt man, dass zB Freitag ein ungünstiger Tag für eine neue LV wäre.
Montag von 10:15 bis 12:00 würde gehen, aber es müssen noch alle anderen Semester und alle weiteren Studiengänge betrachtet werden.
Es bietet sich an die Wochenpläne auszudrucken und auf einem Tisch auszulegen. Dann können Lücken gefunden werden.
Prioritäten:
[Pflicht -> Pflicht] An erster Stelle stehen Konflikte mit anderen Pflichtfächern aus den Studiengängen in denen die neue LV auch ein Pflichtfach darstellt (Pflicht -> in den Klammern).
Empfohlenes Semester
Danach alle weiteren Winter- oder Sommersemester
[Pflicht -> Wahlpflicht] Danach sollen Wahlpflichtfächer berücksichtigt werden.
[Wahlpflicht -> Pflicht] Studiengänge in denen die neue LV ein Wahlpflichtfach darstellt.
[Wahlpflicht -> Wahlpflicht] WPF in Studiengängen in denen die neue LV nur ein Wahlpflichtfach darstellt.
Oben ist die Auflistung einiger Wochenpläne für das Wintersemster, von oben nach unten und von links nach rechts, nach Prioritäten sortiert.
Hier gilt es nun 1-3 Terminslots mit möglichst wenigen Konflikten auszuwählen.
Für diese 1-3 Termine müssen dann Räume gesucht werden.
Abstimmen mit anderen Lehrstühlen:
Gerade die Lehrstühle für Maschinenbau und IDS sollen immer telefonisch oder per Email kontaktiert werden um den Wunschtermin abzusichern.
Install micro-ROS on ESP32
|
Update 17.03.2023: Added instructions for dealing with ESP32-S2 hardware.
This guide should enable you to install ROS 2 Humble on a fresh Ubuntu installation and compile micro-ROS for ESP32 using automated scripts. Additionally, I added the section for running a docker container, so the guide should work on almost any Linux based system which has docker installed.
Section Part II is added, that includes instructions for compiling micro-ROS for ESP32-S2 based devices. The problem here is, that the standard micro-ROS distribution mentioned in the official tutorial does not ship with a current espressif-idf that supports ESP32-S2 boards. It seems like you need an esp-idf above version 4.4.
We will use the FreeRTOS implementation for ESP32. Steps will lead to automatic compilation of provided “app” for the ESP32 and connect it to the ROS humble environment.
Step 1 – Only if using docker is intended
docker run -it --net=host -v /dev:/dev --privileged ros:humble
Step 2: Download micro-ROS and install dependencies
# Source the ROS 2 installation
source /opt/ros/$ROS_DISTRO/setup.bash
# Create a workspace and download the micro-ROS tools
mkdir microros_ws
cd microros_ws
git clone -b $ROS_DISTRO https://github.com/micro-ROS/micro_ros_setup.git src/micro_ros_setup
# Update dependencies using rosdep
sudo apt update && rosdep update
rosdep install --from-paths src --ignore-src -y
# Install pip
sudo apt-get install python3-pip
# Build micro-ROS tools and source them
colcon build
source install/local_setup.bash
Step 3: Create new firmware workspace – this time for platform esp32
Pay attention to use platform “esp32”
# Create step
ros2 run micro_ros_setup create_firmware_ws.sh freertos esp32
Step 4: Configure the firmware to include the application
# Configure step with ping_pong app and serial transport
ros2 run micro_ros_setup configure_firmware.sh ping_pong --transport serial
Step 5: Build the firmware
# Build step
ros2 run micro_ros_setup build_firmware.sh
Step 6: Flash the firmware
Connect esp32 via USB cable or appropriate interface. Most likely all the esp32 dev-boards offer USB connection and will be recognized as ttyUSB0 (USB serial device). In this step, the flash program will find the right port for you automatically.
Caveat ahead: If you are not running the process as root user (which is the recommended approach anyways), you must make sure that you have write permissions on the /dev/ttyUSB0 device. That can be established like follows:
sudo chown $USER /dev/ttyUSB0
Depending on your configuration, it is possible that you need to adapt above command. By running the flash step you will most likely see which device you need to use when the process exits with an error 🙂
# Flash step
ros2 run micro_ros_setup flash_firmware.sh
Step 7: Connect to the device and test the firmware
To make sure everything is working, disconnect the device after the flashing process is done or just reset it via the enable button (EN).
# Download micro-ROS-Agent packages
ros2 run micro_ros_setup create_agent_ws.sh
# Build step
ros2 run micro_ros_setup build_agent.sh
source install/local_setup.bash
# Run a micro-ROS agent
ros2 run micro_ros_agent micro_ros_agent serial --dev [device]
Replace [device] with your serial connection. Again, remember to choose the /dev/ttyUSB* device selected in the flash step. To avoid searching for the correct serial port in the future, you can also find out the discrete device identifier by having a look at this command’s output:
ls /dev/serial/by-id/*
Step 8: Testing the micro-ROS app
At this point, the micro-ROS app is built and flashed and the board is connected to a micro-ROS agent. We now want to check that everything is working.
Open a new command line. We are going to listen to the ping topic with ROS 2 to check whether the micro-ROS Ping Pong node is correctly publishing the expected pings:
At this point, we know that our micro-ROS app is publishing pings. Let’s check if it also answers to someone else’s pings. If this works, it’ll publish a pong.
So, first of all, let’s subscribe with ROS 2 to the pong topic from a new shell (notice that initially we don’t expect to receive any pong, since none has been sent yet):
Also, we expect that, because of having received the fake_ping, the micro-ROS pong publisher will answer with a pong. As a consequence, in the pong subscriber console, we should see the board’s answer to our fake_ping:
In this case, I would advise anybody to use a docker container for the build process, as there is a bunch of dependencies installed in the process that might interfere with your currently running setup.
Step 1
Make a new project directory and cd into it.
cd ~
mkdir microros
cd microros
Step 2
Run a docker container that already includes needed dependencies, for instance a current version of esp-idf.
Select an example project and configure it for esp32-s2. As you can see, it should also be possible to configure the firmware for esp32-s3 and -c3 models. But I have not tested that yet.
cd examples/int32_publisher
# Set target board [esp32|esp32s2|esp32s3|esp32c3]
idf.py set-target esp32s2
Step 5
Use menuconfig to insert the WiFi Credentials and specific settings.
idf.py menuconfig
# Set your micro-ROS configuration and WiFi credentials under micro-ROS Settings
Step 6
Build your micro-ROS application and flash it to your esp32-s2 device. Additionally, start a monitoring process via serial console to see if there are any errors with the device. You will probably see some errors after connecting to the network, as the corresponding micro-ROS agent is not running on the computer yet.
idf.py build
idf.py flash
idf.py monitor
Step 7
Start a micro-ROS agent container to connect to your newly set-up esp node.
Screenshot showing micro-ROS agent on the left and monitor process connected on the right.
In the end, this gives you the tools needed to compile the firmware for a ESP32-S2 based device. After exiting the build-container, the generated files are still available under provided project directory on your computer.
To access code completion and syntax error handling (debugging doesn’t work in PyCharm directly) in PyCharm for ROS2 the paths have to be linked in your project structure. To do so the following steps have to be performed:
Go to Settings -> Project: <Your-Project> -> Project Structure and Click on Add Content Root.
Now add the two directories: /opt/ros/<your-ros-distro>/lib/<python-version>/site-packages and /opt/ros/<your-ros-distro>/local/lib/<python-version>/dist-packages and set them to Resources
Now the code completion and syntax error handling works for ROS2.
Custom Messages & Custom Packages
To resolve import errors and add code completion in PyCharm for ROS2 projects you have to further set some directories as source roots.
First for custom messages (now called interfaces) build your project once with colcon build. Then go to install -> <name-of-msg-package> -> local -> lib -> <python-version> -> dist-packages and mark this folder as Source with the right click drop down menue.
For custom packages you go to install -> <name-of-ros2-package> -> lib -> <python-version> -> site-packages and mark this as Source with the right click drop down menue.
Don’t forget the build your project with colcon build if you change your custom packages/modules so the code is updated in the install directory.
Custom Packages and setup.py
To use custom packages in your ROS2 nodes you need to link them in your setup.py file of your ROS2 package. To do so all packages need to have a __init__.py file depending on your package this file is in general empty. Then you open the setup.py file and add all your packages to the list “packages”. The list should look like this:
In the last decade, MotoGP has taken data analytics and telemetrics to a whole new level which has aided in the development and manufacturing of the motorcycles prototypes and their performance on the track.
However, as in other high-end motorsports competitions such as F1, the technological gap keeps getting smaller and, with so, the real potential advantage is gained during testing by trying to find the optimal setup for that specific track and weather conditions. Due to regulations, time dedicated to testing on track is quite scarce, therefore teams and pilots have to find their way around new prototype setups every week to optimize and extract the best performance of the bike.
The main goal of this project is to develop a Bayesian Optimization algorithm that can aid in the fine tuning of ECU parameters of the motorcycle (fuel injection timing and spark ignition timing) while providing a framework and workflow for this methodology.
The work is done in collaboration with the Chair of Cyber-Physical-Systems at Montanuniversitaet Leoben, and Montan Factory Racing, participant of the VII Edition Motostudent Petrol.
Tentative Work Plan
To achieve our objective, the following concrete tasks will be focused on:
Literature research
Engine and controller model development in Simulink
Bayesian optimization algorithm development
Hardware integration
Testing in practical applications
M.Sc. thesis writing and documentation
Related Work
[R1] Isermann, R., Hafner, M. (2001). Mechatronic Combustion Engines – from Modeling to Optimal Control.
[R2] Schillinger, M., Hartmann, B., Skalecki, P., Meister, M., Nguyen-Tuong, D., & Nelles, O. (2017). Safe active learning and safe Bayesian optimization for tuning a PI-controller. IFAC-PapersOnLine, 50(1), 5967-5972.
[R3] Isermann, R. (2014). Engine modeling and control. Berlin: Springers Berlin Heidelberg, 1017.
[R4] Gerhardt, J., Hönninger, H., & Bischof, H. (1998). A new approach to functional and software structure for engine management systems-BOSCH ME7. SAE transactions, 1173-1184.
Benjamin Schoedinger, B.Sc Master Thesis Student at the Chair of Cyber-Physical-Systems Montanuniversität Leoben Franz-Josef-Straße 18, 8700 Leoben, Austria
Email:
Christoph Andres, B.Sc.
|
Bachelor Thesis Student at the Montanuniversität Leoben
Short bio:
Christoph is a bachelor student in Mechanical Engineering at Montanuniversität Leoben. His fascination for industrial robotics and automation already started at high school, where he worked on a collaborative robotics project during his final thesis.
In June 2023, he finished his bachelor’s thesis at CPS.
Christoph Andres, B.Sc. Bachelor Thesis Student at the Chair of Cyber-Physical-Systems Montanuniversität Leoben Franz-Josef-Straße 18, 8700 Leoben, Austria
