1

CPS 5-Finger Robot Hand 2024

|

Self-made, dexterous 5-finger hand

We use a adult-sized robot hand for learning grasping and object manipulation skills. The hand is mounted on our FRANKA EMIKA Panda robot. 

The hand has 19 degrees-of-freedom and uses 8 smart actuators for precise control (actuators contained inside the unit).

Under actuated design aims to provide the right balance between fine control and conformance to the shape of the objects.

Webserver GUI for Control

Within a student project, an ESP32 based web-server was developed for controlling the hand, see the git repository. 

Videos

  • Research videos using the robot will be presented here. 

 

Publications

Sorry, no publications matched your criteria.



190.015 Applied Machine and Deep Learning (5SH IL, WS)

|

Course Content

In the first week, advanced machine and deep learning methods like multi-layer-perceptrons, convolutional neural networks, variational autoencoder, transformers, simultaneous navigation and mapping approaches, and more will be presented.

These methods can be tested using interactive tools like for example using   https://playground.tensorflow.org. To deepen the knowledge, students will answer well-crafted scientific questions using latex handouts alone or in teams of two students in the lecture room. 

Additionally, Jupyter notebook files were prepared to implement advanced machine and deep learning approaches without installing any software. For all participants of the course user accounts will be created using our JupyterHub at https://jupyter.cps.unileoben.ac.at. The accounts will remain active till the end of the semester. 

Prerequisites & If you Miss Course Contents

During the first week, a laptop or tablet will be needed to use the interactive tools and the Jupyter notebooks. 

Webex Online Sessions of the 1st Week

Find here the link to the online stream during the first week in October: https://unileoben.webex.com/unileoben/j.php?MTID=m5492385776dd885ca5dde72e52563c61

When you miss some course contens

If you miss some course contents due to overlapping events, you can watch recordings of the sessions online. All recordings will be hosted via Moodle at https://moodle.unileoben.ac.at/course/view.php?id=3082.

 

Course Description

Modern machine learning methods and in particular deep learning methods are entering almost all areas of engineering. 

The integrated course enables the students to apply these methods in the application domains of their study.

For this purpose, current problems from the industry are investigated and the possibilities of machine and deep learning methods are tested.

Students gain a deep understanding of method implementations, how data must be prepared, which criteria are relevant for selecting learning methods, and how evaluations must be performed in order to interpret the results in a meaningful way.

Initially, the basics of learning methods are developed in 5-6 lecture units. Then, students select one of the listed industrial problems and work on it alone or as a team (with extended assignments). The project work is accompanied by weekly tutorials with tips and tricks. Finally, the project results are discussed in a written report and presented for a final 10-15min.

Grading is based on the quality of the code, the report, and the final short presentation.

Among others, one of the following industry problems can be chosen:

1. Application and comparison of deep neural networks for steel quality prediction in continuous casting plants with data from the ‘Stahl- und Walzwerk Marienhütte GmbH Graz’.

2. Predictive maintenance of bearing shells using frequency analysis in decision trees and deep neural networks based on acoustic measurement data.

3. Motion analysis and path planning for human-machine interaction in logistics tasks with mobile robots of the Chair of CPS.

4. Autonomous navigation and mapping with RGB-D cameras of the four-legged robot Unitree Go1 for excavation inspection in mining.

The project list is continuously extended.

Links and Resources

Location & Time

  • Location: HS 3 Studienzentrum
  • Dates: 01.10.2024 – 07.10.2024, see the course schedule above.
  • Location: Digital Science Center (Roseggerstraße 11, 8700 Leoben)
  • Date: 22.01.2025, 10:15 – 15:15, final presentation

 

Kickoff meeting of project

All meetings will be conducted at CPS chair. The time please refer to the email, contact us if reschedule is needed.

Previous Knowledge Expected

Formal Prerequisite: Basic Python programming skills, Fundamentals of Statistics.

Recommended Prerequisites: Introduction to Machine Learning (“190.012” and “190.013”).

Slides

Learning objectives / qualifications

  • Implement or independently adapt modern machine learning methods and in particular deep learning methods in Python.
  • Analyze data of complex industrial problems, process (filter) the data, and divide it into training- and test data sets such that a meaningful interpretation is possible.
  • Define criteria and metrics to evaluate evaluations and predictions and generate statistics.
  • Develop, evaluate, and discuss meaningful experiments and evaluations.
  • Identify and describe assumptions, problems, and ideas for improvement of practical learning problems.

Grading

Continuous assessment: During the lectures and the tutorials 0-20 bonus points can be collected through active participation.

Project assignments: Alone or in small groups (2-3 students) one of the listed projects has to be implemented. A written report in form of a git repository wiki page have to be submitted.
– For the implementation (Python Code) 0-40 Points can be obtained.
– For the wiki page report, 0-60 Points will be given.

Grading scheme: 0-49,9 Points (5), 50-65,9 Points (4), 66-79 Points (3), 80-91 Points (2), 92-100 Points (1).

With an overall score of up to 79%, an additional oral performance review MAY (!) also be required if the positive performance record is not clear. In this case, you will be informed as soon as the grades are released. If you have already received a grade via MU online, you will not be invited to another oral performance review.

Literature

Machine Learning and Data-modelling:

– Rueckert Elmar 2022. An Introduction to Probabilistic Machine Learning, https://cloud.cps.unileoben.ac.at/index.php/s/iDztK2ByLCLxWZA

– James-A. Goulet 2020. Probabilistic Machine Learning for Civil Engineers. MIT Press.

– Bishop 2006. Pattern Recognition and Machine Learning, Springer.

Learning method Programming in Python:

– Sebastian Raschka, YuxiH. Liu and Vahid Mirjalili 2022. Machine Learning with PyTorch and Scikit- Learn. Packt Publishing Ltd, UK.

– Matthieu Deru and Alassane Ndiaye 2020. Deep Learning mit TensorFlow, Keras und TensorFlow.js., Rheinwerk-verlag, DE. 

Problem specific Literature:

– B. Siciliano, L.Sciavicco 2009. Robotics: Modelling, Planning and Control, Springer.

– Kevin M. Lynch and FrankC. Park 2017. MODERN ROBOTICS, MECHANICS, PLANNING, AND CONTROL, Cambridge University Press.

– E.T. Turkogan 1996. Fundamentals of Steelmaking. Maney Publishing,UK.



190.018 Introduction to Machine Learning (4SH VU, SS)

|

This course is based on the Machine Learning book by Univ.-Prof. Dr. Elmar Rueckert. 

It is written for experienced undergraduates or for first
semester graduate students.

This lecture with integrated exercises provides the basic knowledge for the application of modern machine learning methods. It includes an introduction to the basics of data modeling and probability theory. Classical probabilistic linear and non-linear regression methods are derived and discussed using practical examples.

Links and Resources

Location & Time

Lecture

 

Exercise

 

Assignments

  Points Presentation Submission Results Discussion
Assignment I 3 21.03 04.04 11.04
Assignment II 5 04.04 11.04 02.05
Assignment III 10 11.04 02.05 16.05
Assignment IV 15 02.05 30.05 20.06
Assignment V 20 16.05 13.06 20.06
  53      

Slides

Course Topics

  1. Introduction to Machine Learning (Data and modelling fundamentals)
  2. Introduction to Probability Theory (Statistics refresher, Bayes Theorem, Common Probability distributions, Gaussian Calculus).
  3. Linear Probabilistic Regression (Linear models, Maximum Likelihood, Bayes & Logistic Regression).
  4. Nonlinear Probabilistic Regression (Radial basis function networks, Gaussian Processes, Recent research results in Robotic Movement Primitives, Hierarchical Bayesian & Mixture Models).
  5. Probabilistic Inference for Time Series (Time series data, basis function models, learning).

Learning objectives / qualifications

  • Students get a comprehensive understanding of basic probability theory concepts and methods.
  • Students learn to analyze the challenges in a task and to identify promising machine learning approaches.
  • Students will understand the difference between deterministic and probabilistic algorithms and can define underlying assumptions and requirements.
  • Students understand and can apply advanced regression, inference and optimization techniques to real world problems.
  • Students know how to analyze the models’ results, improve the model parameters and can interpret the model predictions and their relevance.
  • Students understand how the basic concepts are used in current state-of-the-art research in robot movement primitive learning and in neural planning.

Grading

The course will be graded based on a written exam (100 Points). 50% of all questions need to be answered correctly to be positive. The exam will take place in the classroom, or online, depending on the current university regulations.

In addition, up to 10 bonus points obtained in regular quiz sessions in the classroom, and 20% of the achieved points of the Machine Learning Lab will be added to your exam result. Note that bonus points can only be obtained when attending the lectures in person. 

Grading scheme: 0-49.9Pts (5), 50-65.9Pts (4), 66-79Pts (3), 80-91Pts (2), 92-100Pts (1).

Forthcoming exam dates are:

  • XX.06.2025 at 13:15 HS 1 Studienzentrum
  • XX.10.2025 at 13:15 – 14:45  (location not fixed)
  • More dates upon request via email to cps@unileoben.ac.at (send your request one month in advance to the desired exam date).

Literature

  • The Probabilistic Machine Learning book by Univ.-Prof. Dr. Elmar Rueckert. 
  • James-A. Goulet. Probabilistic Machine Learning for Civil Engineers. ISBN 978-0-262-53870-1.
  • Daphne Koller, Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. ISBN 978-0-262-01319-2
  • Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer (2006). ISBN 978-0-387-31073-2.
  • David Barber. Bayesian Reasoning and Machine Learning, Cambridge University Press (2012). ISBN 978-0-521-51814-7.
  • Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. ISBN 978-0-262-01802-9

Note that all books are available at our library or at the chair of CPS. 



Geislinger GmbH

|

Laufende Projekte, Bachelor- und Masterarbeiten

  • Automatisierungslösungen in der Produktion


1 Senior Researcher / Tenure Track Professorship Option, 2406WPE

|

One vacant position for a full-time Senior Scientist (m/f/d) at the Chair of Cyber-Physical-Systems in the Department Product Engineering from the earliest possible date in a 3-year fixed-term employment contract; with the option of a qualification position after positive evaluation according to §99 para. 5 (UG 2002), equivalent to a tenure track assistant professorship.

Salary Group B1 according to the Uni-KV, monthly minimum salary excl. Szlg.: € 4.752,30 for 40 hours per week (14 x per year).

We are looking for a researcher with high personal motivation for scientific excellence and integrity, with the ability to solve problems and enjoy working in research teams in an interdisciplinary and internationally oriented environment.

Core Responsibilities

  • Independent teaching in the areas of digitalization / data modelling / programming, artificial intelligence/machine learning and robotics.
  • Extension and active participation in the group’s research on topics such as Computer Vision, Deep Learning, Large Language Models, Machine Learning, Reinforcement Learning, Smart Devices, Tactile Learning, Autonomous Systems, Mixed Reality, Autonomous Navigation, Human-Robot Interaction, Interactive Learning, Manipulation, Probabilistic Inference, Robot Learning, Simulation, Spiking Neural Networks and Transfer Learning, demonstrated by publications at international conferences (e.g. AAAI, AISTATS, CoRL, ICML, ICRA, IJCAI, IROS, NeurIPS, RSS) and in renowned journals (e.g. AURO, IJRR, JMLR, MLJ, Neural Computation, SciReps, TRo).
  • Leading project management tasks and applying for funding to support and further develop the chair’s research priorities.
  • Promotion of interdisciplinary collaboration and publication of research results in high-ranking scientific journals and conferences.

Requirements

  • Degree in computer science, physics, telematics, electrical engineering, mechanics, robotics,
    mathematics or related studies with a doctorate in a relevant subject
  • Experience in at least one of the following areas or related topics: Machine Learning, Neural Networks,
    Robot Learning or Learning Sensor Systems.
  • Willingness and ability to co-supervise scientific work and to publish research results.
  • Programming experience in one of the following languages: C, C++, C#, JAVA, Python or similar.
  • Ability to work in a team, sociability, self-motivation and reliability in a growing team.
  • Good command of English and willingness to travel for research and technical presentations.

Desired Additional Qualifications

Several years of professional experience in the above-mentioned subject areas or experience as a postdoc or research group leader in international teams.

Application Documents Required:

A complete application includes a (1) detailed curriculum vitae, (2) a letter of motivation with a reference to the desired research and teaching field from the above-mentioned subject areas, (3) two letters of recommendation, (4) all relevant certificates of previous education for bachelor’s, master’s and doctoral studies, (5) name, email and telephone number of two further references for contact, (6) previously published or submitted publications, the doctoral thesis and potential patents in the form of links integrated into a PDF file.

We Offer:

The Senior Scientist will be employed for a fixed term of three years. After one year, the position will be evaluated with the possibility of extending the position or extending the position to a career position in accordance with §99 para. 5 (UG 2002). 

We offer a varied and independent job. A team-oriented working atmosphere, intensive cooperation with project partners and involvement in teaching offer ideal opportunities for professional and personal development. The Montanuniversität promotes career paths and offers excellent conditions for social diversity in a contemporary working environment.

For further information, please contact cps@unileoben.ac.at.

Reference ID: 2406WPE Job portal of the university
Application deadline: 08.08.2024

The Montanuniversitaet Leoben intends to increase the number of women on its faculty and therefore specifically invites applications by women. Among equally qualified applicants women will receive preferential consideration. Scientific experience demonstrated through publications in international conferences and journals on machine learning, neural networks, robotics, or embedded systems. Good team-leading skills and the ambition to supervise doctoral students. Experience in obtaining external funding and in collaborating with industrial partners is advantageous but not a requirement. Excellent English skills and willingness to travel for research and to give technical presentations are required.



190.021 Einführung in die Datenmodellierung (4ECTS VU, WS)

|

Note, this course is held in German, for English-speaking students there is an exercise group (Group E) in which the lecture part is repeated at the beginning of each unit.  

A course description in English can be found via MUOnline. 

Dieser Kurs wird durch ein online Buch zur Datenmodellierung unterstützt. Dieses Buch ist in der Entstehungsphase und wird während des Semesters mit Inhalten befüllt. 

Kursbeschreibung

Die Datenmodellierung spielt eine wesentliche Rolle in modernen Unternehmen. Unzählige Prozessdaten werden gespeichert und zur Prozessoptimierung, zur Qualitätssicherung oder zum Verbessern der Arbeitssicherheit verwendet.

Aber um welche Daten handelt es sich dabei, wie werden die Daten gespeichert und verarbeitet, und wie können damit die obigen Ziele, z.b. durch KI-Methoden erreicht werden? Diese Fragen werden in dieser Lehrveranstaltung behandelt. 

Ziele der Lehrveranstaltung (LV)

Das Ziel der LV ist die Vermittlung von Kompetenzen zur Datenmodellierung und nicht die Anwendung spezieller Tools.

Um das zu erreichen werden Konzepte (z.B. was sind Datenstörungen, wie erkennt man sie und wie können sie behoben werden) vermittelt und an Beispielen im HS interaktiv und eigenständig erprobt. Es ist keine Datenbankenvorlesung.  

Erworbene Kompetenzen

  • Grundlagen der Datenmodellierung kennen. Dazu gehören die unterschiedlichen Daten (quantitative und kategoriale Variablen) und Aufnahme/Modellierungsarten (z.B. Exceltabellen, Datenbanken, Arrays, etc.), Diskretisierungsgrundlagen (Nyquist-Shannon Theorem) und Informationstheorie-Grundlagen (Entropie der Daten).
  • Die Notwendigkeit von Kenntnissen zur Datenmodellierung durch Expertenvorträge aus der deutschsprachigen Industrie oder von Experten der MUL erkennen. Eingeladene Experten stellen ihre Daten, Modellierungsverfahren und die nötigen Kenntnisse der Mitarbeiter vor.  
  • Korrelationen erkennen und für Vorhersagen nützen.
  • Daten visuell aufbereiten können, z.b. mit online Tools wie Jupyter Notebooks, https://www.chartle.com,https://plotdb.com, oder unserem zukünftigen Research Data Management Repository (https://inveniordm.web.cern.ch).
  • Probleme wie Ausreißer, Störungen, fehlende Messwerte in Daten erkennen können und Lösungsstrategien anwenden können.
  • Unterschiedliche Datensatzformate und Zugriffsmöglichkeiten kennen, z.B.: online Datenbanken kennen und verarbeiten können (https://www.statista.com, https://trends.google.com/trends/, https://ourworldindata.org).
  • Eigene Datenmanagementsysteme erstellen können (SQLLite, mariadb, CSV, Excel) und mit Visualisierungstools verknüpfen können (graphana, Jupyter NB, etc).
  • Grundlagen der Statistik auf Daten anwenden können (Mittelwert, Median, Standardabweichung, Quantiles, Tests auf Normalverteilung der Daten, Korrelationen visualisieren).
  • Das Funktionsprinzip von maschinellen Lernmethoden auf Daten beschreiben können: Begriffe wie Vorhersagen, abhängige Variablen, erklärende Variablen erklären können.
  • Onlinetools (lineare- und nichtlineare Regressionen) zur Vorhersage anwenden können.

Schwerpunktthemen

  • Grundlagen der Datenmodellierung (Prozesse -> Sensoren -> Daten -> Variablen)
  • Grundlagen der Datenspeicherung (Diskretisierung, Nyquist-Shannon Theorem, Datenspeicherungsarten)
  • Imformationstheorie-Grundlagen (Entropie der Daten)
  • Grundlagen der Datenanalyse (Ausreißer, Störungen, fehlende Messwerte)
  • Grundlagen Statistik zur Datenanalyse (Ziele, Werkzeuge, Auswertungsbeispiele)
  • Ausgewählte Beispiele zur Datenmodellierung (lineare/nicht-lineare Regression, neuronale Netze)

Unterrichtsformat

Die LV baut auf vier Säulen auf:

  1. Grundlagenvermittlung per Frontalunterricht im Hörsaal mit interaktiven Elementen.
  2. Praktische Übungen in Gruppen in Computerräumen. Hier werden Jupyter Notebooks, Exceltabellen und online Tools zur Datenmodellierung verwendet.  
  3. Expertenvorträge zur Datenmodellierung aus Unternehmenssicht (was ist der Stand der Technik im Unternehmen, was müssen Studierende beherrschen, wenn sie bei diesen Unternehmen arbeiten wollen).
  4. Expertenvorträge zu weiterführenden Inhalten an der Montanuniversität (Wo und wie wende ich im Laufe des Studiums die Kenntnisse an, z.B. Machine Learning, IoT, etc.).

Links and Ressourcen

Empfohlene Fachliteratur

  • Jiawei, Han, and Kamber Micheline. Data mining: concepts and techniques. Morgan kaufmann, 2006. ISBN 978-0-12-381479-1.
    Link: https://myweb.sabanciuniv.edu/rdehkharghani/files/2016/02/The-Morgan-Kaufmann-Series-in-Data-Management-Systems-Jiawei-Han-Micheline-Kamber-Jian-Pei-Data-Mining.-Concepts-and-Techniques-3rd-Edition-Morgan-Kaufmann-2011.pdf
  • Keim, Daniel, Kai-Uwe Sattler, and AG Technologische Wegbereiter. “Von Daten zu KI.” Intelligentes Datenmanagement als Basis für Data Science und den Einsatz Lernender Systeme. Whitepaper aus der Plattform Lernende Systeme, München. Abgerufen am 05.09.2024.
    Link: https://www.plattform-lernende-systeme.de/files/Downloads/Publikationen/AG1_Whitepaper_Von_Daten_zu_KI.pdf
  • Ilyas, I. F., & Chu, X. (2019): Data Cleaning. ACM Press. ISBN:978-1-4503-7152-0

Ort und Zeit

  • Vorlesungen und Expertenvorträge: HS 1 Studienzentrum
    • jede Woche am Montag (11:00-13:00), ab dem 11.11.2024
    • jede Woche am Donnerstag (11:00-12:00), ab dem 14.11.2024
  • Übungen: Es gibt 10 Gruppen zur Auswahl mit unterschiedlichen Zeiten. Gruppe E wird in Englisch abgehalten. Aufgrund von Ausnahmen, bitte im MUOnline die genauen Termine beachten.
    • Gruppe 1: Dienstag (14:00-16:00) CR Hilbert [Tanja]
    • Gruppe 2: Dienstag (16:00-18:00) CR Hilbert [Juki]
    • Gruppe 4: Mittwoch (14:00-16:00) CR Hilbert [Tanja]
    • Gruppe 5: Mittwoch (16:00-18:00) CR Hilbert [Rino]
    • Gruppe 6: Mittwoch (18:00-20:00) CR Hilbert [Rino]
    • Gruppe Englisch: Mittwoch (16:00-18:30) CR IL/IT, CR IZR [Juki]
  • Bei Bedarf:
    • Gruppe 7: Montag (16:00-18:00) CR Hilbert
    • Gruppe 8: Dienstag (16:00-18:00) CR IZR
    • Gruppe 9: Mittwoch (14:00-16:00) CR IZR
  • Wird nicht angeboten: 
    • Gruppe 3 (Wird nicht abgehalten): Dienstag (18:00-20:00) HS TPT

 

Notwendiges Vorwissen

Keine. 

Folien und Unterlagen

Folgende Termine sind für die LV vorgesehen. Jedoch gilt die Liste als vorläufig und nicht alle Termine werden benötigt.  

Benotung

Die Benotung erfolgt immanent. Insgesamt können 100 Punkte durch aktives Mitarbeiten, Übungsblätter und durch Prüfungen erworben werden. Die Punkte werden über Moodle verwaltet und können jederzeit eingesehen werden. 

Die finale schriftliche Prüfung wird über Moodle abgehalten. 

Details zur Benotung werden in der ersten Vorlesungseinheit vorgestellt, d.h. am 11.11.2024. 

Benotungsschema: 0-49,9 Punkte (5), 50-65,9 Punkte (4), 66-79 Punkte (3), 80-91 Punkte (2), 92-100 Punkte (1).

Bei einer Gesamtpunktzahl von bis zu 79 % KANN (!) auch eine zusätzliche mündliche Leistungsüberprüfung erforderlich sein, wenn der positive Leistungsnachweis nicht eindeutig ist. In diesem Fall werden Sie informiert, sobald die Noten bekannt gegeben werden. Wenn Sie bereits eine Note über MU online erhalten haben, werden Sie nicht zu einer weiteren mündlichen Leistungskontrolle eingeladen.

Literatur



190.017 Advanced Machine and Deep Learning (5SH IL, WS)

|

Course Content & Topics

Theoretical and practical aspects of computing and learning with neural networks. Examination of the most commonly used algorithms for deep learning. From a practical point of view, different learning algorithms and types of neural networks are implemented and applied to artificial and real-world problems.

Initially, the selected methods are developed in 5-6 lecture units. Students then select a machine learning method and apply it independently. The project work is accompanied by weekly tutorials with tips and tricks. Finally, the project results are discussed in a written report and finally presented for 10-15 minutes. The grade is based on the quality of the code, the report and the final short presentation.

Learning Objectives

After successfully completing the integrated course, students will be able to:

  • describe and apply basic concepts and commonly used architectures of deep learning.
  • explain in detail how deep neural networks are designed and trained in supervised and unsupervised learning scenarios.
  • describe the advantages of different neural network models and algorithms, as well as their relationship to important machine learning concepts such as generalization and regularization.
  • apply the acquired practical skills in the implementation and application of state-of-the-art deep learning methods to solve various problems.

Location & Time

Location: TBD. Starting in November, lectures are planned to be held in HS 3 Studienzentrum (35SZ02211).

Time: TBD. 

Grading

Immanent examination character. Details will be presented in the first lecture unit.

* Active participation: 0-10 bonus points for active participation.

* Task sheets: 0-30 points for independent work on theoretical aspects.

* Project submissions:
– The implementation (Python code) will be graded 0-30 points.
– The report is assessed with 0-20 points.
– The slides, the final presentation and the discussion are awarded 0-20 points.

* Grading scale: 0-49.9 points (5), 50-65.9 points (4),66-79 points (3), 80-91 points (2), 92-100 points (1)

(With an overall result of up to 79%, an additional oral performance review MAY (!) also be prescribed if the positive performance record is not clear. In this case, you will be informed as soon as the grades are released. If you have already received a grade via MU online, you will not be invited to another oral examination).

Prerequisites

Literature

– Ian Goodfellow, Yoshua Bengio and Aaron Courville, “Deep Learning”, 2016.
– Christopher M Bishop, “Pattern Recognition and Machine Learning”, 2006.



BSc. Thesis, Merisa Salkic – Smart conversations: Enhancing robotic task execution through advanced language models

|

Supervisor: Linus Nwankwo, M.Sc.;
Univ.-Prof. Dr Elmar Rückert
Start date:  As soon as possible

 

Theoretical difficulty: mid
Practical difficulty: High

Abstract

In this thesis, we aim to enhance the method proposed in [1] for robust natural human-autonomous agent interaction through verbal and textual conversations. 

The primary focus would be to develop a system that can enhance the natural language conversations, understand the 

semantic  context of the robot’s task environment, and abstract this information into actionable commands or queries. This will be achieved by leveraging the capabilities of pre-trained large language models (LLMs) – GPT-4, visual language models (VLMs) – CLIP, and audio language models (ALMs) – AudioLM.

Tentative Work Plan

To achieve the objectives, the following concrete tasks will be focused on:

  • Initialisation and Background:
    • Study the concept of LLMs, VLMs, and ALMs.
    • How LLMs, VLMs, and ALMs can be grounded for autonomous robotic tasks.
    • Familiarise yourself with the methods at the project website – https://linusnep.github.io/MTCC-IRoNL/.
    •  
  • Setup and Familiarity with the Simulation Environment
    • Build a robot model (URDF) for the simulation (optional if you wish to use the existing one).
    • Set up the ROS framework for the simulation (Gazebo, Rviz).
    • Recommended programming tools: C++, Python, Matlab.
    •  
  • Coding
    • Improve the existing code of the method proposed in [1] to incorporate the aforementioned modalities—the code to be provided to the student.
    • Integrate other LLMs e.g., LLaMA and VLMs e.g., GLIP modalities into the framework and compare their performance with the baseline (GPT-4 and CLIP).
    •  
  • Intermediate Presentation:
    • Present the results of your background study or what you must have done so far.
    • Detailed planning of the next steps.
    •  
  • Simulation & Real-World Testing (If Possible):
    • Test your implemented model with a Gazebo-simulated quadruped or differential drive robot.
    • Perform the real-world testing of the developed framework with our Unitree Go1 quadruped robot or with our Segway RMP 220 Lite robot.
    • Analyse and compare the model’s performance in real-world scenarios versus simulations with the different LLMs and VLMs pipelines.
    •  
  • Optimize the Framework for Optimal Performance and Efficiency (Optional):
    • Validate the model to identify bottlenecks within the robot’s task environment.
    •  
  • Documentation and Thesis Writing:
    • Document the entire process, methodologies, and tools used.
    • Analyse and interpret the results.
    • Draft the project report or thesis, ensuring that the primary objectives are achieved.
    •  
  • Research Paper Writing (optional)
    •  

Related Work

[1]  Linus Nwankwo and Elmar Rueckert. 2024. The Conversation is the Command: Interacting with Real-World Autonomous Robots Through Natural Language. In Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’24). Association for Computing Machinery, New York, NY, USA, 808–812. https://doi.org/10.1145/3610978.3640723.

[2]  Nwankwo, L., & Rueckert, E. (2024). Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models. arXiv preprint arXiv:2403.12273.



After Business Trip Paperwork

|

New Obligation: Submit along comparitive offers

(This starts from 30 July 2024)

You are required to submit a comparative analysis of the prices for your trip along with other documents for claims.

Documents to submit and print in hard copy:

  • Conference/summer school schedule
  • Transport ticket (flight/intercity train/city train/bus)
  • Registration fee
  • Spesenabrechnung/Reisekostennachweis (from SAP)
  • Accomodation
  • Comparative analysis of the prices (only be paid 50% of the flight costs, if this document is not submitted along)

New Obligation: Monthly data entry for öbb tickets

(This starts from 26 Jun 2024)

You can find the entry form at here: https://cloud.cps.unileoben.ac.at/index.php/s/GTFTrT8btK7mMtW

Procedure to submit paperwork to Financial Department

Published on 21 May 2024

Update 1 on 26 Jun 2024

Update 2 on 30 July 2024

1. Login into SAP

At your SAP, click on “Meine Reisen und Spesen”.

2. Click on your desired Trip

In my case, I will show example in Austria.

 

Click “Welter” to proceed.

3. At the Main Page with 4 steps

Step 1: Verify every information especially Kontierung (Your project number)

Next, click on checkbox with * and then proceed with “Belege erfassen”

Step 2: Add all related claims

Step 3: Validate

There are two options: Save it for future or Sent it to financial department


4. Final step

  • Prepare all the original receipts and keep a copy with you.
  • Print out the above from system
  • Put the documents at “Dienstreisen Folder” at Regina’s place
  • Bring the folder to Uni Post Office at 1st floor of old building.



Print a Poster

|

Kindly ask for permission before proceed to poster printing.

 

To print a poster, you can either go for:

Option 1: Mail Boxes Leoben

Price list:

A0: ~20.00 euro

CPS account: KST 101900

Email them, and they will record at CPS account.


Option 2: ÖH Leoben

Fill the form at : https://www.oeh-leoben.at/de/plotauftrag

Price list:

A0: ~6.63 euro

A1: ~3.35 euro

Only cash payment, and pay it when obtaining.