Supervisor: Fotios Lygerakis and Prof. Elmar Rueckert
Start Date: 1st March 2023
Theoretical difficulty: low
Practical difficulty: mid
Abstract
As the interaction with robots becomes an integral part of our daily lives, there is an escalating need for more human-like communication methods with these machines. This surge in robotic integration demands innovative approaches to ensure seamless and intuitive communication. Incorporating sign language, a powerful and unique form of communication predominantly used by the deaf and hard-of-hearing community, can be a pivotal step in this direction.
By doing so, we not only provide an inclusive and accessible mode of interaction but also establish a non-verbal and non-intrusive way for everyone to engage with robots. This evolution in human-robot interaction will undoubtedly pave the way for more holistic and natural engagements in the future.
![DALL·E 2023-02-09 17.32.48 – robot hand communicating with sign language DALL·E 2023-02-09 17.32.48 - robot hand communicating with sign language](https://cps.unileoben.ac.at/wp/elementor/thumbs/DALL·E-2023-02-09-17.32.48-robot-hand-communicating-with-sign-language-1-q1wymbedfbkuk57gdfv9hzy2j3nhz0gxbtujihotcw.png)
Thesis
Project Description
The implementation of sign language in human-robot interaction will not only improve the user experience but will also advance the field of robotics and artificial intelligence.
This project will encompass 4 crucial elements.
- Human Gesture Recognition with CNNs and/or Transformers – Recognizing human gestures in sign language through the development of deep learning methods utilizing a camera.
- Letter-level
- Word/Gloss-level
- Chat Agent with Large Language Models (LLMs) – Developing a gloss chat agent.
- Finger Spelling/Gloss gesture with Robot Hand/Arm-Hand –
- Human Gesture Imitation
- Behavior Cloning
- Offline Reinforcement Learning
- Software Engineering – Create a seamless human-robot interaction framework using sign language.
- Develop a ROS-2 framework
- Develop a robot digital twin on simulation
- Human-Robot Interaction Evaluation – Evaluate and adopt the more human-like methods for more human-like interaction with a robotic signer.
![1024-1364 1024-1364](https://cps.unileoben.ac.at/wp/elementor/thumbs/1024-1364-qd9e0pg1yi0yt2whdyf3fc4vlgys59o1qew9eq3s80.jpg)
![](https://cps.unileoben.ac.at/wp/pipeline.gif)