Start date: Open
Location: Leoben
Position Types: Thesis/Internship
Duration: 3-6 months, depending on the level of applicant’s proficiency on the asked qualifications.
Keywords: Human-Robot Interaction (HRI), Human Gesture Recognition, Sign Language, Robotics, Computer Vision, Large Language Models (LLMs), Behavior Cloning, Reinforcement Learning, Digital Twin, ROS-2
Supervisor:
You can work on this project either by doing a B.Sc or M.Sc. thesis or an internship*.
Abstract
As the interaction with robots becomes an integral part of our daily lives, there is an escalating need for more human-like communication methods with these machines. This surge in robotic integration demands innovative approaches to ensure seamless and intuitive communication. Incorporating sign language, a powerful and unique form of communication predominantly used by the deaf and hard-of-hearing community, can be a pivotal step in this direction.
By doing so, we not only provide an inclusive and accessible mode of interaction but also establish a non-verbal and non-intrusive way for everyone to engage with robots. This evolution in human-robot interaction will undoubtedly pave the way for more holistic and natural engagements in the future.
Project Description
The implementation of sign language in human-robot interaction will not only improve the user experience but will also advance the field of robotics and artificial intelligence.
This project will encompass 4 crucial elements.
- Human Gesture Recognition with CNNs and/or Transformers – Recognizing human gestures in sign language through the development of deep learning methods utilizing a camera.
- Letter-level
- Word/Gloss-level
- Chat Agent with Large Language Models (LLMs) – Developing a gloss chat agent.
- Finger Spelling/Gloss gesture with Robot Hand/Arm-Hand –
- Human Gesture Imitation
- Behavior Cloning
- Offline Reinforcement Learning
- Software Engineering – Create a seamless human-robot interaction framework using sign language.
- Develop a ROS-2 framework
- Develop a robot digital twin on simulation
- Human-Robot Interaction Evaluation – Evaluate and adopt the more human-like methods for more human-like interaction with a robotic signer.
Qualifications
- Currently pursuing a Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, Mechanical Engineering, Mathematics or related fields.
- Strong programming skills in Python and experience with deep learning frameworks such as PyTorch or TensorFlow.
- Experience working with robotics hardware.
- Knowledge of computer vision and image processing techniques
- Strong problem-solving skills and ability to work independently and collaboratively.
- Good written and verbal communication skills in English.
- Passion for creating technology that is accessible and inclusive for everyone
- Experience in working on research projects or coursework related to robotics or artificial intelligence is a plus
Project Outcomes
Publications:
TBA
Datasets:
TBA
Patents:
TBA
M.Sc. Theses
- Iye Szin Ang: “ROS2-based Human-Robot Interaction Framework with Sign Language“, August 2023 [Download pdf]
B.Sc. Thesis
TBA
Opportunities
This project provides an excellent opportunity to gain hands-on experience in cutting-edge research, working with a highly collaborative and supportive team. The student/intern will also have the opportunity to co-author research papers and technical reports, and participate in conferences and workshops.
Application/Questions
Send us your CV accompanied by a letter of motivation at fotios.lygerakis@unileoben.ac.at with the subject: “Internship/Thesis Application | Sign Language Robot Hand”
Funding
* This project does not offer a funded position. Below we list some relevant grant application details.
CEEPUS grant (European for undergrads and graduates)
Find details on the Central European Exchange Program for University Studies program at https://grants.at/en/ or at https://www.ceepus.info.
In principle, you can apply at any time for a scholarship. However, also your country of origin matters and there exist networks of several countries that have their own contingent.
Ernst Mach Grant (Worldwide for PhDs and Seniors)
Find details on the program at https://grants.at/en/ or at https://oead.at/en/to-austria/grants-and-scholarships/ernst-mach-grant.
Rest Funding Resourses
Apply online at http://www.scholarships.at/