HRI-SL: Human-Robot Interaction with Sign Language

image_pdfimage_print

Start date: Open

Location: Leoben

Position Types: Thesis/Internship

Duration: 3-6 months, depending on the level of applicant’s proficiency on the asked qualifications.

 

Keywords: Human-Robot Interaction (HRI), Human Gesture Recognition, Sign Language, Robotics, Computer Vision, Large Language Models (LLMs), Behavior Cloning, Reinforcement Learning, Digital Twin, ROS-2

Supervisor:

You can work on this project either by doing a B.Sc or M.Sc. thesis or an internship*.

Abstract

As the interaction with robots becomes an integral part of our daily lives, there is an escalating need for more human-like communication methods with these machines. This surge in robotic integration demands innovative approaches to ensure seamless and intuitive communication. Incorporating sign language, a powerful and unique form of communication predominantly used by the deaf and hard-of-hearing community, can be a pivotal step in this direction. 

By doing so, we not only provide an inclusive and accessible mode of interaction but also establish a non-verbal and non-intrusive way for everyone to engage with robots. This evolution in human-robot interaction will undoubtedly pave the way for more holistic and natural engagements in the future.

DALL·E 2023-02-09 17.32.48 - robot hand communicating with sign language

Project Description

The implementation of sign language in human-robot interaction will not only improve the user experience but will also advance the field of robotics and artificial intelligence.

This project will encompass 4 crucial elements.

  1. Human Gesture Recognition with CNNs and/or Transformers – Recognizing human gestures in sign language through the development of deep learning methods utilizing a camera.
    • Letter-level
    • Word/Gloss-level
  2. Chat Agent with Large Language Models (LLMs) – Developing a gloss chat agent.
  3. Finger Spelling/Gloss gesture with Robot Hand/Arm-Hand –
    • Human Gesture Imitation
    • Behavior Cloning
    • Offline Reinforcement Learning
  4. Software Engineering – Create a seamless human-robot interaction framework using sign language.
    • Develop a ROS-2 framework
    • Develop a robot digital twin on simulation
  5. Human-Robot Interaction Evaluation – Evaluate and adopt the more human-like methods for more human-like interaction with a robotic signer.
1024-1364
Hardware Set-Up for Character-level Human-Robot Interaction with Sign language.
Example of letter-level HRI with sign language: Copying agent

Qualifications

  • Currently pursuing a Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, Mechanical Engineering, Mathematics or related fields.
  • Strong programming skills in Python and experience with deep learning frameworks such as PyTorch or TensorFlow.
  • Experience working with robotics hardware.
  • Knowledge of computer vision and image processing techniques
  • Strong problem-solving skills and ability to work independently and collaboratively.
  • Good written and verbal communication skills in English.
  • Passion for creating technology that is accessible and inclusive for everyone
  • Experience in working on research projects or coursework related to robotics or artificial intelligence is a plus

Opportunities

This project provides an excellent opportunity to gain hands-on experience in cutting-edge research, working with a highly collaborative and supportive team. The student/intern will also have the opportunity to co-author research papers and technical reports, and participate in conferences and workshops.

Application/Questions

Send us your CV accompanied by a letter of motivation at fotios.lygerakis@unileoben.ac.at with the subject: “Internship/Thesis Application | Sign Language Robot Hand”

Funding

* This project does not offer a funded position. Below we list some relevant grant application details.

CEEPUS grant (European for undergrads and graduates)

Find details on the Central European Exchange Program for University Studies program at https://grants.at/en/ or at https://www.ceepus.info.

In principle, you can apply at any time for a scholarship. However, also your country of origin matters and there exist networks of several countries that have their own contingent.

Ernst Mach Grant (Worldwide for PhDs and Seniors)

Rest Funding Resourses

Apply online at http://www.scholarships.at/