Meeting 03/11​

Done

• Virtual machine setup & running experiments
  • speed is x3 slower than RTX 3090

Started working on

• develop experiments with new dataset (Caltech101)
• assess experiments
• test transfer learning capabilities of CR-VAE

MS Students Updates

• Melanie
  • Object Detection using Resnet
• Julian
  • basic tutorials on ROS
  • Teleoperation of Turtlebot using PS5 controller
  • UR3 on ROS using MoveIt

Next on

• description of caltech
• talk with Konrad about the cluster
• Update paper with correct results and send it to CVPR
• Hyperparameter grid search experiments
• literature review on representation learning
• experiments with artificial datasets
• develop methods
  • contrastive learning for spiking neural networks.
  • mode-seeking kl divergence

Meeting 11/11​

Progress

• Updated draft of the CR-VAE paper.
• Missed author registration deadline for CVPR >_<*
• Run experiments with simple AE / CR-AE
• Test if InfoMax objective actually works -> it doesn’t.
• New implementation of the loss function
• Tried input normalization and MSE for reconstruction error

• Assess experiments
• Hyperparameter grid search experiments for CR-VAE
• Literature review on representation learning (review paper)
• Going through ROS 2 documentation
• Get acquanted with UR3

MS Students Updates

• Melanie
  • Object Detection using Resnet
  • Next on: Image Segmentation
• Julian
  • Teleoperation of Turtlebot using PS5 controller
  • Next on: UR3 on Gazebo using MoveIt

Other activities

• Discussion with Sahar and Vedant about how Sahar could frame her Reinforcement learning research problem.

Next on

• assess experiments from grid search with the new loss function
  • hopefully there will be some distinct difference of the 3 methods
• further assess the value of MI as an auxilary task for unsupervised representation learnning
  • show that MI in InfoMax actually introduce noise
• Denoising AE/CR-AE/VAE/CR-VAE with augmented images.
• develop experiments with new dataset (Caltech101)
• description of caltech
• test transfer learning capabilities of CR-VAE
• literature review on representation learning
• experiments with artificial datasets
• develop methods
  • contrastive learning for spiking neural networks.
  • mode-seeking kl divergence

   

Journal ideas

• Find the best CL method for CR-VAE
• Transfer learning

Meeting 17/11

Progress

• gpu grid setbacks
• caltech101 dataset
• ROS2 refresh
• new results

MS Students Updates

• Melanie
  • Image Segmentation on Steel Defect dataset
  • Next on: Contrastive Learning
• Julian
  • Sick
  • Next on: UR3 on Gazebo using MoveIt

Other activities

• Study abroad fair speech

Next on

• study the big performance gap in KLD between CR-VAE and VAE
• further assess the value of MI as an auxilary task for unsupervised representation learnning
  • show that MI in InfoMax actually introduce noise
• Denoising AE/CR-AE/VAE/CR-VAE with augmented images.
• develop experiments with new dataset (Caltech101)
• description of caltech
• test transfer learning capabilities of CR-VAE
• literature review on representation learning
• experiments with artificial datasets
• develop methods
  • contrastive learning for spiking neural networks.
  • mode-seeking kl divergence

   

Journal ideas

• Find the best CL method for CR-VAE
• Transfer learning
• develop a contrastive regularization layer for NN

Meeting 23/11

Updates

• AAAI paper submission update
  • received an email that the file was never uploaded even though I have a verification email. Still in the process of figuring out
• new results on smaller architecture -> more distinct results

MS Students Updates

• Melanie
  • Image Segmentation on Steel Defect dataset
  • Next on: Contrastive Learning
• Julian
  • Dropped
  • Subject was not aligned with his program
  • working at the lab did not fit his schedule

Other activities

• christmas & hololens 2 unboxing videos
• storage place or display for PS5 controler, hololense, etc?
• plan to publish the AR project as internship position
  • LinkedIn -> CPS page?
  • MUL
  • Emails

Next on

• reconstruct the paper
  • ICML
  • IJCAI
• caltech101 dataset
• literature review on representation learning
• Hololens 2 review
• seminar talk for latent space representation and explainability in neural networks, feature maps. organize meetings (1 paper per week)

Meeting 21/10

Done

• experiment assessing with small custom architecture

Next on

• find a new controller
• set up computer for Melanie & Julian
• Virtual machine setup

Meeting 25/10

Done

• preliminary experiment assessing with resnet architecture
• schedule new experiments on resnet architecure
• preparation and meetings with MS studentsmeetings notes per month

Next on

• assess experiments
• literature review on representation learning
• Virtual machine setup
• experiments with artificial datasets
• develop methods
  • contrastive learning for spiking neural networks.
  • mode-seeking kl divergence

Increase your Productivity

Schedule your weekly tasks, meetings, courses or activities!

Increase your Flexibility

Access your files from any computer, tablet or phone!

Work as a Team

Edit together in real-time with easy sharing, and use comments, suggestions, and action items to keep things moving. Or use @-mentions to pull relevant people, files, and events into your online files for rich collaboration.

Important Links

• Calendar
• Documents(Google’s “Word”)
• Sheets(Google’s “Excel”)
• Slides (Google’s PowerPoint)
• Forms

Participants

Niko, Fotis, Linus, Vedant

Agenda

• First discussion on the projects structure of CPS Hub
• Initial plan & examples

Notes

• Components (e.g. UR3, RH8D_hand, Glove, Hololens2) are independent repositories
• Projects (e.g. TacProMPs, HololensTeleop) are independent repositories that use the above repos.
• No custom messages without previous team meeting
• Use Foxy ROS2

Description

Mixed Reality (AR) interface based on Unity 3D for intuitive programming of robotic manipulators (UR3). The interface will be implemented within on the ROS 2 robotic framework.

Note: This project is also offered under the Integrated CPS project course or as B.Sc or M.Sc. Thesis

Qualifications

• Basic skills in Python or C++
• ROS
• Unity3D or C#

 Keywords: Augmented Reality, Robotic Interfaces, Engineering, Graphical Design

Duration

Minimum 3 months. Preferably 5-6 months.

Abstract

Robots will become a necessity for every business in the near future. Especially companies that rely heavily on the constant manipulation of objects will need to be able to constantly repurpose their robots to meet the ever changing demands. Furthermore, with the rise of Machine Learning, human collaborators or ” robot teachers” will need a more intuitive interface to communicate with them, either when interacting with them or when teaching them.

In this project we will develop a novel Mixed (Augmented) Reality Interface for teleoperating the UR3 robotic manipulator. For this purpose we will use AR glasses to augment the user’s reality with information about the robot and enable intuitive programming of the robot. The interface will be implemented on a ROS 2 framework for enhanced scalability and better integration potential to other devices.

Outcomes

This internship will result to an innovative graphical interface that enables non-experts to program a robotic manipulator.

The intern will get valuable experience in the Robot Operating System (ROS) framework and developing graphical interfaces on Unity. The student will also get a good understanding of robotic manipulators (like UR3) and develop a complete engineering project.

Funding

Please contact us via cps@unileoben.ac.at if you want to join us for an internship.

We will support you during your application for an internship grant. Below we list some relevant grant application details.

CEEPUS grant (European for undergrads and graduates)

Find details on the Central European Exchange Program for University Studies program at https://grants.at/en/ or at https://www.ceepus.info.

In principle, you can apply at any time for a scholarship. However, also your country of origin matters and there exist networks of several countries that have their own contingent.

Ernst Mach Grant (Worldwide for PhDs and Seniors)

Find details on the program at https://grants.at/en/ or at https://oead.at/en/to-austria/grants-and-scholarships/ernst-mach-grant.

Rest Funding Resourses

Apply online at http://www.scholarships.at/

Supervisors:

• Fotios Lygerakis
  
• Univ. Prof. Dr Elmar Rückert

Start date: October 2022

Qualifications

• Basic skills in Python or C++
• ROS or Unity3D/C#

 Keywords: Augmented Reality, Robotic Interfaces, Engineering, Graphical Design

Description

Mixed Reality (AR) interface based on Unity 3D for intuitive programming of robotic manipulators (UR3). The interface will be implemented within on the ROS 2 robotic framework.

Note: This project is also offered as Internship position.

Abstract

Robots will become a necessity for every business in the near future. Especially companies that rely heavily on the constant manipulation of objects will need to be able to constantly repurpose their robots to meet the ever changing demands. Furthermore, with the rise of Machine Learning, human collaborators or ” robot teachers” will need a more intuitive interface to communicate with them, either when interacting with them or when teaching them.

In this project we will develop a novel Mixed (Augmented) Reality Interface for teleoperating the UR3 robotic manipulator. For this purpose we will use AR glasses to augment the user’s reality with information about the robot and enable intuitive programming of the robot. The interface will be implemented on a ROS 2 framework for enhanced scalability and better integration potential to other devices.

Outcomes

This thesis will result to an innovative graphical interface that enables non-experts to program a robotic manipulator.

The student will get valuable experience in the Robot Operating System (ROS) framework and developing graphical interfaces on Unity. The student will also get a good understanding of robotic manipulators (like UR3) and develop a complete engineering project.

Interested?

If this project sounds like fun to you, please contact Fotios Lygerakis by email at fotios.lygerakis@unileoben.ac.at or simple visit us at our chair in the Metallurgie building, 1st floor.

Agenda

Next steps after AAAI submission

Upcoming research questions to answer

1. Normalize total loss
2. What is the performance of CR-VAE with ResNet architecture on MNIST and CIFAR-10 Datasets?
3. What is the performance of MoCo on MNIST and CIFAR-10 Datasets?
4. How does CR-VAE-BIG compare with MoCo?
5. What is better, SGD or Adam? Why?
6. What is better, E2E or Modular? Why?
7. How can we train on ImageNet? Maybe alternative datasets?
8. New architecture: decoder input -> concatenated latent representations from q and k encoders.
9. Can we incorporate all representation techniques into one?

post paper submission setbacks

• KL divergence computation was wrong. When fixed, performance was different
• With a weight factor of 1 for the KL divergence, the learned features performance in classification task diminish.
• This report shows this problem.
• Same behavior for CR-VAE. Untill the reconstruction and the contrastive losses are in the same scale with the KLD loss, the performance will continue to deviate. This happens because KLD dominates numerically the total loss.
• Way to mitigate it:
  • Descending beta value
    • currently exploring different scheduling techniques
    • report
• Note: CR-VAE does not seem novel now
• Normalizing total loss (weight loss inversively with their magnitude) might lead to better performance

Agenda

1. Data / Visualization framework in Python
   • API framework/usage guidelines
   • input  format
   • output format
   • Gui to inspect the data.
     • Original Data view
     • subsection view (using the rotated and subselected image parts).
     • Slider to adjust the time.
     • Play function.
     • Replay speed adjustment.
     • Basic statistics of the shown data (e.g. histogramms of the two images, min, max, mean, boxplots, number of blobs [1], …).
2. Symmetry measurements
   • Develop measures and visualization tools to detect asymmetries between the two images.
   • Find examples of such asymmetries.
   • Analyze them.
3. Occlusion removal
   • Classical CV approach
   • Learning-based approach(De-Occlusion)
4. Abnormality detection
5. Thesis writing

Topic 1: Data / Visualization framework in Python

Deliverables due to September

use pandas dataframe library

Topic 2: Symmetry measurements 

Deliverables due to October:

• Develop automatic detection methods and selection tools for your guidance
  • e.g., highlight these events in your time-line in the gui with red bars, or create a list of events that can be selected

Topic 3: Occlusion removal

Concerned about its appicability to this project. we cannot assume it will work with non-face data

If we don’t know the dynamics of the liquid we cannot reconstruct maintaining the true underlying information.

due to November

https://arxiv.org/pdf/1612.08534.pdf

https://github.com/zhaofang0627/face-deocc-lstm

Next Steps

• Abnormality detection
  • due December
• Thesis writing
  • due January

Next Meeting: TBA

Agenda

• Presentation on Metallurgy review
• Next Steps
  • Study dense NN
    • MNIST dataset
  • Study CNNs
    • Classification
    • Bounding Box
    • Segmentation
    • Feature matching
  • Autoencoders
    • Anomaly detection
  • FlowNet 2.0

Topic 1: Presentation on Metallurgy review

Great introductory presentation on each book’s(3) content

No need to study the math on the properties of mixtures on the second book

Next Steps

1. Present an introduction to NN/CNNs
2. Small jupyter tutorial on DNN/CNNs
3. Presentation of FlowNet paper

Next Meeting: Tue 26 July

Agenda

• Present Paper Concept: Contrastive VAE
  • https://docs.google.com/presentation/d/1zBnog1A9mlHpZ4sFhwS12w6UBMrNYRfKgxCbLvcysUA/edit?usp=sharing

Topic 1: ConVAE

Present concept, math and next steps

Notes

• Add an intermediate step in the introduction: “Why we want the latent representations”
• Motivate Unsupervised Learning
• Consider using a different distribution for the prior: L1 norm for example
• What has changed in the behavior of the ConVAE in comparison with VAE from an Information Theory perspective
• Mathematical proof that it can work in all datasets
• Train later on ImageNet