Control of a robot using real time hand gesture recognition

Student: Nikola Smrečki

This work presents the development of a system for controlling a UR5 robot using hand-gesture recognition, enabling intuitive human–robot interaction without the need for specialized equipment.

The system is based on real-time video processing from a mobile camera, where hand gestures are recognized and mapped to robot control commands. Gesture recognition is implemented using MediaPipe Hands, which tracks hand landmarks and enables reliable gesture classification.

A lightweight web application is used to initiate and manage gesture recognition, designed with a simple and user-friendly interface using HTML, CSS, and JavaScript. Recognized gestures are transmitted via WebSocket communication to a Node.js server, which then forwards control commands to the UR5 robot over TCP/IP for execution.

This work demonstrates how modern computer vision and web technologies can be combined to create accessible and intuitive robot control systems, bringing human–robot interaction closer to everyday users beyond industrial environments.

Repository: https://repozitorij.fsb.unizg.hr/object/fsb:13679