Overview
Extended reality teleoperation of a Universal Robots UR3e arm using ROS2 and Unity, developed as coursework.
Details
Subject TBC (Semester TBC) · High Distinction
Extended reality teleoperation of a Universal Robots UR3e using ROS2 and Unity. I built the perception system end to end, alongside a custom inverse-kinematics control stack and the project’s containerised setup.
My Contributions
Perception
- RGB and point cloud streaming from an Intel RealSense camera into the Unity scene using plain points and flat quad scaling. Scaled accurately (1 Unity Unit = 1 Meter)
- Floating HUD window giving the operator a live image feed for continuous visual feedback
- YOLO segmentation (Ultralytics COCO) integrated into the stream, with segmentation outlines drawn over the image feed for object detection. Optimised for CPU using Open Vino. Can also be run on GPU via CUDA
- Colour-based button detection with a custom HSV calibration tool to tune detection thresholds reliably
- AR Tag Detection utilising Keijiros April Tag Detector package. Accurately gives 6-DOF pose of tags
- Hand-Eye Calibration using opencv. AR Tag data publishes out to ROS. Individual ROS bags are recorded to capture UR3e joint state and AR-Tag pose data. All bags are run through a custom python hand-eye calibration solver that outputs various solutions using multiple solvers such as Tsai-Lenz and Park, Horaud, etc. This data can be saved as a json and passed into Unity as the transofrm between EEF and Realsense camera
Control
- Custom RMRC inverse-kinematics controller driven by a draggable goal puck, where moving the puck generates the pose error the end effector converges on
- Task-space visualiser to reason about reach and visual constraints
ROS-Unity Bridge
- Migrated both halves of the bridge to ROS2 Humble: forked the Python endpoint and the C# Unity connector, ported them off ROS1 and the deprecated Foxy/Galactic, and added full ROS2 action support (UUID goal IDs,
action_msgs) - Re-architected it for high-rate streaming so 90Hz RealSense depth and colour reach Unity at a smooth 60Hz without stalling: parallel stream callbacks, BEST_EFFORT QoS on sensor topics, latest-frame-only delivery, and CDR serialisation deferred off the ROS thread
- Kept the Unity side from falling behind by dropping stale frames from the receive queue, reading asynchronously instead of polling, and auto-reconnecting on a dropped link
- Trimmed what goes over the wire by throttling chatty topics and giving each topic a short numeric ID instead of repeating its full name on every message
- Tuned the TCP transport on both ends with
TCP_NODELAY, enlarged buffers, and batched sends
Infrastructure
- Dockerised the full stack with appropriate volume mounts for straightforward install and setup
Repositories