Autonomous Cart
This smart off-road cart was designed for outdoor adventurers who need a reliable hands-free way to transport gear. It uses OAK-D depth sensing and YOLO object detection to visually follow a specific person while actively avoiding obstacles.
Built on ROS 2 and powered by a Raspberry Pi 5, a Hailo AI accelerator, and an ESP32, the system offers real-time responsiveness in rugged environments. For precise tracking and communication, it integrates ultra-wideband (UWB) and Bluetooth.
The cart also features memory-based path learning, allowing it to retrace routes and adapt to complex outdoor terrains. I developed a custom power management system and used a motor controller to handle varied terrain with ease.
From Sensing to Steering: How It Works
ROS 2 (Robot Operating System 2) acts as the central nervous system of the smart off-road cart, managing everything from vision to motion. This framework allows different modules—like perception, control, and hardware communication—to work together seamlessly, as shown in the system map.
The OAK-D camera captures depth and visual data, which is processed through a series of nodes for person detection, gesture recognition, and feature extraction. These inputs are used by the control system to determine how the cart should move, factoring in the user’s location and any nearby obstacles.
Movement commands are sent to an ESP32 microcontroller, which controls a servo for steering and two ESCs (Electronic Speed Controllers) that adjust the speed of each wheel independently—much like a car’s differential system. This setup enables smooth turns and stable navigation across rough terrain.
From camera to wheels, the entire system works together to deliver hands-free following with real-time adaptability in outdoor environments.
Collaboration & Interactive Showcase
In collaboration with the Asian Art Museum in San Francisco, the cart was used to explore a novel form of human–robot interaction. Using its vision system, the cart could detect visitors who were sitting and potentially disengaged—such as those looking at their phones—and approach them. An AI-powered display on the cart then invited the visitor to play 猜字谜 (cāi zì mí, a traditional character-riddle game popular in many Asian countries). Visitors could also ask the cart questions about specific artworks, with answers pulled from a curated database. This experiment tested how autonomous devices can initiate playful, culturally rooted interactions while also providing educational engagement in public spaces.