Autonomous Camera-Tracking Robot Rover

The objective was to architect a highly robust, multi-agent robotic platform capable of autonomous target tracking and teleoperation. I engineered a heterogeneous dual-compute architecture: an NVIDIA Jetson Nano acts as the high-level cognitive node for AI perception, while a deterministic ESP32 microcontroller handles real-time kinematics and low-level PWM motor control. This decoupled approach ensures safety-critical operations never stall during heavy GPU inference workloads.

The resulting 6-wheel-drive rover integrates a multi-axis camera gimbal that autonomously tracks dynamic targets using onboard AI. It features a custom web-based teleoperation interface, closed-loop active camera stabilization, and rigorous hardware guardrails protecting actuators from algorithmic failure.

• Deterministic Control Arbitration: Seamless web UI toggle between manual joystick driving and autonomous "follow-me" state machine. The AI perception pipeline continuously tracks targets independently of the locomotion mode. • Ultra-Low Latency Teleoperation: Achieves 20–30ms end-to-end latency at a 50Hz update rate over WebSockets, crucial for stable real-time manual control. • Edge AI Vision Pipeline: Utilizes a Luxonis OAK-D stereo depth camera for onboard hardware-accelerated MobileNet-SSD person detection and MediaPipe hand landmark tracking. Implements spatial gating to reject false positives. • Hand & Gesture Tracking: Implemented robust hand and gesture tracking using OpenCV and the camera to facilitate vision-based interaction. • Adaptive Control & Limit Measurements: Rather than relying on naive PID controllers that saturate at hardware limits, the system implements an adaptive control strategy where limit measurements were figured out through measuring the IMU's change. If a servo reaches a physical boundary, the system gracefully degrades, disabling the constrained axis while maintaining active tracking on remaining degrees of freedom. • IMU-Driven Active Stabilization: Exploits the OAK-D's integrated BNO085 IMU for state estimation. Real-time orientation quaternions are projected into pitch/roll/yaw corrections to counter-rotate the turret servos, isolating the camera frame from chassis perturbations. • Embedded Hardware Guardrails: Firmware implements a strict 60% PWM saturation limit, 500ms watchdog timers for communication loss, and dependent-axis inverse kinematics constraints to prevent self-collision.

Splitting compute responsibilities across two processors was the single most consequential architectural decision. Heavy non-deterministic AI vision workloads execute on the GPU, while safety-critical motor timing runs on a deterministic RTOS-like microcontroller loop.

Hardware Configuration: • High-Level Compute: NVIDIA Jetson Nano 4GB (AI vision, state machine, inverse kinematics) • Low-Level Controller: ESP32 Dev Module (PWM synthesis, WiFi server, safety interlocks) • Perception Sensor: Luxonis OAK-D (Stereo depth, VPU acceleration, IMU) • Actuation: Cytron MDDS30 dual motor driver and PCA9685 I2C 16-channel PWM driver for MG996R servos

Software Stack: • Firmware (ESP32): C++ (Arduino framework), asynchronous WebSockets for telemetry, high-speed USB-serial protocol. • Autonomy Node (Jetson Nano): Python 3, DepthAI SDK, OpenCV, custom closed-loop controllers. • Client Interface: Responsive HTML5/JavaScript dashboard for remote monitoring.

To hit the necessary performance metrics for autonomous control, I performed rigorous latency budgeting across the system:

1. Teleoperation Path (Web → ESP32 → Actuators): A zero-allocation JSON parser processes WebSocket commands directly to PWM registers. Latency: ~20–30ms. 2. Perception Path (Camera → Jetson → Servos): Target detection on the VPU feeds into Jetson-side PID controllers generating direct GPIO PWM. Latency: ~11–33ms. 3. Autonomous Locomotion (Camera → Jetson → ESP32 → Wheels): Depth-based distance calculations generate velocity vectors transmitted over USB-serial to the ESP32. Latency: ~21–62ms.

By routing locomotion commands through the ESP32 while keeping gimbal control on the Jetson, I guaranteed sub-50ms manual driving response while isolating the critical drive systems from potential Python process crashes.

Early testing revealed a "Phantom Stop" anomaly where the locomotion system would halt abruptly. Systematic tracing from the WebSocket packet down to the PWM logic analyzer output revealed the 500ms safety watchdog was triggering due to UI slider event sparsity. I re-architected the front-end to emit a 20Hz heartbeat, satisfying safety constraints without weakening the watchdog.

Additionally, I developed an instrumented variant of the ESP32 firmware that logged the entire control pipeline from raw vector math to actual GPIO pin states allowing empirical, data-driven debugging of complex mechatronic interactions.

This project demonstrates the ability to architect, build, and deploy a real-time, safety-critical embedded robotic system. It highlights proficiency in quantitative architecture trade-off analysis, state estimation, closed-loop control, and resilient fault-tolerant engineering which are all core competencies for advanced research in autonomous systems and applied robotics.