S-Bot Multimodal Edge AI Assistant

S-Bot is a localized, multimodal AI assistant architected entirely for edge execution on an NVIDIA Jetson Nano Orin. The primary engineering goal was to build a highly concurrent, fully private agentic system capable of asynchronous visual perception, biometric voice identification, and Large Language Model (LLM) reasoning without relying on opaque vendor ecosystems.

This project is a rigorous exercise in distributed systems integration, latency optimization, and concurrent programming on constrained edge hardware. It requires six independent, real-time subsystems (Audio Capture, Wake Detection, VAD, STT, Vision, and TTS) to operate continuously without thread starvation or memory leaks.

• Grammar-Constrained Wake Activation: Continuous listening via a Vosk KaldiRecognizer utilizing a strict acoustic grammar model to eliminate false positives in noisy environments. • LLM Orchestration: Integration with Anthropic's Claude API (and quantized local models like Llama-3) for context-aware, multi-turn reasoning and agentic decision making. • Biometric Speaker Verification: A custom, zero-dependency audio processing pipeline that computes MFCCs to silently identify and tag individual speakers. • Hardware-Accelerated Computer Vision: Exploiting the OAK-D's VPU for a complex neural pipeline (Face Detection → Head-Pose Estimation → ArcFace Recognition) executed entirely on the sensor. • Visual Agentic Execution: The LLM can autonomously trigger exteroceptive visual capture based on conversational context, enabling dynamic spatial Q&A. • Telemetry Dashboard: A Flask-SocketIO dashboard visualizing internal state vectors, audio spectrograms, and bounding-box data in real-time.

Hardware Foundation: • Compute Node: NVIDIA Jetson Nano Orin (ARM64) • Exteroceptive Sensor: Luxonis OAK-D Stereo Depth Camera • Audio Interface: Waveshare WM8960 Audio HAT

Software & Machine Learning Stack: • Signal Processing: Custom WebRTC Voice Activity Detection (VAD) with RMS energy-based fallbacks. • Transcription & Synthesis: Int8 quantized Faster-Whisper for STT and ONNX-based Piper TTS. • Computer Vision: DepthAI SDK for asynchronous graph execution. • Middleware: Python 3 multi-threading, inter-process communication (IPC), and WebSockets.

1. Preventing Acoustic Feedback (Self-Deafening) Closing and reopening ALSA audio streams to prevent the microphone from recording the agent's TTS output introduced unacceptable latency and race conditions. I engineered a non-blocking thread-safe Event flag injected directly into the high-frequency sounddevice audio callback. Frames are silently dropped at the buffer level when the flag is active, solving feedback with zero latency overhead.

2. Zero-Dependency Audio Feature Extraction Standard Python audio libraries (librosa) introduced massive overhead on ARM64 architectures. I wrote a bespoke biometric extraction pipeline using pure NumPy/SciPy. It executes pre-emphasis filtering, Hamming window framing, FFT power spectrum computation, and Discrete Cosine Transforms (DCT) to generate a 26-dimensional L2-normalized embedding. Identification is performed via optimized cosine-similarity matrix operations, resulting in microsecond execution times.

3. Asynchronous Video Stream Synchronization The OAK-D executes three chained neural networks asynchronously. Naive rendering caused extreme bounding-box flickering due to frame mismatch. I implemented a TwoStageHostSeqSync algorithm that buffers output queues and aligns results purely by sequence number, guaranteeing that rendered annotations perfectly match their corresponding color frames.

S-Bot successfully bridges the gap between fragile software scripts and highly reliable embedded deployments. By prioritizing thread safety, asynchronous I/O, and extreme dependency minimization, the system operates deterministically for extended periods. This project serves as a strong demonstration of software architecture, signal processing, and AI integration for roles in edge computing, autonomous agents, and software engineering.