Orbi

A robot that only moves when you type is a remote control. I wanted speech in, a model that can look, move, and remember, and a page where I can see what it decided.

The problem

The loop had to run without me sitting on a keyboard. It still had to show its work: the words it heard, the tool it called, and the camera frame it used. A voice demo that hides the tool call is a speaker. The person it frustrates is me, later, trying to tell whether the robot moved because of the words or because of a bug in the tool.

It was worth building the page and the tools together. A log without a camera, or a camera without the transcript, is half of the debugging.

How I built it

faster-whisper transcribes on the machine. The audio does not have to leave for the first step, which matters when I am iterating on the loop and do not want a network round trip before I know whether the words were even right.

The model has four tools. It can see a webcam frame. It can send a move command toward an ESP32. It can remember a note in a local file, and it can recall one. Those four are the whole action space. Cloud models are tried in order: Gemini, then gpt-4o-mini, then Claude Haiku, with a local Gemma fallback when the cloud calls do not answer. Speech back out is ElevenLabs, played in the browser.

The page is FastAPI. It shows the log and the camera. A dev flag mocks the hardware so the loop runs on a laptop. The flag is how I can hear the chain without the ESP32 on the desk. It is not how I claim the robot moved.

Challenges and what I learned

The hardware is optional for the loop and mandatory for the claim. With the mock flag on, I can hear the chain on a laptop, and I still do not have a robot that moved. The order of the cloud models is a fallback, not a benchmark. Gemma is there so a dead API does not end the session.