Spent most of the long weekend with a pot of coffee and my laptop open to OpenAI's new Realtime API docs instead of doing anything useful around the house. My partner is not thrilled about the state of the garage right now, for what its worth.
If you missed it, OpenAI ran their DevDay thing last Tuesday (Oct 1) in San Francisco, no livestream this year, just in-person sessions, which I thought was a weird choice for a company that usually loves broadcasting everything. Buried under the headline stuff (GPT-4o updates, o1 chatter) was the Realtime API, which lets you open a WebSocket straight to a voice model and just talk to it. No separate whisper-transcribe-then-TTS pipeline. You send audio in, you get audio back, low latency, same connection the whole time.
I've been bolting together voice assistants the old dumb way for about a year now - record audio, ship it to Whisper, get text, ship the text to GPT-4o, get a reply, ship that to a TTS model, wait for the file, play it back. Each hop adds a few hundred milliseconds and the whole thing always sounded like talking to someone on a very bad satellite phone call. The gaps kill any illusion that youre having a conversation with something.
So Saturday morning I gutted my little home-assistant side project and pointed it at the new endpoint instead. Model's called gpt-4o-realtime-preview, its still labeled beta, and you talk to it over a websocket sending base64-encoded audio chunks and JSON events back and forth. Honestly the docs are rougher than Id like for something this new, theres a decent amount of trial and error figuring out how the session config, the voice activity detection, and the function-calling events all interact. I burned about forty minutes just figuring out why my mic input kept getting cut off mid-sentence before I realized their VAD settings default to something way too aggressive for a slightly noisy room.
Once it worked though, it worked well. The latency is the whole point and it delivers, responses start coming back in maybe half a second, and because its actually one continuous audio model rather than three stitched-together services, the voice doesnt have that flat robotic cadence you get from most TTS. It laughs a little weird sometimes, kind of a hiccup-y "ha," which is unsettling in a way I cant fully explain.
Now the part that'll matter to anyone actually trying to ship something with this: its not cheap. Audio input runs around six cents a minute, audio output is about twenty-four cents a minute, and thats on top of whatever text tokens you push through the same session for function calls or system prompts. A ten minute back-and-forth conversation could run you a few dollars without even trying. For a hobby project thats fine, I dropped maybe four bucks total messing around this weekend, but I dont see how anyone builds a consumer voice product on this pricing without either eating a huge loss per user or charging a subscription that scares people off. Twilio and the phone-tree voice bot companies have to be watching this closely.
The other DevDay announcement worth mentioning, since I know it'll get buried under all the Realtime API noise: prompt caching. If your prompt (system instructions, tool definitions, whatever) is over 1,024 tokens and you keep reusing the same prefix, OpenAI now automatically caches it and charges half price on the cached portion. No config needed, it just kicks in. For anyone running an agent with a big fixed system prompt and short user turns, thats a real cost cut with zero code changes, which is rare enough that I actually went and checked our billing dashboard at work Monday morning to confirm it was applying.
Anyway. If youre building anything voice-related Id say go try the Realtime API this week while its still got that new-toy feeling and before everyone's rate limits get squeezed under demand. Just budget for the audio costs and dont expect the docs to hold your hand. I'm going to let mine sit for a bit and actually go deal with that garage.