Realtime API

Note

Before proceeding, you should be familiar with the OpenAI Realtime API and the relevant OpenAI API reference

Prerequisites

Follow the prerequisites in the Text-to-Speech guide. And set the following environmental variables:

• CHAT_COMPLETION_BASE_URL to the base URL of an OpenAI API compatible endpoint | Config
• CHAT_COMPLETION_API_KEY if the API you are using requires authentication | Config

Demo

TODO

Limitations

• "response.cancel" and "conversation.item.truncate" client events are not supported
• "conversation.item.create" with content field containing input_audio message is not supported

Architecture

TODO

Next Steps

• Image support
• Speech-to-speech model support
• Optimizations and feature