OpenAI Realtime (Speech-to-Speech)

Build a voice agent using OpenAI’s GPT-4o Realtime API for native speech-to-speech processing. OpenAI Realtime processes audio directly without intermediate text conversion, delivering the lowest latency voice conversations. Best for: Applications requiring minimal latency and native multimodal AI capabilities.

How Speech-to-Speech Differs

Standard Pipeline (STT → LLM → TTS):

Standard pipelineAudio in · Audio out

DeepgramSpeech-to-Text
Text
OpenAILLM
Text
CartesiaText-to-Speech

Speech-to-Speech (Direct):

Speech-to-speechAudio in · Audio out

OpenAI RealtimeDirect audio-to-audio

Speech-to-speech models process audio natively, preserving tone, emotion, and context that may be lost in text transcription.

Prerequisites

Service	What You Need
Plivo	Auth ID, Auth Token, Voice-enabled phone number
OpenAI	API key from platform.openai.com with Realtime API access

Installation

pip install "pipecat-ai[openai]"

Environment Variables

# Plivo credentials
PLIVO_AUTH_ID=your_auth_id
PLIVO_AUTH_TOKEN=your_auth_token
PLIVO_PHONE_NUMBER=+1234567890

# OpenAI credentials
OPENAI_API_KEY=sk-your_openai_key

Pipeline Configuration

from pipecat.services.openai import OpenAIRealtimeLLMService

# Speech-to-Speech service
llm = OpenAIRealtimeLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    # voice="alloy",  # Choose voice: alloy, echo, fable, onyx, nova, shimmer
)

OpenAI Realtime Features

Feature	Description
Minimal latency	Direct audio processing for fastest response times
Voice activity detection	Multiple VAD options including semantic-based
Function calling	Seamless integration with external APIs
Multiple voices	Choose from built-in voice personalities
Context management	Advanced conversation flow handling

Available Voices

Voice	Description
`alloy`	Neutral, balanced
`echo`	Warm, friendly
`fable`	Expressive, storytelling
`onyx`	Deep, authoritative
`nova`	Bright, energetic
`shimmer`	Clear, professional

Architecture

With OpenAI Realtime, the pipeline is simplified:

ArchitectureOpenAI Realtime

Phone Call
Plivo
WebSocket
Pipecat
OpenAI Realtime

A single service handles:

Speech recognition
Language understanding
Response generation
Voice synthesis

Quick Start

Inbound Calls

git clone https://github.com/pipecat-ai/pipecat-examples.git
cd pipecat-examples/plivo-chatbot/inbound

# Configure environment
cp env.example .env
# Edit .env with Plivo and OpenAI credentials

# Modify bot.py to use OpenAIRealtimeLLMService
# Start server
uv sync && uv run server.py

# Expose with ngrok (development)
ngrok http 7860

Configure your Plivo number’s Answer URL to your ngrok URL.

Outbound Calls

cd pipecat-examples/plivo-chatbot/outbound

cp env.example .env
uv sync && uv run server.py

# Initiate a call
curl -X POST http://localhost:7860/start \
  -H "Content-Type: application/json" \
  -d '{"phone_number": "+1234567890"}'

When to Use OpenAI Realtime

Choose OpenAI Realtime when:

Latency is your top priority
You want the simplest integration
Built-in voices meet your needs
You’re already using OpenAI

Choose standard STT → LLM → TTS when:

You need specific voice characteristics (ElevenLabs cloning, Cartesia emotion)
You want to mix providers for cost optimization
You need fine-grained control over each component

Pipecat Overview

Architecture and setup

OpenAI Realtime Docs

Full configuration

OpenAI Realtime Guide

Official documentation

Concepts

Integration Guides

Workflows

Deploy

Reference

OpenAI Realtime (Speech-to-Speech)

How Speech-to-Speech Differs

Prerequisites

Installation

Environment Variables

Pipeline Configuration

OpenAI Realtime Features

Available Voices

Architecture

Quick Start

Inbound Calls

Outbound Calls

When to Use OpenAI Realtime

Pipecat Overview

OpenAI Realtime Docs

OpenAI Realtime Guide

​How Speech-to-Speech Differs

​Prerequisites

​Installation

​Environment Variables

​Pipeline Configuration

​OpenAI Realtime Features

​Available Voices

​Architecture

​Quick Start

​Inbound Calls

​Outbound Calls

​When to Use OpenAI Realtime

​Related

Pipecat Overview

OpenAI Realtime Docs

OpenAI Realtime Guide

How Speech-to-Speech Differs

Prerequisites

Installation

Environment Variables

Pipeline Configuration

OpenAI Realtime Features

Available Voices

Architecture

Quick Start

Inbound Calls

Outbound Calls

When to Use OpenAI Realtime

Related