> ## Documentation Index
> Fetch the complete documentation index at: https://plivo.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI Realtime (Speech-to-Speech)

> Build voice agents with OpenAI GPT-4o Realtime for direct speech-to-speech processing

Build a voice agent using OpenAI's GPT-4o Realtime API for native speech-to-speech processing. OpenAI Realtime processes audio directly without intermediate text conversion, delivering the lowest latency voice conversations.

**Best for:** Applications requiring minimal latency and native multimodal AI capabilities.

***

## How Speech-to-Speech Differs

**Standard Pipeline (STT → LLM → TTS):**

<div className="sipflow">
  <div className="sipflow__head">
    <span>Standard pipeline</span>
    <span className="sipflow__head-right"><span className="sipflow__led" aria-hidden="true" />Audio in · Audio out</span>
  </div>

  <ol className="sipflow__body">
    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">Deepgram</span><span className="sipflow__sub">Speech-to-Text</span></span></div>
      <div className="sipflow__link"><span className="sipflow__rail" aria-hidden="true" /><span className="sipflow__edge-label">Text</span><span className="sipflow__arrow" aria-hidden="true" /></div>
    </li>

    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">OpenAI</span><span className="sipflow__sub">LLM</span></span></div>
      <div className="sipflow__link"><span className="sipflow__rail" aria-hidden="true" /><span className="sipflow__edge-label">Text</span><span className="sipflow__arrow" aria-hidden="true" /></div>
    </li>

    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">Cartesia</span><span className="sipflow__sub">Text-to-Speech</span></span></div>
    </li>
  </ol>
</div>

**Speech-to-Speech (Direct):**

<div className="sipflow">
  <div className="sipflow__head">
    <span>Speech-to-speech</span>
    <span className="sipflow__head-right"><span className="sipflow__led" aria-hidden="true" />Audio in · Audio out</span>
  </div>

  <ol className="sipflow__body">
    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">OpenAI Realtime</span><span className="sipflow__sub">Direct audio-to-audio</span></span></div>
    </li>
  </ol>
</div>

Speech-to-speech models process audio natively, preserving tone, emotion, and context that may be lost in text transcription.

***

## Prerequisites

| Service    | What You Need                                                                            |
| ---------- | ---------------------------------------------------------------------------------------- |
| **Plivo**  | Auth ID, Auth Token, Voice-enabled phone number                                          |
| **OpenAI** | API key from [platform.openai.com](https://platform.openai.com) with Realtime API access |

***

## Installation

```bash theme={null}
pip install "pipecat-ai[openai]"
```

***

## Environment Variables

```env theme={null}
# Plivo credentials
PLIVO_AUTH_ID=your_auth_id
PLIVO_AUTH_TOKEN=your_auth_token
PLIVO_PHONE_NUMBER=+1234567890

# OpenAI credentials
OPENAI_API_KEY=sk-your_openai_key
```

***

## Pipeline Configuration

```python theme={null}
from pipecat.services.openai import OpenAIRealtimeLLMService

# Speech-to-Speech service
llm = OpenAIRealtimeLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    # voice="alloy",  # Choose voice: alloy, echo, fable, onyx, nova, shimmer
)
```

***

## OpenAI Realtime Features

| Feature                      | Description                                        |
| ---------------------------- | -------------------------------------------------- |
| **Minimal latency**          | Direct audio processing for fastest response times |
| **Voice activity detection** | Multiple VAD options including semantic-based      |
| **Function calling**         | Seamless integration with external APIs            |
| **Multiple voices**          | Choose from built-in voice personalities           |
| **Context management**       | Advanced conversation flow handling                |

### Available Voices

| Voice     | Description              |
| --------- | ------------------------ |
| `alloy`   | Neutral, balanced        |
| `echo`    | Warm, friendly           |
| `fable`   | Expressive, storytelling |
| `onyx`    | Deep, authoritative      |
| `nova`    | Bright, energetic        |
| `shimmer` | Clear, professional      |

***

## Architecture

With OpenAI Realtime, the pipeline is simplified:

<div className="sipflow sipflow--vertical">
  <div className="sipflow__head">
    <span>Architecture</span>
    <span className="sipflow__head-right"><span className="sipflow__led" aria-hidden="true" />OpenAI Realtime</span>
  </div>

  <ol className="sipflow__body">
    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">Phone Call</span></span></div>

      <div className="sipflow__link sipflow__link--bi" aria-hidden="true">
        <span className="sipflow__arrow sipflow__arrow--back" />

        <span className="sipflow__rail" />

        <span className="sipflow__arrow" />
      </div>
    </li>

    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">Plivo</span></span></div>

      <div className="sipflow__link sipflow__link--bi" aria-hidden="true">
        <span className="sipflow__arrow sipflow__arrow--back" />

        <span className="sipflow__rail" />

        <span className="sipflow__arrow" />
      </div>
    </li>

    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">WebSocket</span></span></div>

      <div className="sipflow__link sipflow__link--bi" aria-hidden="true">
        <span className="sipflow__arrow sipflow__arrow--back" />

        <span className="sipflow__rail" />

        <span className="sipflow__arrow" />
      </div>
    </li>

    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">Pipecat</span></span></div>

      <div className="sipflow__link sipflow__link--bi" aria-hidden="true">
        <span className="sipflow__arrow sipflow__arrow--back" />

        <span className="sipflow__rail" />

        <span className="sipflow__arrow" />
      </div>
    </li>

    <li>
      <div className="sipflow__node"><span className="sipflow__node-body"><span className="sipflow__title">OpenAI Realtime</span></span></div>
    </li>
  </ol>
</div>

A single service handles:

* Speech recognition
* Language understanding
* Response generation
* Voice synthesis

***

## Quick Start

### Inbound Calls

```bash theme={null}
git clone https://github.com/pipecat-ai/pipecat-examples.git
cd pipecat-examples/plivo-chatbot/inbound

# Configure environment
cp env.example .env
# Edit .env with Plivo and OpenAI credentials

# Modify bot.py to use OpenAIRealtimeLLMService
# Start server
uv sync && uv run server.py

# Expose with ngrok (development)
ngrok http 7860
```

Configure your Plivo number's Answer URL to your ngrok URL.

### Outbound Calls

```bash theme={null}
cd pipecat-examples/plivo-chatbot/outbound

cp env.example .env
uv sync && uv run server.py

# Initiate a call
curl -X POST http://localhost:7860/start \
  -H "Content-Type: application/json" \
  -d '{"phone_number": "+1234567890"}'
```

***

## When to Use OpenAI Realtime

**Choose OpenAI Realtime when:**

* Latency is your top priority
* You want the simplest integration
* Built-in voices meet your needs
* You're already using OpenAI

**Choose standard STT → LLM → TTS when:**

* You need specific voice characteristics (ElevenLabs cloning, Cartesia emotion)
* You want to mix providers for cost optimization
* You need fine-grained control over each component

***

## Related

<CardGroup cols={2}>
  <Card title="Pipecat Overview" icon="https://mintcdn.com/plivo/rP0r53Ttc3NtJi4l/images/pipecat-logo.svg?fit=max&auto=format&n=rP0r53Ttc3NtJi4l&q=85&s=a479c973bdfcce8625fa3050a825bff7" href="/docs/voice-agents/audio-streaming/integration-guides/pipecat/overview" width="24" height="24" data-path="images/pipecat-logo.svg">
    Architecture and setup
  </Card>

  <Card title="OpenAI Realtime Docs" icon="https://mintcdn.com/plivo/tp9S4VDTfvUYlINm/images/openai-logo.svg?fit=max&auto=format&n=tp9S4VDTfvUYlINm&q=85&s=96340c4c16e79846fbc74bd30a3db071" href="https://docs.pipecat.ai/api-reference/server/services/s2s/openai" arrow={true} width="2406" height="2406" data-path="images/openai-logo.svg">
    Full configuration
  </Card>

  <Card title="OpenAI Realtime Guide" icon="https://mintcdn.com/plivo/tp9S4VDTfvUYlINm/images/openai-logo.svg?fit=max&auto=format&n=tp9S4VDTfvUYlINm&q=85&s=96340c4c16e79846fbc74bd30a3db071" href="https://developers.openai.com/api/docs/guides/realtime" arrow={true} width="2406" height="2406" data-path="images/openai-logo.svg">
    Official documentation
  </Card>
</CardGroup>