STT for Agents

This page describes the WebSocket protocol for Krisp streaming speech-to-text (STT) with the krisp-realtime-stt-agents-v1 model, built for voice AI agents. It returns fast partial results and predicts the end of the user's turn.

Endpoint

• WebSocket URL: wss://streaming.krisp.ai/stt
• Auth: temporary session token. See Authentication.

Use the session token as the API key when opening the WebSocket. Do not send the master token on the WebSocket.

Authorization: api-key <session-token>

If your client cannot set headers, pass the same value as the authorization query parameter. See the API overview.

Connection and handshake flow

1) Client initiates connection + sends handshake message

After opening the WebSocket connection with the session token, the client must send an initial JSON handshake that defines the session.

Handshake message (client → server, JSON):

{
  "model": "krisp-realtime-stt-agents-v1",
  "vocabulary": ["word1", "word2"],
  "sample_rate": 16000,
  "channels": 1,
  "audio_format": "pcm_s16le",
  "turn": {
    "prediction": true,
    "sensitivity": 0.9,
    "max_silence": 2000,
    "silence_validation": 100
  },
  "language_hints": ["en"],
  "metadata": {}
}

2) Server acknowledges readiness (or returns an error)

Once the server validates the handshake and is ready to receive audio, it responds with a ready message.

Ready response (server → client, JSON):

{
  "type": "ready",
  "request_id": "9f2c...",
  "metadata": {}
}

• request_id: unique identifier for the streaming session. Include it if you contact Krisp support.
• metadata: echo of the metadata object from the handshake.

Do not send audio before this message. If no handshake is received within 15 seconds, the server closes the session.

Error response

If the server cannot accept the session, it responds with an error message and closes the connection. Besides the errors below, the common errors apply.

Error response (server → client, JSON):

{
  "type": "error",
  "request_id": "9f2c...",
  "error": {
    "code": "invalid_api_key",
    "status_code": 401,
    "message": "Invalid Authorization header.",
    "details": {
      "hint": "Check authorization header."
    }
  }
}

Audio streaming

3) Client streams audio frames (raw bytes)

After receiving type: "ready", the client sends raw audio bytes as binary WebSocket messages.

• Message type: binary (raw bytes)
• Encoding: according to audio_format in the handshake: pcm_s16le (PCM signed 16-bit little-endian) or opus
• Sample rate: 16,000 Hz
• Channels: 1 (mono)
• Recommended chunk duration: 80 ms, or multiples of 80 ms
• Maximum message size: 65,536 bytes

Stream audio in realtime. Do not wrap audio in JSON.

Server frames

Every server message is a JSON text frame, and all frames carry a type. Finals and partials are both transcript frames and differ in transcript.isFinal.

All times are milliseconds of audio from the session's first sample, including in turn frames. The only wall-clock value is a turn frame's processing_duration, also in milliseconds.

4) Final transcript: isFinal: true

{
  "type": "transcript",
  "metadata": {},
  "transcript": {
    "words": [
      {
        "text": " weather",
        "start": 12480,
        "end": 12880,
        "language": "en",
        "language_confidence": 0.9921
      }
    ],
    "isFinal": true
  }
}

• words holds one entry: a whole word with turn prediction, a word piece without it.
• text keeps its leading space, so joining text in arrival order gives the transcript.
• start and end bound the word. language and language_confidence are present when the server runs language ID (labels: see Supported languages).
• metadata is always empty.
• A final never changes and is never repeated.
• Finals trail speech by about 2.4 s, except at a turn end, when the turn's pending words arrive at once.

5) Partial transcript: isFinal: false

{
  "type": "transcript",
  "metadata": {},
  "transcript": {
    "words": [
      { "text": " is",    "start": 13040, "end": 13200, "language": "en", "language_confidence": 0.9934 },
      { "text": " nice",  "start": 13200, "end": 13600, "language": "en", "language_confidence": 0.9917 },
      { "text": " today", "start": 13600, "end": 14000, "language": "en", "language_confidence": 0.9917 }
    ],
    "isFinal": false
  }
}

• words is the whole partial region: everything after the last final, one entry per token piece (a whole word or part of one, with a leading space only at a word start). Join text the same way as finals.
• Each partial frame replaces all earlier partial content, and words: [] clears it.
• Show the finals followed by the latest partial frame.
• Partial text may change or disappear in the next frame. language and language_confidence are present as on finals, except on a piece whose audio has not been labelled yet.
• While there is speech, a partial frame is sent every 240 ms, even when its text did not change. In silence, a partial frame is sent only when finals overtook part of the region, and one more frame clears it.

6) Turn: type: "turn"

{
  "type": "turn",
  "confidence": 0.934,
  "reason": "model",
  "detected_at": 14560,
  "finalized_until": 14640,
  "processing_duration": 95
}

At a turn end the server sends the turn's remaining finals, then the turn frame, then an empty partial frame (words: []). The model or the timeout ends at most one turn per utterance: a new turn needs at least 0.18 s of new speech, and a model or timeout turn with no new text since the previous turn sends no turn frame.

7) Force finalization

Force the end of a turn at any point during the stream:

{"type": "finalize"}

The server sends the pending words as finals, then a turn frame with reason: "client". Every finalize is answered by exactly one such turn frame, in request order, even when nothing is pending. Without turn prediction (turn.prediction: false) it is ignored.

8) Stream end

To end the stream, the client sends one text frame after its last audio message:

{"type": "end_of_audio"}

The server then sends the remaining finals in one frame, an empty partial frame (words: []) and end, and closes the socket itself. A turn that the end of the stream overtook arrives before end, with "eof": true. The client shouldn't close the socket; it should read frames until end arrives.

After end_of_audio, the last frame the server sends is end, and it closes the socket right after:

{
  "type": "end",
  "language": {
    "overall_language": "en",
    "overall_confidence": 0.982,
    "current_language": "en",
    "current_confidence": 0.991,
    "per_language": {
      "de": 0.003,
      "en": 0.982,
      "es": 0.004,
      "fr": 0.002,
      "it": 0.001,
      "nl": 0.001,
      "pt": 0.001,
      "zh-CN": 0.001,
      "ru": 0.003,
      "hy-AM": 0.002
    }
  }
}

The last frame of a session. language is present when the server runs language ID:

• overall_* averages the whole session.
• current_* covers the last 0.5 s.
• per_language holds the mean probability of every language ID label over the session (all ten, see Supported languages).

Supported languages

The model transcribes the ten languages below. Output is limited to Latin, Cyrillic, Armenian and Chinese characters, so languages written in other scripts are not transcribed. The language ID labels are the values of words[].language and the keys of end.language.per_language; language_hints takes the bare code.

Several hints allow the union of their scripts. Hints covering Cyrillic, Armenian and Chinese together, an unknown code or an empty list allow every script. Language ID only knows these ten labels: speech in another language is labelled as the closest of them.

Example (Python)

Streams a 16 kHz, mono, 16-bit PCM WAV file at realtime pace and prints finals, partials and turn events. Requires pip install "websockets>=14".

import asyncio
import json
import sys
import wave

import websockets

SESSION_TOKEN = "YOUR_SESSION_TOKEN"  # see the Authentication page
URL = "wss://streaming.krisp.ai/stt"
CHUNK_MS = 80


async def main(path: str) -> None:
    headers = {"Authorization": f"api-key {SESSION_TOKEN}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "model": "krisp-realtime-stt-agents-v1",
            "sample_rate": 16000,
            "channels": 1,
            "audio_format": "pcm_s16le",
            "turn": {"prediction": True, "sensitivity": 0.9, "max_silence": 2000},
        }))
        hello = json.loads(await ws.recv())
        if hello["type"] != "ready":
            raise RuntimeError(hello)

        async def send_audio() -> None:
            with wave.open(path, "rb") as wav:
                frames = 16000 * CHUNK_MS // 1000
                while chunk := wav.readframes(frames):
                    await ws.send(chunk)
                    await asyncio.sleep(CHUNK_MS / 1000)
            await ws.send(json.dumps({"type": "end_of_audio"}))

        sender = asyncio.create_task(send_audio())
        async for message in ws:
            msg = json.loads(message)
            if msg["type"] == "transcript":
                kind = "final" if msg["transcript"]["isFinal"] else "partial"
                text = "".join(w["text"] for w in msg["transcript"]["words"])
                print(kind, text)
            elif msg["type"] == "turn":
                print("end of turn", msg["reason"], msg["confidence"])
            elif msg["type"] == "end":
                break
        await sender


asyncio.run(main(sys.argv[1]))

Changelog

• 2026-10-01: Page created.