Real-time Speech-to-Text

Transcribe speech to text in real-time using a persistent Socket.IO connection. Audio is streamed from the client to the server, and transcription results are returned as they become available.

Recognition runs on cantonese.ai's own Cantonese speech recognition model and outputs Traditional Chinese. The server detects pauses itself: an utterance is committed after about 0.7 seconds of silence (or after 20 seconds of continuous speech), so there is nothing to configure. Usage is billed at 2 credits per second of audio sent (minimum 3 seconds per session).

How It Works

1. Connect to the Socket.IO endpoint at /api/stt-realtime-socket with your API key.
2. Emit a start-session event to begin a transcription session.
3. Stream audio by emitting audio-chunk events with base64-encoded PCM audio data.
4. Receive transcription results via the transcript-message event.
5. Emit stop-session to end the session.

Connection

Connect using a Socket.IO client with the following configuration:

Client Events (Client → Server)

Server Events (Server → Client)

Transcript Message Types

The transcript-message event payload contains a message_type field indicating the type of transcription result:

Audio Format

Audio must be sent as base64-encoded 16-bit PCM (signed integer, little-endian). The recommended sample rate is 16000 Hz, mono channel. If your audio source uses a different sample rate, pass the actual sample rate in the start-session and audio-chunk events.

Example

const { io } = require("socket.io-client");

const API_KEY = "YOUR_API_KEY";
const BASE_URL = "https://cantonese.ai";

// Connect to the real-time STT socket
const socket = io(BASE_URL, {
  path: "/api/stt-realtime-socket",
  extraHeaders: {
    "x-api-key": API_KEY,
  },
});

socket.on("connect", () => {
  console.log("Connected to real-time STT");
  // Start a transcription session
  // endSilenceS (optional, 0.2-5, default 2): seconds of silence that end an utterance
  socket.emit("start-session", { sampleRate: 16000, endSilenceS: 1 });
});

socket.on("session-started", () => {
  console.log("Session started, ready to receive audio");
  // Send audio chunks (base64-encoded PCM)
  // socket.emit("audio-chunk", {
  //   audio_base_64: "<base64-encoded-pcm-audio>",
  //   sample_rate: 16000,
  // });
});

socket.on("transcript-message", (message) => {
  switch (message.message_type) {
    case "partial_transcript":
      process.stdout.write(`\rPartial: ${message.text}`);
      break;
    case "committed_transcript":
      console.log(`\nFinal [${message.start_s}s - ${message.end_s}s]: ${message.text}`);
      break;
  }
});

socket.on("session-error", (data) => {
  console.error("Session error:", data.error);
});

socket.on("session-closed", (data) => {
  console.log("Session closed:", data.code, data.reason, `${data.audio_seconds}s of audio`);
});

// To stop the session (pending utterances are committed, then session-closed arrives):
// socket.emit("stop-session");
// ...and call socket.disconnect() in the session-closed handler.

Status Codes

A Socket.IO connection without a valid API key (or signed-in browser session) is refused with aconnect_error whose message is Unauthorized.

The Socket.IO handshake endpoint returns standard HTTP status codes: