# Streaming
URL: https://docs.munito.ai/streaming

Set `stream: true` on Chat Completions, Responses or Messages, and the Munito AI API sends the answer as server-sent events while the model writes it.
Each stream ends with the token usage of the request. To stop the model, close the connection.

The OpenAI SDK uses the base URL `https://api.munito.ai/inference/v1`, and the Anthropic SDK uses `https://api.munito.ai/inference`. Both read the key from `MUNITO_API_KEY`.

## Chat Completions [#chat-completions]

```bash
pip install openai
```

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.munito.ai/inference/v1",
    api_key=os.environ["MUNITO_API_KEY"],
)
stream = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
    max_tokens=2048,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices:
        delta = chunk.choices[0].delta
        reasoning = (delta.model_extra or {}).get("reasoning")
        if reasoning:
            print(reasoning, end="", flush=True)
        if delta.content:
            print(delta.content, end="", flush=True)
    elif chunk.usage:
        print("\n", chunk.usage.prompt_tokens, "input tokens,", chunk.usage.completion_tokens, "output tokens")
```

The stream is a sequence of `data:` lines, each with one `chat.completion.chunk`:

| Chunk                 | Content                                                                           |
| --------------------- | --------------------------------------------------------------------------------- |
| The first chunk       | `delta.role` is `assistant`.                                                      |
| Reasoning chunks      | `delta.reasoning` holds the text of the reasoning. It comes before the answer.    |
| Answer chunks         | `delta.content` holds the text of the answer.                                     |
| Tool call chunks      | `delta.tool_calls` holds the parts of each call. Join the `arguments` by `index`. |
| The last choice chunk | `finish_reason` is `stop`, `length` or `tool_calls`.                              |
| The usage chunk       | `choices` is empty, and `usage` holds the token counts of the request.            |
| The end               | `data: [DONE]`                                                                    |

Munito AI always sends the usage chunk, also without `stream_options.include_usage`.
Its `usage.completion_tokens` includes the reasoning tokens. In the stream, `usage.completion_tokens_details.reasoning_tokens` also gives their count.

Each event also has an `id:` line. During a long wait for the first token, Munito AI sends a `: keepalive` comment line every 10 seconds, which SSE clients ignore.
If the generation fails after the stream starts, the last event before `[DONE]` is an error:

```json
{ "error": { "message": "generation failed", "type": "server_error", "param": null, "code": null } }
```

## Chat Completions in TypeScript [#chat-completions-in-typescript]

```bash
npm install openai
```

```ts
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.munito.ai/inference/v1",
  apiKey: process.env.MUNITO_API_KEY,
});
const stream = await client.chat.completions.create({
  model: "qwen/qwen3.8-27b",
  messages: [{ role: "user", content: "Write a haiku about the sea." }],
  max_tokens: 2048,
  stream: true,
});
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta;
  if (delta?.content) process.stdout.write(delta.content);
  if (chunk.usage) console.log(`\n${chunk.usage.completion_tokens} output tokens`);
}
```

## Responses [#responses]

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.munito.ai/inference/v1",
    api_key=os.environ["MUNITO_API_KEY"],
)
stream = client.responses.create(
    model="qwen/qwen3.8-27b",
    input="Write a haiku about the sea.",
    max_output_tokens=2048,
    stream=True,
)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
    elif event.type == "response.completed":
        print("\n", event.response.usage.output_tokens, "output tokens")
```

Each event has an `event:` line with its type, an `id:` line with its `sequence_number`, and a `data:` line.
The events are `response.created`, `response.in_progress`, `response.output_item.added`, the `response.reasoning_text.delta` events of the reasoning, the `response.output_text.delta` events of the answer, the `.done` events, and at the end `response.completed` with the usage.
`stream_options` returns `400` on Responses, because the last event always holds the usage.

## Messages [#messages]

```bash
pip install anthropic
```

```python
import os
from anthropic import Anthropic

client = Anthropic(
    base_url="https://api.munito.ai/inference",
    api_key=os.environ["MUNITO_API_KEY"],
)
with client.messages.stream(
    model="qwen/qwen3.8-27b",
    max_tokens=2048,
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
print("\n", final.usage.output_tokens, "output tokens")
```

The stream has the Anthropic events: `message_start`, then a `content_block_start`, `content_block_delta` events and a `content_block_stop` for each block, then `message_delta` with the `stop_reason` and the usage, and `message_stop`.
The reasoning comes first, as a `thinking` block with `thinking_delta` events. The answer follows as a `text` block with `text_delta` events.

## Cancel a stream [#cancel-a-stream]

Close the connection to stop the generation. Munito AI stops the model, and you pay only for the tokens that it made until then.

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.munito.ai/inference/v1",
    api_key=os.environ["MUNITO_API_KEY"],
)
stream = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "Count from 1 to 1000, one number on each line."}],
    max_tokens=8192,
    reasoning_effort="low",
    stream=True,
)
received = 0
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        received += len(chunk.choices[0].delta.content)
    if received > 200:
        stream.close()
        break
print("stopped after", received, "characters")
```

In TypeScript, call `stream.controller.abort()`, or pass an `AbortSignal` to the request.

## Background requests [#background-requests]

A background request continues when the connection closes, and you can connect to its stream again.

- **Chat Completions:** send the header `Prefer: respond-async`. The API answers `202` with a `Location` header. Send `GET /inference/v1/chat/completions/{id}` with `Accept: text/event-stream` to read the stream, and a `Last-Event-ID` header to continue after the last event that you received.
- **Responses:** send `background: true` and `stream: true`. Send `GET /inference/v1/responses/{id}?stream=true&starting_after=<sequence_number>` to continue the stream, and `POST /inference/v1/responses/{id}/cancel` to stop the request.

See the [chat reference](https://docs.munito.ai/api/createChatCompletion), the [Responses reference](https://docs.munito.ai/api/createResponse) and the [Messages reference](https://docs.munito.ai/api/createMessage).