Stream tokens as server-sent events on Chat Completions, Responses and Messages, read the usage, and cancel a stream.
Set stream: true on Chat Completions, Responses or Messages, and the Munito AI API sends the answer as server-sent events while the model writes it.
Each stream ends with the token usage of the request. To stop the model, close the connection.
The OpenAI SDK uses the base URL https://api.munito.ai/inference/v1, and the Anthropic SDK uses https://api.munito.ai/inference. Both read the key from MUNITO_API_KEY.
Chat Completions
pip install openaiimport os
from openai import OpenAI
client = OpenAI(
base_url="https://api.munito.ai/inference/v1",
api_key=os.environ["MUNITO_API_KEY"],
)
stream = client.chat.completions.create(
model="qwen/qwen3.8-27b",
messages=[{"role": "user", "content": "Write a haiku about the sea."}],
max_tokens=2048,
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices:
delta = chunk.choices[0].delta
reasoning = (delta.model_extra or {}).get("reasoning")
if reasoning:
print(reasoning, end="", flush=True)
if delta.content:
print(delta.content, end="", flush=True)
elif chunk.usage:
print("\n", chunk.usage.prompt_tokens, "input tokens,", chunk.usage.completion_tokens, "output tokens")The stream is a sequence of data: lines, each with one chat.completion.chunk:
| Chunk | Content |
|---|---|
| The first chunk | delta.role is assistant. |
| Reasoning chunks | delta.reasoning holds the text of the reasoning. It comes before the answer. |
| Answer chunks | delta.content holds the text of the answer. |
| Tool call chunks | delta.tool_calls holds the parts of each call. Join the arguments by index. |
| The last choice chunk | finish_reason is stop, length or tool_calls. |
| The usage chunk | choices is empty, and usage holds the token counts of the request. |
| The end | data: [DONE] |
Munito AI always sends the usage chunk, also without stream_options.include_usage.
Its usage.completion_tokens includes the reasoning tokens. In the stream, usage.completion_tokens_details.reasoning_tokens also gives their count.
Each event also has an id: line. During a long wait for the first token, Munito AI sends a : keepalive comment line every 10 seconds, which SSE clients ignore.
If the generation fails after the stream starts, the last event before [DONE] is an error:
{
"error": {
"message": "generation failed",
"type": "server_error",
"param": null,
"code": null
}
}Chat Completions in TypeScript
npm install openaiimport OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.munito.ai/inference/v1",
apiKey: process.env.MUNITO_API_KEY,
});
const stream = await client.chat.completions.create({
model: "qwen/qwen3.8-27b",
messages: [{ role: "user", content: "Write a haiku about the sea." }],
max_tokens: 2048,
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta;
if (delta?.content) process.stdout.write(delta.content);
if (chunk.usage) console.log(`\n${chunk.usage.completion_tokens} output tokens`);
}Responses
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.munito.ai/inference/v1",
api_key=os.environ["MUNITO_API_KEY"],
)
stream = client.responses.create(
model="qwen/qwen3.8-27b",
input="Write a haiku about the sea.",
max_output_tokens=2048,
stream=True,
)
for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.completed":
print("\n", event.response.usage.output_tokens, "output tokens")Each event has an event: line with its type, an id: line with its sequence_number, and a data: line.
The events are response.created, response.in_progress, response.output_item.added, the response.reasoning_text.delta events of the reasoning, the response.output_text.delta events of the answer, the .done events, and at the end response.completed with the usage.
stream_options returns 400 on Responses, because the last event always holds the usage.
Messages
pip install anthropicimport os
from anthropic import Anthropic
client = Anthropic(
base_url="https://api.munito.ai/inference",
api_key=os.environ["MUNITO_API_KEY"],
)
with client.messages.stream(
model="qwen/qwen3.8-27b",
max_tokens=2048,
messages=[{"role": "user", "content": "Write a haiku about the sea."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
final = stream.get_final_message()
print("\n", final.usage.output_tokens, "output tokens")The stream has the Anthropic events: message_start, then a content_block_start, content_block_delta events and a content_block_stop for each block, then message_delta with the stop_reason and the usage, and message_stop.
The reasoning comes first, as a thinking block with thinking_delta events. The answer follows as a text block with text_delta events.
Cancel a stream
Close the connection to stop the generation. Munito AI stops the model, and you pay only for the tokens that it made until then.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.munito.ai/inference/v1",
api_key=os.environ["MUNITO_API_KEY"],
)
stream = client.chat.completions.create(
model="qwen/qwen3.8-27b",
messages=[{"role": "user", "content": "Count from 1 to 1000, one number on each line."}],
max_tokens=8192,
reasoning_effort="low",
stream=True,
)
received = 0
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
received += len(chunk.choices[0].delta.content)
if received > 200:
stream.close()
break
print("stopped after", received, "characters")In TypeScript, call stream.controller.abort(), or pass an AbortSignal to the request.
Background requests
A background request continues when the connection closes, and you can connect to its stream again.
- Chat Completions: send the header
Prefer: respond-async. The API answers202with aLocationheader. SendGET /inference/v1/chat/completions/{id}withAccept: text/event-streamto read the stream, and aLast-Event-IDheader to continue after the last event that you received. - Responses: send
background: trueandstream: true. SendGET /inference/v1/responses/{id}?stream=true&starting_after=<sequence_number>to continue the stream, andPOST /inference/v1/responses/{id}/cancelto stop the request.
See the chat reference, the Responses reference and the Messages reference.