Submit Chat Completions and Responses without an open connection. Retrieve results later, resume streams, and understand retention.
Async mode lets a worker submit a request and release its connection before generation finishes. Munito continues the request after the connection closes. Another job can retrieve the result or resume its stream. This frees your workers while the model runs. It does not reduce the model's generation time.
Choose an API
| API | Enable async mode | Initial response | Retrieve the result |
|---|---|---|---|
| Chat Completions | Header Prefer: respond-async | HTTP 202, an id, and a Location header | GET /chat/completions/{id} |
| Responses | JSON field background: true | HTTP 200 with status: "queued", unless stream: true | GET /responses/{id} |
These paths use the base URL https://api.munito.ai/inference/v1.
Use the same API host and a key for the same workspace when you retrieve a result.
For a regional host, see Regions.
The Location header contains a path relative to your API host.
OCR also accepts Prefer: respond-async. See PDF to Markdown for its request and result format.
Responses
Submit a request
Set background=True with the OpenAI Python SDK.
The API returns an ID before generation finishes.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.munito.ai/inference/v1",
api_key=os.environ["MUNITO_API_KEY"],
)
response = client.responses.create(
model="qwen/qwen3.8-27b",
input="Summarize the history of the printing press in five sentences.",
max_output_tokens=2048,
background=True,
)
print(response.id)Save response.id in your job record before you release your worker.
Schedule a separate job to retrieve the result.
A loop that waits for completion in the same worker keeps that worker occupied.
Retrieve the result
Set RESPONSE_ID to the ID from submission.
Run this code in the job that retrieves the result:
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.munito.ai/inference/v1",
api_key=os.environ["MUNITO_API_KEY"],
)
response = client.responses.retrieve(os.environ["RESPONSE_ID"])
if response.status == "completed":
print(response.output_text)
elif response.status in ("queued", "in_progress"):
print("Schedule another result check.")
else:
print(response.status, response.error, response.incomplete_details)queued and in_progress mean that the request is not finished.
Schedule another check after a delay, such as two seconds.
Treat failed, cancelled, and incomplete as terminal results that need separate action.
Background Responses store their result for 30 days, even without an explicit store: true.
The combination background: true and store: false returns 400.
Chat Completions
Submit a request
Send Prefer: respond-async as an HTTP header.
Add store: true to retain the result for later retrieval.
curl --fail-with-body --include https://api.munito.ai/inference/v1/chat/completions \
-H "Authorization: Bearer $MUNITO_API_KEY" \
-H "Content-Type: application/json" \
-H "Prefer: respond-async" \
-d '{
"model": "qwen/qwen3.8-27b",
"messages": [{"role": "user", "content": "Write one sentence about Lisbon."}],
"max_tokens": 2048,
"store": true
}'The API returns HTTP 202, an id, and a Location header.
Save the ID or the location before you release your worker.
With the OpenAI Python SDK, pass the header as extra_headers={"Prefer": "respond-async"}.
Retrieve the result
Set COMPLETION_ID to the ID from submission.
curl --fail-with-body --include \
"https://api.munito.ai/inference/v1/chat/completions/${COMPLETION_ID}" \
-H "Authorization: Bearer $MUNITO_API_KEY"HTTP 202 means that generation is not finished.
Schedule another check after a delay.
HTTP 200 returns the completion, with choices and usage.
Check finish_reason and the message content before you use the result.
Handle an error response separately. See Errors.
Resume a stream
A background request continues after a connection closes. A resume request delivers events after the cursor that you supply. It does not start generation again. Replay works during generation and after completion, while the events remain available.
Chat Completions
Create the request with Prefer: respond-async.
Read its stream with GET /chat/completions/{id}?stream=true.
Save each SSE id after your application processes its event.
On reconnection, send the last saved ID in Last-Event-ID:
curl --fail-with-body --no-buffer \
"https://api.munito.ai/inference/v1/chat/completions/${COMPLETION_ID}?stream=true" \
-H "Authorization: Bearer $MUNITO_API_KEY" \
-H "Last-Event-ID: ${LAST_EVENT_ID}"Omit Last-Event-ID on the first connection.
The stream ends with data: [DONE].
Responses
Set both background: true and stream: true when you create the response.
The initial response is an SSE stream, rather than a JSON object with status: "queued".
Save the response ID from the response.created event.
Save each event's sequence_number after your application processes the event.
curl --fail-with-body --no-buffer https://api.munito.ai/inference/v1/responses \
-H "Authorization: Bearer $MUNITO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen3.8-27b",
"input": "Write all integers from 1 through 40, separated by spaces.",
"max_output_tokens": 2048,
"background": true,
"stream": true
}'Set RESPONSE_ID and LAST_SEQUENCE_NUMBER to the saved values.
Resume with starting_after to exclude events through that sequence number:
curl --fail-with-body --no-buffer \
"https://api.munito.ai/inference/v1/responses/${RESPONSE_ID}?stream=true&starting_after=${LAST_SEQUENCE_NUMBER}" \
-H "Authorization: Bearer $MUNITO_API_KEY"A response created without stream: true supports result retrieval, but not stream replay.
See Streaming for event formats, reasoning events, and token usage.
Retention
Stream replay and stored results have separate retention periods.
| Data | Retention |
|---|---|
| Background stream events | Up to one hour. Events can expire earlier if replay capacity is reached. |
Chat result with store: true | 30 days, unless you delete it first. |
| Background Responses result | 30 days, unless you delete it first. |
Chat Completions without store: true depend on the temporary replay window for result retrieval.
Use store: true when another job must retrieve the result later.
If stream replay is unavailable, retrieve the stored result with a normal GET, without stream=true.
The field store: true alone does not enable async mode or stream replay.
Stored requests retain their input and output. They do not use the default zero data retention policy. See Data privacy for storage, regions, and deletion.
Cancellation and retries
To stop a background Responses request, send POST /responses/{id}/cancel.
Cancellation stops generation. It does not pause it for a later resume.
Deleting a stored result is separate from cancellation.
See Cancel a response, Delete a response, and Delete a chat completion.
After the API accepts a request, retrieve it by ID. Do not submit the same request again.
The inference API has no idempotency key. A repeated POST can create another request and another charge.
Handle HTTP errors and respect Retry-After as described in Errors.
Async mode still uses your normal model limits, rate limits, and service tier.
Background requests do not use overflow.
Choose a service tier for your workload; async mode does not select flex automatically.