# Async mode
URL: https://docs.munito.ai/async

Async mode lets a worker submit a request and release its connection before generation finishes.
Munito continues the request after the connection closes.
Another job can retrieve the result or resume its stream.
This frees your workers while the model runs. It does not reduce the model's generation time.

## Choose an API [#choose-an-api]

| API              | Enable async mode              | Initial response                                          | Retrieve the result          |
| ---------------- | ------------------------------ | --------------------------------------------------------- | ---------------------------- |
| Chat Completions | Header `Prefer: respond-async` | HTTP `202`, an `id`, and a `Location` header              | `GET /chat/completions/{id}` |
| Responses        | JSON field `background: true`  | HTTP `200` with `status: "queued"`, unless `stream: true` | `GET /responses/{id}`        |

These paths use the base URL `https://api.munito.ai/inference/v1`.
Use the same API host and a key for the same workspace when you retrieve a result.
For a regional host, see [Regions](https://docs.munito.ai/regions).
The `Location` header contains a path relative to your API host.

OCR also accepts `Prefer: respond-async`. See [PDF to Markdown](https://docs.munito.ai/pdf-to-markdown) for its request and result format.

## Responses [#responses]

### Submit a request [#submit-a-request]

Set `background=True` with the [OpenAI Python SDK](https://docs.munito.ai/quickstart).
The API returns an ID before generation finishes.

```python
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.munito.ai/inference/v1",
    api_key=os.environ["MUNITO_API_KEY"],
)
response = client.responses.create(
    model="qwen/qwen3.8-27b",
    input="Summarize the history of the printing press in five sentences.",
    max_output_tokens=2048,
    background=True,
)
print(response.id)
```

Save `response.id` in your job record before you release your worker.
Schedule a separate job to retrieve the result.
A loop that waits for completion in the same worker keeps that worker occupied.

### Retrieve the result [#retrieve-the-result]

Set `RESPONSE_ID` to the ID from submission.
Run this code in the job that retrieves the result:

```python
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.munito.ai/inference/v1",
    api_key=os.environ["MUNITO_API_KEY"],
)
response = client.responses.retrieve(os.environ["RESPONSE_ID"])

if response.status == "completed":
    print(response.output_text)
elif response.status in ("queued", "in_progress"):
    print("Schedule another result check.")
else:
    print(response.status, response.error, response.incomplete_details)
```

`queued` and `in_progress` mean that the request is not finished.
Schedule another check after a delay, such as two seconds.
Treat `failed`, `cancelled`, and `incomplete` as terminal results that need separate action.

Background Responses store their result for 30 days, even without an explicit `store: true`.
The combination `background: true` and `store: false` returns `400`.

## Chat Completions [#chat-completions]

### Submit a request [#submit-a-request-1]

Send `Prefer: respond-async` as an HTTP header.
Add `store: true` to retain the result for later retrieval.

```bash
curl --fail-with-body --include https://api.munito.ai/inference/v1/chat/completions \
  -H "Authorization: Bearer $MUNITO_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Prefer: respond-async" \
  -d '{
    "model": "qwen/qwen3.8-27b",
    "messages": [{"role": "user", "content": "Write one sentence about Lisbon."}],
    "max_tokens": 2048,
    "store": true
  }'
```

The API returns HTTP `202`, an `id`, and a `Location` header.
Save the ID or the location before you release your worker.
With the OpenAI Python SDK, pass the header as `extra_headers={"Prefer": "respond-async"}`.

### Retrieve the result [#retrieve-the-result-1]

Set `COMPLETION_ID` to the ID from submission.

```bash
curl --fail-with-body --include \
  "https://api.munito.ai/inference/v1/chat/completions/${COMPLETION_ID}" \
  -H "Authorization: Bearer $MUNITO_API_KEY"
```

HTTP `202` means that generation is not finished.
Schedule another check after a delay.
HTTP `200` returns the completion, with `choices` and `usage`.
Check `finish_reason` and the message content before you use the result.
Handle an error response separately. See [Errors](https://docs.munito.ai/errors).

## Resume a stream [#resume-a-stream]

A background request continues after a connection closes.
A resume request delivers events after the cursor that you supply.
It does not start generation again.
Replay works during generation and after completion, while the events remain available.

### Chat Completions [#chat-completions-1]

Create the request with `Prefer: respond-async`.
Read its stream with `GET /chat/completions/{id}?stream=true`.
Save each SSE `id` after your application processes its event.
On reconnection, send the last saved ID in `Last-Event-ID`:

```bash
curl --fail-with-body --no-buffer \
  "https://api.munito.ai/inference/v1/chat/completions/${COMPLETION_ID}?stream=true" \
  -H "Authorization: Bearer $MUNITO_API_KEY" \
  -H "Last-Event-ID: ${LAST_EVENT_ID}"
```

Omit `Last-Event-ID` on the first connection.
The stream ends with `data: [DONE]`.

### Responses [#responses-1]

Set both `background: true` and `stream: true` when you create the response.
The initial response is an SSE stream, rather than a JSON object with `status: "queued"`.
Save the response ID from the `response.created` event.
Save each event's `sequence_number` after your application processes the event.

```bash
curl --fail-with-body --no-buffer https://api.munito.ai/inference/v1/responses \
  -H "Authorization: Bearer $MUNITO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.8-27b",
    "input": "Write all integers from 1 through 40, separated by spaces.",
    "max_output_tokens": 2048,
    "background": true,
    "stream": true
  }'
```

Set `RESPONSE_ID` and `LAST_SEQUENCE_NUMBER` to the saved values.
Resume with `starting_after` to exclude events through that sequence number:

```bash
curl --fail-with-body --no-buffer \
  "https://api.munito.ai/inference/v1/responses/${RESPONSE_ID}?stream=true&starting_after=${LAST_SEQUENCE_NUMBER}" \
  -H "Authorization: Bearer $MUNITO_API_KEY"
```

A response created without `stream: true` supports result retrieval, but not stream replay.
See [Streaming](https://docs.munito.ai/streaming) for event formats, reasoning events, and token usage.

## Retention [#retention]

Stream replay and stored results have separate retention periods.

| Data                           | Retention                                                                |
| ------------------------------ | ------------------------------------------------------------------------ |
| Background stream events       | Up to one hour. Events can expire earlier if replay capacity is reached. |
| Chat result with `store: true` | 30 days, unless you delete it first.                                     |
| Background Responses result    | 30 days, unless you delete it first.                                     |

Chat Completions without `store: true` depend on the temporary replay window for result retrieval.
Use `store: true` when another job must retrieve the result later.
If stream replay is unavailable, retrieve the stored result with a normal `GET`, without `stream=true`.
The field `store: true` alone does not enable async mode or stream replay.

Stored requests retain their input and output. They do not use the default zero data retention policy.
See [Data privacy](https://docs.munito.ai/data-privacy) for storage, regions, and deletion.

## Cancellation and retries [#cancellation-and-retries]

To stop a background Responses request, send `POST /responses/{id}/cancel`.
Cancellation stops generation. It does not pause it for a later resume.
Deleting a stored result is separate from cancellation.
See [Cancel a response](https://docs.munito.ai/api/cancelResponse), [Delete a response](https://docs.munito.ai/api/deleteResponse), and [Delete a chat completion](https://docs.munito.ai/api/deleteChatCompletion).

After the API accepts a request, retrieve it by ID. Do not submit the same request again.
The inference API has no idempotency key. A repeated `POST` can create another request and another charge.
Handle HTTP errors and respect `Retry-After` as described in [Errors](https://docs.munito.ai/errors).

Async mode still uses your normal model limits, rate limits, and service tier.
Background requests do not use [overflow](https://docs.munito.ai/overflow).
Choose a [service tier](https://docs.munito.ai/service-tiers) for your workload; async mode does not select `flex` automatically.