# Retrieval-augmented generation
URL: https://docs.munito.ai/rag

To build retrieval-augmented generation (RAG) with Munito AI, read your documents with `POST /ocr`, cut the Markdown into chunks, find the best chunks for a question, and send them to a chat model.
The model answers only from those chunks and cites the page of each claim.

## How it works [#how-it-works]

| Step                         | Where it runs | Tool                                                                                    |
| ---------------------------- | ------------- | --------------------------------------------------------------------------------------- |
| 1. Read each PDF             | Munito AI     | [`POST /ocr`](https://docs.munito.ai/api/createOcr) with `munito/ocr-auto`                          |
| 2. Cut each page into chunks | Your code     | Plain Python                                                                            |
| 3. Find the best chunks      | Your code     | A BM25 keyword index ([`rank_bm25`](https://pypi.org/project/rank-bm25/))               |
| 4. Answer with citations     | Munito AI     | [`POST /chat/completions`](https://docs.munito.ai/api/createChatCompletion) with `qwen/qwen3.8-27b` |

Munito AI has no embeddings endpoint, so this guide retrieves with a keyword index that runs on your machine.
To use vector search, make the embeddings with another tool and keep steps 1 and 4 on Munito AI.

## Complete script [#complete-script]

```bash
pip install openai requests rank_bm25
export MUNITO_API_KEY="munitoai_..."
```

Save this file as `rag.py`. Put your own PDF URLs in `DOCUMENTS`:

```python
# pip install openai requests rank_bm25
import os
import re
import sys
import time

import requests
from openai import OpenAI
from rank_bm25 import BM25Okapi

HOST = "https://api.munito.ai"
BASE_URL = f"{HOST}/inference/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['MUNITO_API_KEY']}"}
DOCUMENTS = {
    "transformer": "https://arxiv.org/pdf/1706.03762",
}


def ocr_pages(url: str) -> list[str]:
    """Read a PDF by URL with Munito OCR. Return the Markdown of each page."""
    response = requests.post(
        f"{BASE_URL}/ocr",
        headers=HEADERS,
        json={"model": "munito/ocr-auto", "document": {"type": "url", "url": url}},
        timeout=300,
    )
    response.raise_for_status()
    body = response.json()
    while body.get("status") in ("accepted", "running"):
        time.sleep(2)
        poll = requests.get(HOST + body["location"], headers=HEADERS, timeout=60)
        poll.raise_for_status()
        body = {**poll.json(), "location": body["location"]}
    return [page["markdown"] for page in body["pages"]]


def chunks(doc: str, pages: list[str], size: int = 800, overlap: int = 200) -> list[dict]:
    """Cut each page into overlapping chunks of characters. Keep the source of each chunk."""
    out = []
    for number, text in enumerate(pages, start=1):
        for start in range(0, max(len(text), 1), size - overlap):
            piece = text[start : start + size].strip()
            if piece:
                out.append({"id": f"{doc} p.{number}", "text": piece})
    return out


def tokens(text: str) -> list[str]:
    return re.findall(r"\w+", text.lower())


def answer(question: str, index: BM25Okapi, corpus: list[dict], k: int = 5) -> str:
    """Retrieve the k best chunks and answer only from them, with citations."""
    scores = index.get_scores(tokens(question))
    best = sorted(range(len(corpus)), key=lambda i: scores[i], reverse=True)[:k]
    context = "\n\n".join(f"[{corpus[i]['id']}]\n{corpus[i]['text']}" for i in best)
    client = OpenAI(base_url=BASE_URL, api_key=os.environ["MUNITO_API_KEY"])
    completion = client.chat.completions.create(
        model="qwen/qwen3.8-27b",
        messages=[
            {
                "role": "system",
                "content": "Answer from the sources only. After each claim, cite its source in brackets, "
                "such as [transformer p.3]. If the sources do not answer, say so.",
            },
            {"role": "user", "content": f"Sources:\n{context}\n\nQuestion: {question}"},
        ],
        reasoning_effort="low",
        max_tokens=2048,
    )
    return completion.choices[0].message.content


if __name__ == "__main__":
    corpus = [chunk for doc, url in DOCUMENTS.items() for chunk in chunks(doc, ocr_pages(url))]
    index = BM25Okapi([tokens(c["text"]) for c in corpus])
    question = sys.argv[1] if len(sys.argv) > 1 else "Which optimizer trains the model, and how many warmup steps does it use?"
    print(answer(question, index, corpus))
```

Run it with a question:

```bash
python rag.py "Which optimizer trains the model, and how many warmup steps does it use?"
```

The answer cites the page of each claim:

```text
The model is trained using the Adam optimizer (with β1 = 0.9, β2 = 0.98, and ϵ = 10⁻⁹) [transformer p.7].
It uses 4000 warmup steps [transformer p.7].
```

## Make it your own [#make-it-your-own]

- **Local files.** Send a file as base64 in place of a URL: `{"type": "base64", "data": ..., "filename": "report.pdf"}`. See [PDF to Markdown](https://docs.munito.ai/pdf-to-markdown).
- **Read once.** OCR each document one time, and save the chunks with the index, for example as JSON. Munito AI does not keep the result after it delivers it.
- **Chunk size.** Larger chunks give the model more context for each source. Smaller chunks make the citations more precise.
- **More sources.** Increase `k` when an answer needs facts from many pages. Each chunk adds input tokens.
- **Answer text.** `qwen/qwen3.8-27b` reasons before it answers. Keep `max_tokens` large, or the answer can stop before it starts, with `content` set to `null`.
- **Region.** To keep all processing in the EU, use `https://api.eu.munito.ai` in place of `https://api.munito.ai`. See [Regions](https://docs.munito.ai/regions).

## Cost [#cost]

A RAG answer costs the OCR pages of each document, one time, plus the chat tokens of each question: the chunks and the question as input, and the answer as output.
See [Models and pricing](https://docs.munito.ai/models#prices).