Skip to content

Quick Start

OpenRAG is an open source Retrieval Augmented Generation (RAG) solution. This guide is a step-by-step walkthrough to help you get started with OpenRAG.

  • Docker and Docker Compose
  • Your hardware should meet these specifications:
    • CPU deployment: Minimum 13 GiB RAM for light PDF parsers (PyMuPDFLoader), or 23 GiB RAM for heavier parsers like MarkerLoader (refer to this section for details)
    • GPU deployment: 16 GB GPU memory recommended (for systems with separate CPU and GPU memory)
Cloning the OpenRag repository
git clone --recurse-submodules git@github.com:linagora/openrag.git
cd openrag/
git checkout main # or a given release

The Docker Compose stack lives under infra/compose/, next to its .env.example template. Generate a .env from it β€” this fills in every credential with a fresh random value, which the stack requires to start:

Create your .env from the template
python3 scripts/gen_env.py

Then fill in the blanks the generator cannot guess: your LLM/VLM endpoints and the embedder.

The template marks each credential __GENERATE_ME__ rather than shipping a working default, and OpenRag refuses to start on any value published in this repository. A plain cp .env.example .env therefore fails at startup with a message naming the variables still to set β€” deliberately, because a default nobody changed is the most common way a deployment ends up with a known credential.

Here is the minimal set of variables to get started β€” see the full environment-variable reference for every other option:

infra/compose/.env
# ============================================================================
# OpenRAG β€” minimal .env
#
# Only the variables you must set for the default compose stack to boot are
# listed here. Every other knob (PDF/audio loaders, chunking, retriever,
# reranker, Ray Serve, admin UI, OIDC/SSO, rate limiting, MCP server, web
# search, …) has a sensible default and is documented in full at:
#
# https://linagora.github.io/openrag/documentation/env_vars/
# ============================================================================
# ── LLM (external, OpenAI-compatible) ───────────────────────────────────────
BASE_URL=
API_KEY=
MODEL=
LLM_SEMAPHORE=10
# Allow clients to supply their own LLM endpoint through `metadata.llm_override`.
# LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT=true
# ── VLM (vision model, used for image understanding) ────────────────────────
# Can reuse the LLM values above if that model accepts images.
VLM_BASE_URL=
VLM_API_KEY=
VLM_MODEL=
VLM_SEMAPHORE=20
# ── Embedder (HuggingFace model served by the bundled vLLM) ─────────────────
# Point these at an external embedding service instead of the bundled vLLM:
# Changing the model of an install that has already indexed documents needs a
# reindex: vectors from two models don't match, even at the same dimension
# (both give 1024). An install indexed with the previous default keeps it with
# EMBEDDER_MODEL_NAME=jinaai/jina-embeddings-v3.
EMBEDDER_MODEL_NAME=Qwen/Qwen3-Embedding-0.6B
# EMBEDDER_BASE_URL=http://vllm:8000/v1
# Client for that server: vllm (default, needs GET /models), ollama, or tei
# (Hugging Face text-embeddings-inference, which has no /models):
# EMBEDDER_PROVIDER=vllm
# EMBEDDER_API_KEY=EMPTY
# MAX_MODEL_LEN=2047
# Extra `vllm serve` flags for the bundled vLLM embedder, appended after the
# built-in ones, so a repeated flag overrides its default:
# EMBEDDER_EXTRA_ARGS=--gpu_memory_utilization 0.1
# ── Reranker (re-scores retrieved chunks; bundled Infinity server) ───────────
RERANKER_PROVIDER=infinity # 'infinity' (bundled, default) or 'openai' (bundled vLLM, or an external endpoint)
RERANKER_MODEL=Alibaba-NLP/gte-multilingual-reranker-base
RERANKER_ENABLED=true
# Extra `vllm serve` flags for the bundled vLLM reranker (RERANKER_PROVIDER=openai).
# vLLM doesn't recognise gte-multilingual-reranker-base's `NewForSequenceClassification`
# architecture on its own, so that model needs:
# RERANKER_EXTRA_ARGS=--hf-overrides '{"architectures": ["GteNewForSequenceClassification"]}'
## Point these at an external reranker instead of the bundled Infinity / VLLM(Openai) server:
# RERANKER_BASE_URL=http://reranker:7997
# RERANKER_API_KEY=EMPTY
# ── PDF parser (default: PyMuPDF β€” lightweight, CPU-friendly) ────────────────
# Switch to MarkerLoader for OCR / scanned PDFs, complex layouts & embedded
# images (heavier β€” more RAM/GPU). Other option: DoclingLoader.
PDFLOADER=PyMuPDFLoader
# PyMuPDF tuning (whenever PyMuPDF parses, including a preset's pymupdf strategy; unset = one parse at a time, no ceiling):
# PYMUPDF_POOL_SIZE=4 # PDFs parsed at once per indexer worker, each in its own child process
# PYMUPDF_PARSE_MEMORY_LIMIT_MB=1024 # ceiling on one parse child; a PDF over it fails alone instead of OOM-killing the worker
# PYMUPDF_MAX_TASKS_PER_CHILD=20 # parses per child before it is replaced (0 = never)
# Marker tuning (only when PDFLOADER=MarkerLoader):
# MARKER_POOL_SIZE=1 # marker worker actors (β‰ˆ 1 per cluster node / Machine).
# MARKER_MAX_PROCESSES=2 # concurrent PDFs per worker (raise with more GPU)
# ── Image captioning & chunk contextualization (both ON by default) ──────────
# Both run during indexing and call the VLM/LLM β€” set to false to index
# faster and cheaper (with some retrieval-quality trade-off).
# IMAGE_CAPTIONING=false # stop describing images in documents via the VLM
# CONTEXTUAL_RETRIEVAL=false # stop prepending LLM-generated context to each chunk (Anthropic technique)
# Keep saved defaults fixed during a staged rollout; turn this back on after
# all older application instances have stopped.
# PROMPTS_REFRESH_DEFAULTS=false
# ── Secrets ──────────────────────────────────────────────────────────────────
# Every __GENERATE_ME__ below must be replaced before the stack will start β€”
# the application refuses to boot on a value this project publishes. Fill them
# all in one command instead of by hand:
#
# python3 scripts/gen_env.py
#
# MINIO_* are shared by the minio service and Milvus β€” both sides must match.
MINIO_ACCESS_KEY=__GENERATE_ME__
MINIO_SECRET_KEY=__GENERATE_ME__
POSTGRES_PASSWORD=__GENERATE_ME__
# POSTGRES_USER=root
# Fresh Milvus 3 installations can keep the default queue selection. During an
# upgrade, set this to the queue already used by the existing deployment.
# MILVUS_MQ_TYPE=rocksmq
# ── API and Chainlit (chat interface) authentication ──────────────────────────────
# Bearer token that bootstraps the admin user and guards the API.
# Generated by scripts/gen_env.py; `openssl rand -hex 16` works by hand.
# For SSO instead, see https://linagora.github.io/openrag/documentation/oidc/
AUTH_TOKEN=__GENERATE_ME__
# AUTH_MODE=token # 'token' (default) or 'oidc' for SSO
# SUPER_ADMIN_MODE=true
# # FastAPI port (the API and Chainlit chat interface share the same port).
# APP_PORT=8080
# # Optional name used when the assistant introduces itself to a user.
# # Leave unset for a neutral introduction, or set it for a white-label deployment.
# ASSISTANT_NAME=Your Assistant
# # Rate limiting, activated by default (recommended).
# RATE_LIMIT_ENABLED=false
# # Path prefixes the limiter skips (CSV, matched by prefix β€” keep the trailing
# # slash so siblings like /chainlithack stay limited). Unset uses the default shown
# # here; set-but-empty disables the exemption entirely.
# RATE_LIMIT_EXEMPT_PATHS=/chainlit/,/assets/
# Admin-only retrieval diagnostics are limited separately per API process.
# RETRIEVAL_DIAGNOSTICS_RATE_LIMIT=120/minute
# # Reverse-proxy trust. The client IP behind rate limits, and the scheme behind the
# # OIDC cookie `Secure` flag, are only read from X-Forwarded-* headers when the peer
# # is listed here. The bundled admin-ui proxy is NOT on loopback, so keeping the
# # 127.0.0.1 default makes every chat user share a single rate-limit bucket keyed on
# # the proxy's address. Prefer your compose/k8s proxy subnet (e.g. 172.16.0.0/12).
# # Avoid `*` unless the API port is unreachable except through the proxy: `*` trusts
# # X-Forwarded-For from ANY peer, so a caller reaching the API directly can forge the
# # header and evade the per-IP brute-force limit on /auth/*. See env_vars.md
# # (UVICORN_FORWARDED_ALLOW_IPS / Rate Limiting) for the full proxy-trust guidance.
# UVICORN_FORWARDED_ALLOW_IPS=172.16.0.0/12
# ──────────────── Chainlit: Chat interface ────────────────
# Session-cookie signing secret β€” required once auth is enabled; keep it stable
# (rotating it invalidates every live chat session).
# Generated by scripts/gen_env.py; by hand:
# python -c "import secrets; print(secrets.token_urlsafe(32))"
CHAINLIT_AUTH_SECRET=__GENERATE_ME__
# ──────────────── Admin / Indexer UI (React SPA) ────────────────
# Host port for the admin UI β€” the document ingestion, indexing & management
# interface, served at http://<host>:<ADMIN_UI_PORT>/app/. It also proxies the
# API/auth, so it is the OIDC front door. Zero-config otherwise (same-origin, no
# CORS); VITE_* build-time options are documented in the env vars reference.
# ADMIN_UI_PORT=8081
# GRAFANA_ADMIN_USER=admin
# GRAFANA_ADMIN_PASSWORD=__GENERATE_ME__
# Direct Grafana access:
# GRAFANA_URL=http://localhost:3000/d/openrag-http/openrag-http-metrics
# Or through the Admin UI proxy:
# GRAFANA_URL=http://localhost:8081/grafana/d/openrag-http/openrag-http-metrics
# GF_SERVER_ROOT_URL=http://localhost:8081/grafana/
# GF_SERVER_SERVE_FROM_SUB_PATH=true
# ──────────────── Prometheus metrics ────────────────
# GET /metrics on the API port (APP_PORT) is not gated by AUTH_TOKEN and fails
# closed: without METRICS_TOKEN every scrape gets 403. Set a random value
# (`openssl rand -hex 16`); the monitoring overlay hands the same value to its
# Prometheus, so the bundled Grafana needs nothing else. An external
# Prometheus sends it as `Authorization: Bearer <METRICS_TOKEN>`.
# METRICS_TOKEN=replace-with-a-random-secret
# Only when APP_PORT is unreachable from outside your network and you'd rather
# scrape without a token: open the endpoint to anyone who can reach the port.
# The admin-ui proxy (ADMIN_UI_PORT) never serves /metrics either way.
# METRICS_ALLOW_UNAUTHENTICATED=true
# ── Ray (kept as-is by the compose stack; see the docs for what each does) ───
RAY_DEDUP_LOGS=0
# Both of the above are required with LOG_FORMAT=json: Ray's dedup rewrites
# the surviving line as "{...} [repeated 2x across cluster]" and its relay
# prefix is ANSI-colorized even on a pipe, so worker lines stop being JSON.
RAY_COLOR_PREFIX=0
RAY_ENABLE_RECORD_ACTOR_TASK_LOGGING=1
RAY_task_retry_delay_ms=3000
RAY_ENABLE_UV_RUN_RUNTIME_ENV=0
# RAY_memory_monitor_refresh_ms=0
# ── Logging ──
# INFO in production. DEBUG logs user questions, chat history and LLM prompts:
# use it only for short troubleshooting.
LOG_LEVEL=INFO
# Personal data in logs, off by default: request lines keep only the client's
# network (/24 for IPv4, /48 for IPv6), and never the query string. Set true to
# log full client addresses, for troubleshooting or security investigations,
# with a retention period that covers them.
# LOG_PERSONAL_DATA=false
# text (colorized, for a terminal) or json (one object per line, for a log
# collector). The logging overlay forces json on the API; set it here only for
# a collector of your own, together with TQDM_DISABLE=1 (progress bars are
# not JSON; Marker's are off with json either way; the overlay sets both).
# LOG_FORMAT=text
# TQDM_DISABLE=1
# ──────────────── Loki log shipping (logging.docker-compose.yaml) ────────────────
# Push endpoint of your Loki; required to start the overlay.
# LOKI_URL=https://loki.example.com/loki/api/v1/push
# Optional basic auth and tenant header (X-Scope-OrgID).
# LOKI_USERNAME=
# LOKI_PASSWORD=
# LOKI_TENANT_ID=

All supported file format parsers are pre-configured. For PDF processing, PyMuPDFLoader is the default parser β€” a lightweight, fast, CPU-friendly engine well suited to searchable PDFs and quick local testing.

All deployment assets now live under infra/:

  • Directoryinfra/
    • Directorycompose/ full stack (recommended)
      • docker-compose.yaml
      • .env.example
      • .env your configured env
    • Directorydocker/ Dockerfiles
      • …
    • Directoryansible/ remote deployment
      • …
  • Directoryopenrag/ application code
    • …
  • Directoryconf/ YAML configuration
    • …

Run the stack from infra/compose/, where the .env you just created lives. Use the GPU tab on a machine with an NVIDIA GPU, or the CPU tab otherwise.

cd infra/compose
docker compose up -d
# stop it later with:
# docker compose down

The LLM and VLM are always external OpenAI-compatible endpoints β€” set BASE_URL/MODEL/API_KEY (and the VLM_* equivalents) in .env.

The embedder and reranker, by contrast, are bundled: docker compose up -d starts a vllm container serving EMBEDDER_MODEL_NAME and a reranker container (RERANKER_PROVIDER, default Infinity). No extra step is needed to use them.

Terminal window
cd infra/compose
docker compose up -d # GPU β€” core stack + bundled embedder & reranker
# docker compose --profile cpu up -d # CPU-only host

To use external embedding/reranking instead β€” reusing existing inference servers and keeping the footprint small:

  • Embedder β€” set EMBEDDER_BASE_URL (+ EMBEDDER_API_KEY) to your endpoint, then stop the local server by commenting out the vllm-gpu / vllm-cpu service in infra/compose/docker-compose.yaml.
  • Reranker β€” set RERANKER_ENABLED=false to skip reranking, or point it at an endpoint with RERANKER_PROVIDER=openai and RERANKER_BASE_URL (+ RERANKER_API_KEY); to also stop the container, comment out the extern/reranker/… line in the include: block.

Once the app is up and running, you can access the provided services β€” see Default ports below.

Clone the repository, then run the deployment script from infra/ansible/ and follow the interactive prompts:

Terminal window
git clone --recurse-submodules https://github.com/linagora/openrag.git
cd openrag/infra/ansible
./deploy.sh

Once the stack is up, OpenRAG exposes the following services by default:

ServicePortDescription
API Documentation8080/docsMain FastAPI for document ingestion and querying. See this
Chainlit UI8080/chainlitUser interface for interacting with the RAG system
Admin UI8081/app/Main user interface for indexing and viewing indexed documents

The Ray dashboard is not published by default. See Ray dashboard to open it.

More information about each service is available in its respective documentation page.