Skip to content
Docs

Deploying OpenRAG on Kubernetes

This guide explains how to deploy the OpenRAG stack on a Kubernetes cluster using Helm. To upgrade a running release to 2.3.0, follow Upgrading OpenRAG — 2.2.x to 2.3.0 on Kubernetes.

Prerequisites

Section titled “Prerequisites”
  • A Kubernetes cluster with GPU nodes available (NVIDIA runtime) and nvidia-gpu-operator installed.
  • A StorageClass that supports ReadWriteMany (RWX) access mode.
    This is required because the Ray cluster workers and the OpenRAG app need to access the same shared volumes (e.g. for .venv, model weights, logs, data).
  • If using ingress, the ingress-nginx controller needs to be installed on the cluster.

Steps

Section titled “Steps”

  1. Create a values.yaml file:

    • Copy or create a new values.yaml at the root of your repo.
    • You can see the full example file inside the chart: values.yaml
    • Customize the values you need (e.g., image tags, resources, ingress host, storage class, environment variables, secrets).

  2. Set environment and secrets:

    • Edit the env.config and env.secrets sections in your values.yaml.
    • Secrets (API keys, tokens, Hugging Face credentials, etc.) will be mounted into the cluster as Kubernetes secrets.
    • For managed PostgreSQL, point the POSTGRES_* values at the external database and disable database auto-creation.
    • Set minioCredentials.accessKey and minioCredentials.secretKey for the bundled object store. They are required: the install fails without them, with a published value such as minioadmin, with an access key shorter than 3 characters, or with a secret key shorter than 12. The chart stores them in the Secret openrag-minio-credentials, which MinIO and Milvus read. To manage that Secret yourself, for example with External Secrets or Vault, create it with the keys accesskey and secretkey and set minioCredentials.create: false. env.existingSecret and the secrets providers don’t cover it. With an external S3 store (milvus.externalS3), leave them unset: the chart writes the external store’s keys to that Secret instead. Existing releases: read Object storage upgrade first.

  3. Install or upgrade the release from GHCR:

    Terminal window
    helm upgrade\
       --install openrag oci://ghcr.io/linagora/openrag-stack\
       -f ./values.yaml\
       --version 0.6.0
    • openrag is the Helm release name.
    • oci://ghcr.io/linagora/openrag-stack is the remote chart location.
    • -f ./values.yaml specifies your custom configuration.
    • --version 0.6.0 ensures you deploy a specific chart version — check Chart.yaml for the current version before installing.

Upgrading to chart 0.6.0

Section titled “Upgrading to chart 0.6.0”

Chart 0.6.0 renames the PVCs, ConfigMap and Secret from fixed rag-* names to {{ fullname }}-*, so they follow the release instead of colliding between two installs in one namespace. With the default fullnameOverride: "openrag":

BeforeAfter
rag-model-weights, rag-data, rag-logs, rag-venvopenrag-model-weights, openrag-data, openrag-logs, openrag-venv
rag-envopenrag-env
rag-env-secretsopenrag-env-secrets

The old PVCs carry helm.sh/resource-policy: keep, so the upgrade does not delete them — but it does not mount them either. It provisions new, empty ones under the new names, and the release comes up as if it had no indexed data. Pick one before upgrading:

  • Keep the existing volumes. Set fullnameOverride: "rag", which reproduces the old names exactly. Also set postgresql.fullnameOverride, milvus.fullnameOverride and vllm.hfTokenSecretName to match (they are kept in sync by hand — values.yaml explains why, and NOTES.txt warns on an HF_TOKEN secret-name mismatch).
  • Migrate to the new names. Copy the data across (e.g. a Job mounting both PVCs), then delete the old ones once the release is healthy.

Upgrading to chart 0.6.7

Section titled “Upgrading to chart 0.6.7”

Two changes apply to every release, whether or not it scrapes anything:

  • Ray workers move their metrics off port 8080. Workers exported on KubeRay’s default 8080, which networkPolicy.externalPorts opens to every source, so their unauthenticated metrics were public. They now export on 8090, like the head. KubeRay does not recreate Ray pods when the RayCluster changes, so existing workers stay on 8080 until they are recreated. Delete them once after the upgrade, at a time when no indexing is running, since anything running on them is interrupted:

    Terminal window
    kubectl delete pod -n <release namespace> \
      -l ray.io/cluster=<RayCluster name>,ray.io/node-type=worker

    The name is <fullname>-raycluster, which is openrag-raycluster with the default fullnameOverride. If you have set another fullnameOverride, or left it empty so that Helm derives the name from the release, kubectl get raycluster -n <release namespace> prints it. The upgrade notes print the whole command with the name filled in. A wrong name selects no pods, and the workers stay on 8080.

    Leave the head out. The chart configures no GCS fault tolerance, so deleting the head restarts the whole Ray cluster, and every actor on it is lost. The head already exported on 8090, so recreating the workers closes the exposure. The head needs recreating only to bring up its own scrape target (see Monitoring Ray, Postgres and Milvus).

  • Postgres no longer accepts connections from other namespaces. The bitnami sub-chart rendered its own NetworkPolicy, which admitted any source on 5432. The chart now turns it off, along with the read replicas’ policy under postgresql.architecture: replication, so Postgres gets the same rules as every other pod. With networkPolicy.enabled (the default), only the release namespace reaches it, and a client in another namespace needs its own NetworkPolicy.

The upgrade notes print each step that applies (the Ray one with ray.enabled, the Postgres one with postgresql.enabled and networkPolicy.enabled) on the upgrade from chart 0.6.6 or earlier, whichever version it moves to, and on no later upgrade: if you skip the worker delete then, nothing reminds you. The notes find that version in the helm.sh/chart label of the <fullname>-default-deny NetworkPolicy. When they cannot, with networkPolicy.enabled: false or under --dry-run=client, they print the steps that apply on every upgrade.

Upgrades that change the indexer generation

Section titled “Upgrades that change the indexer generation”

Some releases bump the indexer actor generation (_INDEXER_ACTOR_PROTOCOL_VERSION in openrag/services/workers/indexer_pool.py), and some of those also replace the shared TaskStateManager actor. 2.2.x runs v7 and 2.3.x runs v13: the generations in between only ever existed on develop, so an upgrade from 2.2.x to 2.3.x moves v7 to v13 and replaces the TaskStateManager. This matters only when OpenRAG pods share a Ray cluster: with ray.enabled: true, or with an external cluster set through RAY_ADDRESS. With ray.enabled: false (the default) each pod runs its own Ray and restarts with it.

The chart’s Deployment sets no strategy, so Kubernetes rolls it with its default RollingUpdate: new pods start while old ones still serve, against the same Ray cluster. The first new pod replaces the TaskStateManager when it lacks a method the new generation needs. The old generation’s workers keep a handle to the killed actor, so the files they are still indexing lose their state writes and stay SERIALIZING in the status API.

For such an upgrade, first stop traffic and wait until no indexing task is active, as in step 1 of the upgrade guide:

Terminal window
curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
  "https://<openrag host>/queue/tasks?task_status=active"
# {"tasks": []}

Scaling OpenRAG down does not stop the old workers: with ray.enabled: true or an external cluster they run on the Ray cluster, not in the OpenRAG pods, and keep indexing after the pods are gone. Then never let the two releases run side by side:

  • Scale OpenRAG to zero before helm upgrade. The upgrade sets the replica count back to openrag.replicas, so there is nothing to scale up afterwards:

    Terminal window
    kubectl scale -n <release namespace> deploy/<fullname>-openrag --replicas=0

  • Or switch the Deployment to Recreate, which stops every old pod before the first new one starts. The chart does not manage the field, so the change survives later helm upgrade runs:

    Terminal window
    kubectl patch -n <release namespace> deploy/<fullname>-openrag \
      -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

The chart keeps the default rather than rendering Recreate with ray.enabled: Recreate takes the API down on every upgrade, not only on the few that change the generation, and under Ray Serve the API runs on the Ray pods, which the Deployment strategy does not reach. KubeRay does not recreate those pods when the chart changes either: follow the Ray step of the release’s upgrade guide, which restarts the Ray cluster and removes the old generation with it.

On an external Ray cluster, which nothing in the chart restarts, retire the old generation once traffic to the old release has stopped and before the new release starts. Retire the generation of the release you are leaving, which is v7 for 2.2.x. To check it, read it from that release’s tag, or list the indexer dispatchers alive on the cluster (each is named IndexerPoolDispatcher-<generation>):

Terminal window
git show v2.2.2:openrag/services/workers/indexer_pool.py | grep '^_INDEXER_ACTOR_PROTOCOL_VERSION'


PYTHONPATH=openrag uv run python -c "import ray; ray.init(address='${RAY_ADDRESS}'); \
  print(sorted(a['name'] for a in ray.util.list_named_actors(all_namespaces=True) \
  if a['name'].startswith('IndexerPoolDispatcher')))"

Then retire it, for 2.2.x:

Terminal window
PYTHONPATH=openrag uv run python -m services.workers.retire_indexer_generation \
  --ray-address "${RAY_ADDRESS}" \
  --generation v7 \
  --timeout 3600

It stops the old generation from accepting work, waits for the files it accepted to settle, then removes its actors. If the timeout expires, it leaves them running and accepting work again: do not start the new release then. Find out what is still indexing, and run the command again. See Retire an old indexer actor generation.

Upgrades that keep the generation

Section titled “Upgrades that keep the generation”

A fix that lives in the Ray actors reaches a shared Ray cluster only when those actors are recreated, and a release that keeps the generation does not recreate them. The fence that holds a file while an orphaned indexing worker may still write to it is such a fix: an upgrade from 2.3.0 or 2.3.1 keeps v13, so the TaskStateManager and the indexer actors from the old release keep running, and an orphaned task can still release its file while its worker runs. The API then logs a warning that the running Ray actors predate the fence.

Recreate them once, at a time when no indexing is running. With ray.enabled: true, delete every Ray pod, head included, as in step 4 of the upgrade guide. Restart an external Ray cluster yourself. With ray.enabled: false (the default) Ray runs inside the OpenRAG pod, so the pod restart of the upgrade recreates the actors.

Notes

Section titled “Notes”

For the default direct-API deployment, startup and liveness probes use /health_check, while the readiness probe uses /ready. When ENABLE_RAY_SERVE=true, the chart automatically uses exec probes against the Ray head because the Ray Serve HTTP proxy does not run on the API pod, on the same paths. Ray Serve requires ray.enabled=true; Helm rejects that invalid combination. The reverse is rejected too: with ray.enabled=true the API must either run on Ray Serve (ENABLE_RAY_SERVE=true) or be pointed at the cluster with env.config.RAY_ADDRESS: ray://<fullname>-raycluster-head-svc:10001, where <fullname> is fullnameOverride (openrag by default) and the render error prints the exact address. Without either, it would start its own Ray inside its pod, the RayCluster would do no work, and the Ray PodMonitor would scrape none of the indexing metrics. Put RAY_ADDRESS in env.config even when env.existingSecret also carries it: the chart cannot read that Secret, and the Secret’s copy still wins at runtime. Readiness returns 503 when startup is incomplete or PostgreSQL, Milvus, or Ray is unavailable. Model checks are reported in the response but do not gate the whole API, so optional VLM/STT and partition-specific model endpoints do not remove healthy replicas from service. READINESS_REQUIRE_EMBEDDER=true also gates on the default embedder being unavailable or unresolvable (not on a probe timeout). Every replica shares that embedder, so its outage, or a restart while vLLM loads the model, then takes all of them out of the Service at once: the admin API and admin UI too, which leaves kubectl port-forward as the only way in. Before enabling it, check that /ready reports checks.embedder: ok: the probe looks the configured model name up in the endpoint’s GET /models list, so an embedder that works can still read unavailable, for example behind a gateway that lists the model under another name, or on another endpoint without a /models route. A Hugging Face text-embeddings-inference (TEI) embedder has no such route: set EMBEDDER_PROVIDER=tei in env.config (or pick the tei vendor in the admin UI) and its /health is probed instead, without checking the model name. On Ollama, recognized by its GET /api/version, an untagged name also matches its :latest tag, as Ollama resolves it. Checks use short timeouts and results are cached for two seconds. Model probes check availability without running inference; they do not guarantee every request will succeed. Use an application image that includes /ready with these probes.

The bundled Infinity reranker answers /health only once its model is loaded and its warmup benchmark has run. On our test machines that took 100 to 125 s on its default 4 CPUs, 180 to 210 s on 2 and 340 to 380 s on 1, whether or not the model was already cached. Its startup probe allows 10 minutes for that, plus a first-boot model download, and the liveness and readiness probes start only once it passes. A start that hangs is therefore restarted after 10 minutes. All three probes are set under reranker.probes, where each is merged over its default, so set only what you change: raise reranker.probes.startup.failureThreshold for slower CPUs or a slow download. Set a probe to null to drop it, a field to null to remove it, and another handler (tcpSocket, exec or grpc) to replace httpGet. The defaults are listed in values.yaml. On CPU, reranker.extraEnv with INFINITY_MODEL_WARMUP: "false" skips the warmup benchmark: on 4 CPUs Infinity then answered /health in under 20 s.

The reranker’s model is set once, in reranker.model.id: Infinity loads it and OpenRAG names it (RERANKER_MODEL). Its port is set once, in reranker.service.port: Infinity listens on it, the Service exposes it, and RERANKER_BASE_URL calls it. Extra Infinity flags go in reranker.extraArgs, which refuses --model-id and --port, and environment variables for the reranker alone in reranker.extraEnv.

Readiness uses the configured model endpoints and API keys, just like inference. HTTP endpoints do not encrypt those credentials; configure HTTPS when transport encryption is required.

Prometheus exposes aggregate endpoint state through openrag_model_endpoint_ready{provider,kind} and discovery health through openrag_model_endpoint_discovery_up. These labels are intentionally bounded, and the public readiness endpoint reports aggregate configuration-reference counts without exposing partition or preset names.

  • openrag.ingress.host is required when openrag.ingress.enabled is true: the chart refuses to render without it. To reach OpenRAG by public IP only, disable the chart’s ingress and expose the service another way.

  • If you later configure a hostname + TLS (via cert-manager), update openrag.ingress.host and redeploy. With an HTTP-01 issuer, cert-manager answers the challenge from a solver pod it starts in the release namespace, on port 8089. The chart’s <fullname>-acme-http01-solver NetworkPolicy lets that port through on the solver pods only, from any source by default, since a host-network Ingress controller (ingress-nginx with hostNetwork, for one) connects from a node address. Narrow it with networkPolicy.acmeHttp01.from: a namespaceSelector when the controller runs on the pod network, an ipBlock with the node addresses when it uses hostNetwork, which no selector can match. Do not add 8089 to networkPolicy.externalPorts, which would open it on every pod.

    networkPolicy:
      acmeHttp01:
        from:
          # Controller on the pod network:
          # - namespaceSelector:
          #     matchLabels:
          #       kubernetes.io/metadata.name: ingress-nginx
          # Controller with hostNetwork, the node addresses:
          # - ipBlock:
          #     cidr: 10.0.0.0/24

  • Ensure your GPU nodes have the correct NVIDIA drivers and nvidia RuntimeClass configured. The bundled vLLM engines run CUDA 12.9 builds (v0.30.0-cu129, and v0.11.2 for the VLM). The plain v0.30.0 tag is the CUDA 13 build and needs a driver >= 580: pin the -cu129 tag unless every GPU node runs one.

  • The bundled llm engine ships at replicaCount: 0, so env.config.BASE_URL and MODEL are empty by default: OpenRAG starts without an LLM and you set one in the admin UI. To use the bundled engine instead, scale it up and set BASE_URL to http://<release>-llm-engine-service/v1/ (the OpenAI-compatible base, ending in /v1/). The same applies to any engine you scale to 0: empty its *_BASE_URL or point it at the provider you use instead. Otherwise the first boot records a URL with no pods behind it as the default endpoint.

    Upgrading a release that already booted with that URL: the endpoint is saved in the database, and an empty BASE_URL does not remove it, even with MODEL_ENDPOINT_SYNC_ON_BOOT=true. Chat keeps failing with Connection error until you repoint or remove that LLM endpoint in the admin UI.

Operator-managed accounts

Section titled “Operator-managed accounts”

Service accounts whose token comes from your secret store (cozy-stack’s, for example) are declared under openrag.seedUsers. The chart renders the list as SEED_USERS in the env ConfigMap, only when it is non-empty, and the API provisions the accounts at startup. The behaviour is described in Operator-managed accounts.

openrag:
  seedUsers:
    - external_user_id: svc-cozy-stack
      display_name: cozy-stack
      token_env: COZY_STACK_TOKEN
      partitions:
        - { name: twake, role: editor }

The list carries env var names only. The token itself goes where the chart’s other secrets go: env.secrets.COZY_STACK_TOKEN, or a COZY_STACK_TOKEN key in the KV path read by env.secretsProvider.externalSecret (dataFrom) or vaultStaticSecret, which project every key of that path as an environment variable. Point cozy-stack at the same key.

A change to openrag.seedUsers changes the ConfigMap checksum and rolls the API Deployment. A rotated token is picked up on the next restart: vaultStaticSecret restarts the Deployment itself, while with externalSecret the Secret refreshes on its refreshInterval and the pods need a kubectl rollout restart. Under Ray Serve the API runs on the Ray head, which neither mechanism restarts: cycle the head pod after a change. Do not also set SEED_USERS in env.config; the render fails when both are present, and it also fails when an entry carries any key besides external_user_id, display_name, token_env, is_admin and partitions (name and role inside partitions), so a token or any other stray value never lands in the ConfigMap.

Each API process applies the list it booted with, so the last pod to start wins. A replica still running the old ConfigMap or Secret mid-rollout, or a Ray pod that was not cycled, re-applies the old list or the old token hash when it restarts. After changing openrag.seedUsers or a token, let the rollout complete and restart the Ray pods, so every process that can boot carries the new values.

The Secret value is hashed exactly as stored, as for AUTH_TOKEN: a trailing newline (common with echo piped into base64) becomes part of the token and the client’s bearer no longer matches. Use printf '%s' or stringData when writing the Secret.

Restarting the pods automatically when an External Secrets Operator Secret rotates (Reloader, or a Secret checksum annotation) is not done here; it is tracked in #985.

Monitoring

Section titled “Monitoring”

The chart ships OpenRAG’s Grafana dashboards and a ServiceMonitor for the API’s GET /metrics. The dashboards exist once, in infra/charts/openrag-stack/dashboards/, and the Docker Compose monitoring overlay loads the same files. They reach a Grafana in one of three ways:

DeploymentPrometheus and GrafanaWhat the chart renders
Kubernetes, next to a platform’s monitoringthe platform’sa ConfigMap per dashboard for its Grafana sidecar, and the API ServiceMonitor
Kubernetes, standalonekube-prometheus-stack, bundled by monitoring.bundledthe same objects, plus that stack
Docker Composethe overlay’s containersnothing: the overlay mounts dashboards/ and Grafana provisions it from disk

Every dashboard reads its data source through a variable that starts on Grafana’s default Prometheus, so the files load unchanged in any of them (see Prometheus metrics).

Next to an existing Prometheus and Grafana

Section titled “Next to an existing Prometheus and Grafana”

Leave monitoring.bundled off and turn on the objects, labelled the way the platform selects them:

monitoring:
  dashboards:
    enabled: true
openrag:
  metrics:
    serviceMonitor:
      enabled: true
      bearerTokenFromSecret: true
      labels:
        release: kube-prometheus-stack   # what the platform's Prometheus selects
env:
  secrets:
    METRICS_TOKEN: "<openssl rand -hex 16>"

Ask whoever runs the platform’s monitoring, and set:

QuestionValue
Which label does your Prometheus select ServiceMonitor objects on, and does it watch this namespace?openrag.metrics.serviceMonitor.labels. The monitor is created in the release namespace.
Does your Grafana run the dashboard sidecar, and which label does it watch?monitoring.dashboards.label and labelValue, by default grafana_dashboard: "1" (kube-prometheus-stack’s default).
Which namespaces does the sidecar search?Nothing to set if it searches all of them (kube-prometheus-stack’s default). If it only watches its own, set monitoring.dashboards.namespace to the Grafana namespace.
Which Grafana folder?monitoring.dashboards.folder (default OpenRAG), written into the annotation named by monitoring.dashboards.folderAnnotation (default grafana_folder). It must match the sidecar’s folderAnnotation, which also needs provider.foldersFromFilesStructure. A sidecar without one uses its default folder.

A ConfigMap or monitor with the wrong label is created and then ignored, which looks exactly like one that works. After installing, check that the dashboards appear in Grafana and that the target is up in Prometheus (Status → Targets). Without the Prometheus Operator, the API pod’s prometheus.io/* annotations serve annotation-based discovery instead; see Scraping in Kubernetes.

Standalone: the bundled stack

Section titled “Standalone: the bundled stack”

On a cluster with no monitoring of its own, monitoring.bundled: true installs kube-prometheus-stack in the release: Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics. It also turns on the dashboard ConfigMaps, OpenRAG’s alert rules, the API ServiceMonitor and, unless the API attaches to an external Ray cluster, the Ray PodMonitor:

monitoring:
  bundled: true
env:
  secrets:
    METRICS_TOKEN: "<openssl rand -hex 16>"

  • METRICS_TOKEN is required. The bundled Prometheus always sends it as the bearer on GET /metrics, like the Compose overlay, and the chart refuses to render without it. With env.existingSecret or an external secrets provider, that Secret must carry the key, since the chart cannot check it.

  • Every monitor is scraped. The bundled Prometheus selects all ServiceMonitor, PodMonitor and PrometheusRule objects in the cluster, not only those labelled with its release. A third-party monitor, such as the GPU Operator’s DCGM exporter, needs no release label.

  • The control plane is not scraped. On a managed cluster (EKS, GKE, AKS and the like) the controller manager, scheduler and etcd run out of reach, and some clusters replace kube-proxy. kube-prometheus-stack would still look for them, and its KubeControllerManagerDown, KubeSchedulerDown and KubeProxyDown alerts would fire forever, so the chart turns the four off, with their rules and dashboards. On a self-managed control plane, turn them back on:

    kubePrometheusStack:
      kubeControllerManager: { enabled: true }
      kubeScheduler: { enabled: true }
      kubeEtcd: { enabled: true }
      kubeProxy: { enabled: true }

    Each component must also serve its metrics on an address Prometheus can reach. kubeadm binds all four to 127.0.0.1, where the scrape is refused and the same alerts fire.

  • History survives a restart. Prometheus keeps 30 days, capped at 18 GB, on a 20 Gi ReadWriteOnce volume from the default StorageClass. Tune it under kubePrometheusStack.prometheus.prometheusSpec.

  • Every workload has requests and a memory limit. Upstream sets none, which would leave them BestEffort: nothing reserved, and evicted first under memory pressure. The values come from a one-node kind cluster under the queries of every bundled dashboard, with room above what each used there. Prometheus is the one that grows, with the number of series rather than the retention: its 1 Gi request holds about 190k series, and values.yaml shows how to estimate yours from the node, pod and API server counts. Watch prometheus_tsdb_head_series and raise kubePrometheusStack.prometheus.prometheusSpec.resources with it.

  • Upstream values go under kubePrometheusStack. The sub-chart is aliased, so an upstream kube-prometheus-stack.x.y key is kubePrometheusStack.x.y here.

  • The operator’s admission webhook is let through the NetworkPolicy. It validates PrometheusRule and AlertmanagerConfig objects before they are stored, and its caller is the API server — not a pod, and on a managed control plane not on the cluster network at all, so no selector can name it. The chart opens that one port, to any source, on the webhook’s own pod: it runs in a separate deployment (kubePrometheusStack.prometheusOperator.admissionWebhooks.deployment), which serves the webhook, /metrics and /healthz only, because the operator’s listener would also expose /debug/pprof/ on the same port. Give networkPolicy.webhookFrom a peer to narrow it where the control plane’s address is known, and do narrow it if you turn that deployment off. Without the opening the call is dropped wherever the API server is off-node, and kube-prometheus-stack’s default failurePolicy (Ignore) then skips validation silently.

  • The dashboard label and folder must match the bundled Grafana. Its sidecar is configured under kubePrometheusStack.grafana.sidecar.dashboards, which cannot follow monitoring.dashboards, so the chart refuses to render when the label, label value or folder annotation differ. Change them on both sides together.

  • No API scrape under Ray Serve. With ray.enabled=true and ENABLE_RAY_SERVE=true, no ServiceMonitor is rendered for the API, and the HTTP dashboard stays empty (see the Limitations in Prometheus metrics).

The install notes print how to reach Grafana:

Terminal window
kubectl -n <namespace> port-forward svc/openrag-grafana 3000:80
# user and password:
kubectl -n <namespace> get secret openrag-grafana -o jsonpath='{.data.admin-user}' | base64 -d
kubectl -n <namespace> get secret openrag-grafana -o jsonpath='{.data.admin-password}' | base64 -d

With kubePrometheusStack.grafana.admin.existingSecret set, as under Argo CD below, Grafana reads them from that Secret instead, under the keys named by admin.userKey and admin.passwordKey (admin-user and admin-password by default). The notes print those.

The OpenRAG dashboards are in the OpenRAG folder, next to kube-prometheus-stack’s own Kubernetes dashboards. To publish Grafana through an Ingress, enable kubePrometheusStack.grafana.ingress and add 3000 to networkPolicy.externalPorts: the default-deny policy admits only 8080 from outside the namespace.

Alertmanager starts with no receiver. Alerts show as firing in Prometheus and Alertmanager but notify nobody until kubePrometheusStack.alertmanager.config routes them.

Before turning it on:

  • Not next to another Prometheus Operator. A second operator competes with the first over the same CRDs and objects. If the cluster already runs one, use the setup in the previous section.

  • It needs cluster-wide rights, and one release per cluster. Besides the operator’s CRDs, it creates cluster-scoped objects (ClusterRoles and their bindings, the admission webhook configurations) and Services in kube-system (CoreDNS’s, and the kubelet’s, which the operator maintains), so rights over the release namespace alone are not enough to install it. Their names are fixed (openrag-monitoring-*, openrag-grafana-*), so a second release with bundled in the same cluster fails at install: point it at the first one’s Prometheus and Grafana instead, as in the previous section.

  • node-exporter needs host access. It runs on every node with hostNetwork, hostPID and the node’s /, /proc and /sys mounted. A namespace that enforces the Pod Security baseline or restricted level rejects its pods. On a node that already runs a node-exporter, the two need the same host port, 9100, and the second one stays Pending. In either case set kubePrometheusStack.nodeExporter.enabled: false. The host panels of the Infrastructure Overview dashboard (CPU, memory, disk, network and load) then stay empty, unless this Prometheus scrapes the existing node-exporter, for example through its ServiceMonitor.

  • An existing release needs the CRDs first. Helm installs CRDs on the first install only, so an upgrade that turns bundled on fails with no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1". Helm never upgrades CRDs either, so repeat this after a chart upgrade that moves kube-prometheus-stack:

    Terminal window
    helm show crds oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
      --version 91.4.1 | kubectl apply --server-side -f -

    --server-side is required: the Prometheus CRD is over 1 MB, past the 256 KiB a client-side apply can record.

  • Argo CD needs two settings. Sync with ServerSideApply=true, for the same reason. Argo CD also renders without cluster lookups, so the generated Grafana password would change on every sync. Point kubePrometheusStack.grafana.admin.existingSecret at a Secret you manage instead.

  • Under Argo CD, rule validation starts once the whole Application is Healthy. The admission webhook’s certificate comes from two Helm-hook Jobs, which Argo CD runs as sync hooks: openrag-monitoring-admission-create before the sync, and openrag-monitoring-admission-patch, which gives the webhook configurations their CA, after it. Argo CD runs that second hook only once every resource in the Application is Healthy, OpenRAG’s included. Until then the API server cannot verify the webhook, and under the default failurePolicy (Ignore) it stores PrometheusRule and AlertmanagerConfig objects unvalidated. Later syncs keep the certificate and the CA. Where cert-manager runs, set kubePrometheusStack.prometheusOperator.admissionWebhooks.certManager.enabled: true: cert-manager issues the certificate instead of the hooks, and validation works from the first sync.

Managed PostgreSQL

Section titled “Managed PostgreSQL”

The chart can run against a database that is provisioned outside OpenRAG, which is the recommended setup on OpenShift or cloud-managed PostgreSQL.

Pre-create the database before installing the release. If POSTGRES_DATABASE is not set, OpenRAG uses partitions_for_collection_<VDB_COLLECTION_NAME>. The app role does not need CREATEDB or superuser rights; it needs to connect to that database and own, or be allowed to create objects in, the target schema.

In values.yaml, disable the bundled PostgreSQL chart, set postgresProvisioning.autoCreateDatabase to false, set postgresProvisioning.runMigrationsInApp to false, and enable postgresProvisioning.migrationJob. Then provide the managed database connection through POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, and optionally POSTGRES_DATABASE.

The migration Job (templates/postgres-migration-job.yaml) is a Helm hook, annotated with helm.sh/hook: pre-install,pre-upgrade. You never invoke it directly: Helm runs it automatically as part of each helm install and helm upgrade, before it creates or updates the OpenRAG Deployment, and waits for it to finish. It applies the Alembic migrations against the pre-created database (it migrates the schema but does not create the database). The OpenRAG API then starts against an already-migrated schema. Because a pre-install hook runs before Helm creates the release’s own resources, the Job reads its own copies of the env ConfigMap and Secret, created as hooks just before it and deleted once it succeeds. With env.secretsProvider set to externalSecret or vaultStaticSecret, the copy is a second ExternalSecret or VaultStaticSecret, and the Job’s pod cannot start until the operator has created that Secret. Helm does not wait for the operator itself, only for the Job, within --timeout (5 minutes by default), so raise --timeout if your operator can take longer to sync. An env.existingSecret is read directly.

If the Job fails, Helm 3.19 and later delete its copies along with the failure. Helm 3.18 and earlier, and Argo CD, leave them in the namespace, as does any Helm version that is interrupted or cannot create the Job, and helm uninstall does not remove them, since Helm does not track hook resources. The Secret copy holds the same credentials as the app’s Secret. The next install, upgrade or sync replaces the copies. To remove them before that, delete the ConfigMap and the Secret whose names end in -migration-env and -migration-env-secrets, and with externalSecret or vaultStaticSecret also the ExternalSecret or VaultStaticSecret named like the Secret. Find them by that suffix, kubectl get configmap,secret -n <namespace> | grep migration-env (add externalsecret or vaultstaticsecret to the list with that provider), rather than by <fullname>: the chart shortens the fullname in these names, to 49 characters for the ConfigMap and 41 for the Secret, so with a long fullname they don’t start with the full <fullname>.

The Job is for an external PostgreSQL only. It runs before Helm creates any of the release’s own resources, the bundled PostgreSQL included, so the chart refuses migrationJob.enabled together with postgresql.enabled.

When postgresProvisioning.migrationJob is disabled (the default), the Job is not rendered at all, and the application runs the migrations itself at startup while postgresProvisioning.runMigrationsInApp is on (the default). With both off, nothing applies them, so the chart refuses to render unless you set postgresProvisioning.externalMigrations: true to say you run python -m services.persistence.migrations.run yourself. Set the app’s side through runMigrationsInApp only: the chart also refuses an env.config.POSTGRES_RUN_MIGRATIONS or env.secrets.POSTGRES_RUN_MIGRATIONS that disagrees with it, since that entry would silently replace it. Do not put POSTGRES_RUN_MIGRATIONS in an env.existingSecret or in a secret from an external secrets operator: the pods read the Secret after the ConfigMap, so its value wins, and the chart cannot read it to check it.

GPU metrics

Section titled “GPU metrics”

The chart deploys no GPU exporter. GPU metrics come from the DCGM exporter that the NVIDIA GPU Operator — a prerequisite above — runs on every GPU node (dcgmExporter.enabled, on by default). What is left is getting Prometheus to scrape it and Grafana to show it.

  1. Scrape the exporter. With the Prometheus Operator (e.g. kube-prometheus-stack), have the GPU Operator create its ServiceMonitor. That is the default from GPU Operator v26.7.0; earlier releases ship it disabled:

    Terminal window
    # Pin the version you already run: --reuse-values without --version also
    # upgrades the GPU Operator to the latest chart.
    helm upgrade gpu-operator nvidia/gpu-operator -n gpu-operator \
      --version <installed version> --reuse-values \
      --set dcgmExporter.serviceMonitor.enabled=true \
      --set dcgmExporter.serviceMonitor.additionalLabels.release=kube-prometheus-stack

    The release label must be whatever your Prometheus serviceMonitorSelector matches (kube-prometheus-stack selects its own release name). An unmatched ServiceMonitor is created but never scraped. Without the Prometheus Operator, the exporter’s nvidia-dcgm-exporter Service (port 9400) carries the prometheus.io/scrape: "true" annotation for annotation-based discovery.

  2. Check it arrived. DCGM_FI_DEV_GPU_UTIL should return series in Prometheus. Expect one series per GPU — or, when the exporter runs with KUBERNETES_VIRTUAL_GPUS=true for time-sliced or MPS-shared GPUs, one per pod using each GPU. The dashboard counts each GPU once either way.

  3. Get the dashboard in front of you. The GPU panels of the Infrastructure Overview dashboard (infra/charts/openrag-stack/dashboards/system-overview.json) read the DCGM exporter here and nvidia_gpu_exporter under Docker Compose — whichever is scraped — so the same file serves both. The chart ships it to Grafana for you: see Monitoring above. Imported by hand instead, it needs no editing — every panel reads a data source variable that starts on Grafana’s default Prometheus.

The GPU panels aggregate every GPU that Prometheus scrapes, not only the nodes running OpenRAG, and the host panels likewise need node-exporter (kube-prometheus-stack ships it) and aggregate every node.

Monitoring Ray, Postgres and Milvus

Section titled “Monitoring Ray, Postgres and Milvus”

The chart wires the stack’s three dependencies into a Prometheus that runs the Prometheus Operator. Every part of it is off by default. monitoring.bundled turns on the Ray PodMonitor (for the RayCluster or the embedded Ray; an external cluster is scraped where it runs) and nothing else here: OpenRAG’s ingestion series and the worker side of its inference series exist on Ray’s endpoint only, and the alerts built on them cannot fire without it. With the bundled stack, set only the Postgres and Milvus enabled switches and the Postgres password: the bundled Prometheus runs in the release namespace and selects every monitor, so it needs neither selector labels nor networkPolicy.metricsFrom. Next to an existing Prometheus (kube-prometheus-stack or a standalone operator), set all of it:

networkPolicy:
  metricsFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring   # the namespace Prometheus runs in
ray:
  metrics:
    podMonitor:
      enabled: true                                  # the RayCluster, or the embedded Ray
      labels: { release: <Prometheus release name> }
postgresql:
  auth:
    postgresPassword: <password>                     # required by metrics.enabled, see below
  metrics:
    enabled: true                                    # adds the exporter sidecar: restarts Postgres
    serviceMonitor:
      enabled: true
      labels: { release: <Prometheus release name> }
milvus:
  metrics:
    serviceMonitor:
      enabled: true
      additionalLabels: { release: <Prometheus release name> }

Ray gets a PodMonitor rather than a ServiceMonitor because every Ray node exports its own metrics, on port 8090. With ray.enabled=false (the default) the only Ray node is embedded in the openrag pod: the chart pins its metrics port to 8090 with RAY_METRICS_EXPORT_PORT, declares it as ray-metrics, and the PodMonitor selects that pod instead of the RayCluster’s. With ray.enabled=false and RAY_ADDRESS set, the API attaches to an external Ray cluster and starts no Ray of its own: the chart then declares no port and renders no PodMonitor, and refuses ray.metrics.podMonitor.enabled. Scrape that cluster where it runs. The chart reads RAY_ADDRESS from env.config and env.secrets only: when it comes from env.existingSecret or an external secrets provider, set ray.externalCluster: true as well. Ray prefixes the metrics OpenRAG records inside its workers with ray_. The PodMonitor strips that prefix, so these metrics are stored under the same openrag_* names the API’s /metrics uses, and the alert rules match them. Ray’s own ray_* metrics keep their names. Milvus already exports from all five components (proxy, mixcoord, datanode, querynode, streamingnode); only its ServiceMonitor is new. None of the three endpoints authenticates, so none is routed through the Ingress.

The Postgres exporter signs in as the postgres superuser, with the password in the sub-chart’s Secret. Unless postgresql.auth.postgresPassword or postgresql.auth.existingSecret sets that password, the sub-chart generates it again on every render that cannot read the live Secret (helm template, Argo CD), while the server keeps the first one, and the exporter’s login fails after a later sync. The chart therefore refuses postgresql.metrics.enabled without one of the two. On a new release, any password works. On a running release, set the one the server already has: kubectl get secret -n <release namespace> openrag-postgresql -o jsonpath='{.data.postgres-password}' | base64 -d.

Three things can go wrong without failing the install:

  • Selector labels. A monitor without the label its Prometheus selects on is created and never scraped. The example assumes kube-prometheus-stack, which selects on release: <its Helm release name> by default; other setups may use another label. Read the selectors with kubectl get prometheus -A -o jsonpath='{..podMonitorSelector}{..serviceMonitorSelector}'. The Postgres sub-chart calls the key labels; the Milvus one calls it additionalLabels.
  • NetworkPolicy. The default-deny policy admits only same-namespace traffic. Until Prometheus’s namespace is listed in networkPolicy.metricsFrom, its targets report up == 0. Each entry opens only the metrics port, and only on the pods that export it. The chart turns off the Postgres sub-chart’s own NetworkPolicy for this: it admitted any source on every port it listed, the exporter’s 9187 included (see Upgrading to chart 0.6.7).
  • Ray pods created by chart 0.6.6 or earlier. KubeRay does not recreate Ray pods when the RayCluster changes. Old workers still export on 8080 and need recreating anyway (Upgrading to chart 0.6.7). The old head exports on 8090, but it also carries the metrics: 8080 port that KubeRay adds to a container without a port of that name. Its target therefore stays down until the head is recreated too. Recreating the head restarts the whole Ray cluster, so leave it for a maintenance window: kubectl delete pod -n <release namespace> -l ray.io/cluster=<RayCluster name>,ray.io/node-type=head.

Once enabled, this should return 1 for every Ray node, the Postgres pod and the five Milvus pods:

up{namespace="<release namespace>", job=~".*(raycluster|openrag-ray|postgresql|milvus).*"}

What to watch

Section titled “What to watch”
DependencyQuerySignal
Rayray_tasks{namespace="<release namespace>", State="PENDING_NODE_ASSIGNMENT"}Tasks no node has the resources to run
Rayray_actors{namespace="<release namespace>", State="RESTARTING"}Actors being restarted. This gauge is sampled, so it catches a crash loop but can miss a single fast restart
Rayray_resources{namespace="<release namespace>", Name="GPU"} by State (USED, AVAILABLE)GPU allocation per node
Rayray_node_mem_used{namespace="<release namespace>"}, and the same for ray_node_cpu_utilization and ray_object_store_memoryNode resources
Postgressum by (instance) (pg_stat_activity_count{namespace="<release namespace>"}) / sum by (instance) (pg_settings_max_connections{namespace="<release namespace>"})Connections against the server limit, per server
Postgrespg_stat_activity_max_tx_duration{namespace="<release namespace>"}Longest open transaction
Postgrespg_database_size_bytes{namespace="<release namespace>"}, pg_locks_count{namespace="<release namespace>"}Database size, lock contention
Milvushistogram_quantile(0.99, sum by (le, function_name) (rate(milvus_proxy_req_latency_bucket{namespace="<release namespace>"}[5m]))), and the same over milvus_proxy_sq_latency_bucket by query_typep99 latency in milliseconds, per request type and per search/query type
Milvussum(rate(milvus_proxy_insert_vectors_count{namespace="<release namespace>"}[5m])), sum(rate(milvus_proxy_search_vectors_count{namespace="<release namespace>"}[5m]))Vectors inserted and searched per second
Milvusmilvus_querycoord_collection_num{namespace="<release namespace>"}Loaded collections
Milvusmilvus_datacoord_segment_num{namespace="<release namespace>"} by segment_state, milvus_datacoord_compaction_task_num{namespace="<release namespace>"}Segment and compaction backlog
Milvussum by (component) (process_resident_memory_bytes{namespace="<release namespace>", component!=""})Memory per component

Not covered

Section titled “Not covered”
  • The API’s connection-pool wait. The pool lives in the OpenRAG process, so Postgres cannot see a request queued for a connection. That needs an application metric.
  • Slow queries. These need pg_stat_statements, which means a shared_preload_libraries change on the server plus the exporter’s --collector.stat_statements. Without it, pg_stat_activity_max_tx_duration still shows long-running transactions.
  • Volume fill. pg_database_size_bytes is the database size, not how full its PVC is; use the kubelet’s kubelet_volume_stats_* series for that.
  • MinIO and etcd, Milvus’s own dependencies.

Series volume

Section titled “Series volume”

Milvus and Ray are chatty. Samples per scrape on a single-node install holding one 5 000-row collection:

TargetSamples per scrape
Milvus, all five components~45 000 (the streaming node alone ~20 000)
Ray, head and one worker, lightly loaded~1 150
Postgres~550

Where the Prometheus belongs to a platform team, agree on that volume before enabling Milvus’s monitor.