How to backup or restore OpenRag partition/data ?

How to backup partitions?

Storage layout (host paths vs. named volumes) and migrating data between storage modes are documented in Deploying OpenRAG on OpenShift.

Backup one partition

Caution

It’s better to stop openrag-cpu (or openrag) service before starting backup.

docker compose --env-file .env \
    run \
    --build \
    --rm \
    -v /my-backup-dir/:/backup:rw \
    --entrypoint "bash /app/scripts/entrypoint-backup.sh ${PARTITION_NAME}" \
    openrag-cpu

By default backup script creates plan text uncomressed file. To make things faster you can use multithread compressor the following way:

docker compose --env-file .env \
    run \
    --build \
    --rm \
    -v /my-backup-dir/:/backup:rw \
    --entrypoint "bash /app/scripts/entrypoint-backup-mt.sh ${PARTITION_NAME}" \
    openrag-cpu

Backup all partitions

docker compose --env-file .env \
  run --build --rm \
  -v ~/backup:/backup:rw \
  --entrypoint "uv run /app/scripts/backup.py -o /backup/test.openrag" \
  openrag

# Use --include-only to specify the partitions to back up.

How to backup partitions?

How to restore OpenRag partition ?

Start with dry run to ensure the backup file is correct:

docker compose --env-file .env \
    run \
    --build \
    --rm \
    -v /my-backup-dir/:/backup:ro \
    --entrypoint "bash /app/scripts/entrypoint-restore-dry-run.sh backup-file-without-path parition-name" \
    openrag-cpu

Backup files are expected to be in /my-backup-dir/. If the dry run is successful, run the following script to insert the data :

docker compose --env-file .env \
    run \
    --build \
    --rm \
    -v /my-backup-dir/:/backup:ro \
    --entrypoint "bash /app/scripts/entrypoint-restore.sh backup-file-without-path parition-name" \
    openrag-cpu

Restore all partitions

docker compose --env-file .env \
  run --build --rm \
  -v ~/backup:/backup:rw \
  --entrypoint "uv run /app/scripts/restore.py /backup/test.openrag"\
  openrag

# Use --include-only to specify the partitions to restore.

Backup dump format

Backups are stored in plain text, with optional xz compression. A backup file consists of multiple sections separated by an empty line.

Each section begins with a single header line, followed by one or more non-empty content lines. Every content line is in JSON format.

There are two types of sections:

• rdb – Multiple sections are possible (one section per partition). The first line of an rdb section specifies the partition. Each subsequent line represents a single document.
• vdb – A single section covering all partitions. One line per chunk.

All rdb sections must appear before the vdb section.

Example:

rdb
{"created": "2025-07-28T16:20:43.144796", "name": "frwiki-nocontext"}
{"created_at": "2025-07-28T16:20:39.612784", "file_id": "10", "file_size": "13.57 KB", "filename": "Algorithmique.txt", "revid": "2962", "source": "/app/data/Algorithmique.txt", "title": "Algorithmique", "url": "https://fr.wikipedia.org/wiki?curid=10"}
{"created_at": "2025-07-28T16:20:40.948772", "file_id": "100", "file_size": "3.79 KB", "filename": "Atoum.txt", "revid": "734387", "source": "/app/data/Atoum.txt", "title": "Atoum", "url": "https://fr.wikipedia.org/wiki?curid=100"}
...

rdb
{"created": "2025-07-10T16:51:25.466016", "name": "enwiki-markdown_splitter-nocontext"}
{"created_at": "2025-07-10T16:51:40.416456", "file_id": "1000", "file_size": "42.03 KB", "filename": "Hercule Poirot.txt", "revid": "25695884", "source": "/app/data/Hercule Poirot.txt", "title": "Hercule Poirot", "url": "https://en.wikipedia.org/wiki?curid=1000"}
{"created_at": "2025-07-10T17:00:02.038049", "file_id": "10000", "file_size": "220.00 B", "filename": "Eiffel.txt", "revid": "5229428", "source": "/app/data/Eiffel.txt", "title": "Eiffel", "url": "https://en.wikipedia.org/wiki?curid=10000"}
...

vdb
{"created_at": "2025-07-28T16:20:39.680783", "file_id": "7", "file_size": "11.01 KB", "filename": "Algèbre linéaire.txt", "page": 1, "partition": "frwiki-nocontext", "revid": "2523928", "source": "/app/data/Algèbre linéaire.txt", "text": "L’algèbre linéaire est ...", "title": "Algèbre linéaire", "url": "https://fr.wikipedia.org/wiki?curid=7", "vector": [0.00012493133544921875, -0.052978515625, ...]}
{"created_at": "2025-07-28T16:20:39.680783", "file_id": "7", "file_size": "11.01 KB", "filename": "Algèbre linéaire.txt", "page": 1, "partition": "frwiki-nocontext", "revid": "2523928", "source": "/app/data/Algèbre linéaire.txt", "text": "Ce n'est qu'au XIXsiècle que ...", "title": "Algèbre linéaire", "url": "https://fr.wikipedia.org/wiki?curid=7", "vector": [-0.0206298828125, -0.09765625, ...]}
...