Milvus Migrations

Milvus Upgrade

OpenRAG has been upgraded from Milvus 2.5.4 to 2.6.11 to leverage the enhancements introduced in the latest releases, particularly the new temporal querying capabilities added in version 2.6.6+.

What’s New in 2.6.x

Milvus 2.6.6+ introduced the TIMESTAMPTZ field type, which enables:

• Comparison and range filtering using standard operators (=, !=, <, >, etc.)
• Interval arithmetic — add or subtract durations (days, hours, minutes) directly in filter expressions
• Time-based indexing for faster temporal queries
• Combined filtering — pair timestamp conditions with vector similarity search

Example — basic comparison:

expr = "tsz != ISO '2025-01-03T00:00:00+08:00'"
results = client.query(
    collection_name,
    filter=expr,
    output_fields=["id", "tsz"],
    limit=10
)

Example — interval arithmetic:

expr = "tsz + INTERVAL 'P1D' > ISO '2025-01-03T00:00:00+08:00'"
results = client.query(
    collection_name,
    filter=expr,
    output_fields=["id", "tsz"],
    limit=10
)

INTERVAL values follow ISO 8601 duration syntax:

• P1D = 1 day
• PT3H = 3 hours
• P2DT6H = 2 days and 6 hours.

Milvus Version Upgrade Steps

Who needs this migration?

This migration is only required if you are upgrading from OpenRAG <= 1.1.7, which shipped with Milvus <= 2.5.x. If your deployment already runs Milvus 2.6.x, skip this section.

For the full official reference, see the Milvus upgrade guide.

Step 1 — Upgrade Milvus to 2.5.16 (intermediate step)

Do not update OpenRAG yet

During this step, keep your current version of OpenRAG (< 1.1.7) running. Only the Milvus image is changed here. OpenRAG itself is updated in Step 2.

Milvus requires an intermediate upgrade to v2.5.16 before jumping to 2.6.x. This step must be done manually before updating OpenRAG.

Temporarily edit vdb/milvus.yaml to set the intermediate Milvus image:

vdb/milvus.yaml

milvus:
  image: milvusdb/milvus:v2.5.4
  image: milvusdb/milvus:v2.5.16

Then restart Milvus and wait for it to be healthy:

docker compose down
docker compose up milvus -d

Verify it is running and healthy before continuing:

docker inspect milvus-standalone --format '{{ .Config.Image }}'
# Expected: milvusdb/milvus:v2.5.16

Step 2 — Update OpenRAG

Once Milvus 2.5.16 is healthy, stop all services and update OpenRAG to the new version. The updated vdb/milvus.yaml already includes Milvus 2.6.11 and the required MinIO and etcd upgrades.

docker compose down

Verify that all containers are stopped:

docker ps | grep milvus

Pull or checkout the new OpenRAG release, then start the stack:

docker compose up -d

Confirm the running Milvus version:

docker inspect milvus-standalone --format '{{ .Config.Image }}'
# Expected: milvusdb/milvus:v2.6.11

Schema Migrations

OpenRAG ships a generic migration runner that discovers and applies all pending Milvus schema migrations in order. You never need to invoke individual migration scripts by hand.

Migrations are versioned. The runner reads the current schema version stored in the collection’s properties and only applies scripts that bring the collection forward (or backward) from that version.

OpenRAG must be stopped

Stop the OpenRAG application before running any migration.

Step 1 — Start only the Milvus container

docker compose up -d milvus

Wait until Milvus is healthy:

docker compose ps milvus

Step 2 — Dry-run (inspect, no changes)

GPU

docker compose run --no-deps --rm --build --entrypoint "" openrag \
    uv run python scripts/migrations/milvus/migrate.py --dry-run

CPU

docker compose --profile cpu run --no-deps --rm --build --entrypoint "" openrag-cpu \
    uv run python scripts/migrations/milvus/migrate.py --dry-run

Review the output to confirm which migrations are pending and what changes they would apply.

Step 3 — Apply all pending migrations

GPU

docker compose run --no-deps --rm --entrypoint "" openrag \
    uv run python scripts/migrations/milvus/migrate.py

CPU

docker compose --profile cpu run --no-deps --rm --entrypoint "" openrag-cpu \
    uv run python scripts/migrations/milvus/migrate.py

The runner will apply each pending migration in order. For the add_temporal_fields migration (v0 → v1) this means:

1. Adding the nullable TIMESTAMPTZ field created_at
2. Creating an STL_SORT index on that field
3. Stamping the collection with schema_version=1 so OpenRAG no longer reports a migration error on startup

Step 4 — Restart OpenRAG

GPU

docker compose up --build -d

CPU

docker compose --profile cpu up --build -d

Targeting a specific version

To upgrade or downgrade to a specific schema version rather than the latest:

GPU

# Upgrade to version 2 only
docker compose run --no-deps --rm --entrypoint "" openrag \
    uv run python scripts/migrations/milvus/migrate.py --target 2

# Downgrade to version 0 (resets version stamp and drops indexes)
docker compose run --no-deps --rm --entrypoint "" openrag \
    uv run python scripts/migrations/milvus/migrate.py --downgrade --target 0

CPU

# Upgrade to version 2 only
docker compose --profile cpu run --no-deps --rm --entrypoint "" openrag-cpu \
    uv run python scripts/migrations/milvus/migrate.py --target 2

# Downgrade to version 0 (resets version stamp and drops indexes)
docker compose --profile cpu run --no-deps --rm --entrypoint "" openrag-cpu \
    uv run python scripts/migrations/milvus/migrate.py --downgrade --target 0

Rollback

Milvus does not support dropping fields. A downgrade only removes indexes and resets the version stamp — fields remain in the schema but are unused by the application:

GPU

docker compose run --no-deps --rm --entrypoint "" openrag \
    uv run python scripts/migrations/milvus/migrate.py --downgrade

CPU

docker compose --profile cpu run --no-deps --rm --entrypoint "" openrag-cpu \
    uv run python scripts/migrations/milvus/migrate.py --downgrade

To fully remove the fields you would need to recreate the collection from scratch.

---

Adding a New Migration Script

Migration scripts live in openrag/scripts/migrations/milvus/. The runner discovers them automatically — no registration step required.

Naming convention

Files must follow the pattern N.short_description.py, where N is the target schema version as a positive integer:

openrag/scripts/migrations/milvus/
  1.add_temporal_fields.py    ← brings the schema to version 1
  2.your_new_migration.py     ← brings the schema to version 2
  migrate.py                  ← generic runner (do not rename)

The numeric prefix determines execution order. Never reuse or change an existing version number.

Required module interface

Each migration script must expose the following at module level:

Name	Type	Description
TARGET_VERSION	int	The schema version this script brings the collection to
upgrade(client, collection_name, dry_run)	function	Applies the migration
downgrade(client, collection_name, dry_run)	function	Reverts the migration (indexes only — fields cannot be dropped)

Minimal template

"""
Milvus migration: <description>  (schema version N-1 → N)
"""

from pymilvus import DataType, MilvusClient
from utils.logger import get_logger

TARGET_VERSION = N  # replace with the actual version number

FIELDS_2_ADD = [
    {"field_name": "my_field", "data_type": DataType.VARCHAR, "max_length": 256, "nullable": True},
]

INDEXES_2_ADD = [
    # add index specs here if needed
]

logger = get_logger()


def upgrade(client: MilvusClient, collection_name: str, dry_run: bool = False) -> None:
    # Add fields and indexes, then bump the version property.
    ...


def downgrade(client: MilvusClient, collection_name: str, dry_run: bool = False) -> None:
    # Drop indexes and reset the version property.
    # Note: Milvus does not support dropping fields.
    ...

Use 1.add_temporal_fields.py as a reference implementation for the full upgrade/downgrade pattern.