release(opencode): v0.2.3

fix(opencode): observable logging — config-only debug, resolved-endpoint log, surfaced errors (#2047 )
* fix(opencode): observable logging — config-only debug, resolved-endpoint log, surfaced errors OpenCode users (notably on Windows) could see tool calls register but no memories land, with zero signal as to why: every retain/recall failure was swallowed via debugLog, the resolved API URL/bank was only logged when debug was on, and HINDSIGHT_DEBUG is unreliable to set for OpenCode's plugin runtime. - Add a Logger that routes through OpenCode's server log stream (client.app.log, service=hindsight) — TUI-safe, visible via --print-logs and the OpenCode log files. Falls back to console.error when no client. - error/warn/info are always emitted; debug is gated on config.debug. - Always log the resolved endpoint + bank at init (a common 'memories aren't saving' cause is silently defaulting to Hindsight Cloud). - Surface retain/recall/hook failures as errors instead of swallowing them; hooks still never throw, so OpenCode is not affected. - Drop the HINDSIGHT_DEBUG env override; 'debug' is now a config-only option (opencode.json plugin options or ~/.hindsight/opencode.json). - Tests for the logger; update config tests; document the change. Refs #1758 * style(opencode): prettier-format plugin.test.ts (pre-existing drift) * docs(opencode): document config-only debug + default error/endpoint logging
2026-06-08 12:28:18 +02:00 · 2026-06-08 12:27:32 +02:00 · 2026-06-08 11:59:07 +02:00 · 2026-06-08 11:28:15 +02:00 · 2026-06-08 11:16:13 +02:00 · 2026-06-08 11:14:32 +02:00
708 changed files with 59954 additions and 8163 deletions
@@ -73,6 +73,11 @@ results = await asyncio.gather(*tasks, return_exceptions=True)
 results = await asyncio.gather(*tasks, return_exceptions=True)
 ```

+### API Layer & Data Access
+- **No direct database access in `api/http.py`** (or any API router). HTTP handlers must not build SQL, call `acquire_with_retry` / `conn.fetch` / `conn.fetchrow` / `conn.execute`, or reference `fq_table(...)`. All persistence and queries live in `MemoryEngine` (the engine layer). A handler parses/validates the request, calls an engine method, shapes the HTTP response, and maps domain results to status codes (e.g. a `None` return → 404).
+- **Authentication/tenancy is enforced inside each engine method, not assumed by the handler.** Every engine method that touches bank-scoped data must authenticate via `request_context` — typically `await self._authenticate_tenant(request_context)` (often indirectly through `get_bank_profile(...)`) — so the correct tenant schema is resolved before any query runs. Handlers must thread `request_context` through to the engine method; never query a tenant-scoped table assuming the schema is already set.
+- Engine methods return typed models (Pydantic/dataclass), not raw dicts (see Type Safety).
+
 ### Branch Hygiene
 - **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
 - **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
@@ -135,6 +140,13 @@ For each new or significantly changed function/endpoint/class:

 Flag any new logic that lacks test coverage.

+**LLM-behaviour changes need a real-LLM judge test, not MockLLM.** If the change alters how the model interprets a prompt — fact/observation extraction, `fact_type` (world/experience) classification, speaker attribution, instruction-following, prompt wording — there MUST be a test marked `pytest.mark.hs_llm_core` that runs the real pipeline and asserts via `tests.llm_judge.assert_meets_criteria` (not string/enum matching). Flag these as findings:
+- A prompt/classification change verified only by MockLLM or string assertions (MockLLM echoes input — such tests pass spuriously). **Should fix.**
+- A test that hard-asserts `fact_type == "world"/"experience"` (or other model-decided output) instead of judging it — non-deterministic, will flake across providers/runs. **Should fix** (move the classification check into the judge `criteria`; keep only genuinely deterministic structural asserts direct).
+- Deterministic mechanics (prompt assembly, suppression/branching logic) that are covered *only* by a slow LLM test — these should also have fast non-LLM unit tests. **Note.**
+
+See CLAUDE.md → Key Conventions → Testing for the full pattern.
+
 ### 7. Check API consistency

 If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
@@ -142,6 +154,12 @@ If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
 - Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
 - Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)

+### 7b. Check API-layer data-access boundary
+
+For each changed handler in `hindsight-api-slim/hindsight_api/api/` (e.g. `http.py`, `mcp.py`):
+- **Flag any direct DB access in the handler** — `acquire_with_retry`, `conn.fetch` / `fetchrow` / `execute`, raw SQL strings, or `fq_table(...)`. These are a **must fix**: the query must be moved into a `MemoryEngine` method that returns a typed model, and the handler must call that method.
+- **Verify authentication is enforced in the engine** — the handler must delegate to an engine method that authenticates via `request_context` (`_authenticate_tenant`, typically through `get_bank_profile`). A handler that reads/writes tenant-scoped data without an engine method enforcing auth is a **must fix** (tenant data could leak across schemas).
+
 ### 8. Check code comments

 For each non-trivial change:
@@ -196,6 +214,8 @@ Present a clear summary organized by severity:
 - Raw dict usage for structured data (including internal code)
 - Multi-item tuple returns (including internal code)
 - Missing tests for new endpoints
+- Direct DB access (raw SQL / `acquire_with_retry` / `fq_table`) in an `api/` handler instead of a `MemoryEngine` method
+- Tenant-scoped data accessed without authentication enforced in the engine (`_authenticate_tenant` / `get_bank_profile`)
 - New integration missing tests, CI job, or release-integration.sh entry
 - New PostgreSQL table missing from `BACKUP_TABLES` in `admin/cli.py` (silent data loss on restore)

@@ -25,7 +25,7 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # Example: MiniMax configuration (1M context window)
 # HINDSIGHT_API_LLM_PROVIDER=minimax
 # HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
-# HINDSIGHT_API_LLM_MODEL=MiniMax-M2.7
+# HINDSIGHT_API_LLM_MODEL=MiniMax-M3  # or MiniMax-M2.7 for the previous generation

 # Example: DeepSeek configuration (https://api.deepseek.com)
 # HINDSIGHT_API_LLM_PROVIDER=deepseek
@@ -80,10 +80,23 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER=

 # Embeddings Configuration (Optional - uses local by default)
-# Provider: "local" (default), "tei", "openai", "cohere", "google", "openrouter", "zeroentropy", "litellm", or "litellm-sdk"
+# Provider: "local" (default), "onnx", "tei", "openai", "cohere", "google", "openrouter", "zeroentropy", "litellm", or "litellm-sdk"
 # HINDSIGHT_API_EMBEDDINGS_PROVIDER=local
 # For local provider:
 # HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=BAAI/bge-small-en-v1.5
+# For ONNX provider (local CPU embeddings without an Ollama/TEI sidecar):
+# HINDSIGHT_API_EMBEDDINGS_PROVIDER=onnx
+# HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_ID=intfloat/multilingual-e5-small
+# HINDSIGHT_API_EMBEDDINGS_ONNX_FILE=onnx/model.onnx
+# HINDSIGHT_API_EMBEDDINGS_ONNX_DIMENSIONS=384
+# HINDSIGHT_API_EMBEDDINGS_ONNX_MAX_TOKENS=512
+# HINDSIGHT_API_EMBEDDINGS_ONNX_POOLING=mean
+# HINDSIGHT_API_EMBEDDINGS_ONNX_NORMALIZE=true
+# HINDSIGHT_API_EMBEDDINGS_ONNX_QUERY_PREFIX="query: "
+# HINDSIGHT_API_EMBEDDINGS_ONNX_PASSAGE_PREFIX="passage: "
+# Optional for local model paths or pre-downloaded artifacts:
+# HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH=/models/multilingual-e5-small/onnx/model.onnx
+# HINDSIGHT_API_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH=/models/multilingual-e5-small
 # Optional for China network / restricted HF access:
 # HF_ENDPOINT=https://hf-mirror.com
 # For TEI provider:
@@ -24,6 +24,7 @@ on:
          - recall
          - recall-with-observations
          - consolidation
+          - graph-maintenance
        default: ""
      locomo_conversations:
        description: "LoComo conversation IDs (space-separated). Blank = curated set (conv-26 conv-30 conv-43)."
@@ -33,6 +34,18 @@ on:
        description: "Skip LoComo job"
        type: boolean
        default: false
+      obs_skip:
+        description: "Skip observation-dedup benchmark job"
+        type: boolean
+        default: false
+      obs_dataset:
+        description: "Obs benchmark dataset substring (blank = English hermes transcript)."
+        type: string
+        default: ""
+      obs_fraction:
+        description: "Obs benchmark fraction (0-1] of each document to run."
+        type: string
+        default: "1.0"
      ref:
        description: "Git ref to test (branch, tag, or SHA). Defaults to main."
        type: string
@@ -198,3 +211,86 @@ jobs:
        PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: ./scripts/benchmarks/publish-locomo-results.sh hindsight-dev/benchmarks/locomo/results/benchmark_results.json
+
+  obs:
+    # Observation-dedup quality benchmark: ingests a transcript, drains consolidation
+    # (serial SyncTaskBackend + embedded pg0 — no external DB / worker), and reports the
+    # near-duplicate observation rate. Real LLM via VertexAI, mirroring the LoComo job.
+    if: inputs.obs_skip != true
+    runs-on: ubuntu-latest
+    env:
+      HINDSIGHT_API_LLM_PROVIDER: vertexai
+      HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
+      HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
+      HINDSIGHT_API_ENABLE_OBSERVATIONS: "true"
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ inputs.ref || github.ref }}
+
+    - name: Setup GCP credentials
+      run: |
+        printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
+        PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
+        echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Cache HuggingFace models
+      uses: actions/cache@v5
+      with:
+        path: ~/.cache/huggingface
+        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
+        restore-keys: |
+          ${{ runner.os }}-huggingface-
+
+    - name: Pre-download models
+      working-directory: ./hindsight-api-slim
+      run: |
+        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
+        from sentence_transformers import SentenceTransformer
+        print('Downloading embedding model...')
+        SentenceTransformer('BAAI/bge-small-en-v1.5')
+        print('Model downloaded successfully')
+        "
+
+    - name: Install hindsight-dev dependencies
+      run: |
+        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
+
+    - name: Run obs benchmark
+      # Default to the English hermes transcript at full fraction — a clean, deterministic
+      # consolidation-dedup signal (the Chinese variant adds a cross-lingual embedding
+      # confound). Override dataset/fraction via workflow_dispatch.
+      run: |
+        DATASET="${{ inputs.obs_dataset }}"
+        if [ -z "$DATASET" ]; then DATASET="hermes_session_2026-05-15_en"; fi
+        FRACTION="${{ inputs.obs_fraction }}"
+        if [ -z "$FRACTION" ]; then FRACTION="1.0"; fi
+        cd hindsight-dev
+        uv run python -m benchmarks.obs.obs_benchmark \
+          --dataset "$DATASET" --fraction "$FRACTION" --wipe-bank --output obs-results.json
+
+    - name: Upload obs results
+      if: always()
+      uses: actions/upload-artifact@v7
+      with:
+        name: obs-results-${{ github.sha }}
+        path: hindsight-dev/obs-results.json
+        retention-days: 90
+
+    - name: Publish obs to dashboard
+      if: success() && (github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch')
+      env:
+        PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
+        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run: ./scripts/benchmarks/publish-obs-results.sh hindsight-dev/obs-results.json
@@ -39,20 +39,25 @@ jobs:
      integrations-litellm: ${{ steps.filter.outputs.integrations-litellm }}
      integrations-pydantic-ai: ${{ steps.filter.outputs.integrations-pydantic-ai }}
      integrations-ag2: ${{ steps.filter.outputs.integrations-ag2 }}
+      integrations-autogen: ${{ steps.filter.outputs.integrations-autogen }}
+      integrations-langgraph: ${{ steps.filter.outputs.integrations-langgraph }}
      integrations-llamaindex: ${{ steps.filter.outputs.integrations-llamaindex }}
      integrations-paperclip: ${{ steps.filter.outputs.integrations-paperclip }}
      integrations-opencode: ${{ steps.filter.outputs.integrations-opencode }}
      integrations-n8n: ${{ steps.filter.outputs.integrations-n8n }}
      integrations-cloudflare-oauth-proxy: ${{ steps.filter.outputs.integrations-cloudflare-oauth-proxy }}
+      integrations-superagent: ${{ steps.filter.outputs.integrations-superagent }}
      integrations-lockfiles: ${{ steps.filter.outputs.integrations-lockfiles }}
      integrations-openai-agents: ${{ steps.filter.outputs.integrations-openai-agents }}
      integrations-pipecat: ${{ steps.filter.outputs.integrations-pipecat }}
      integrations-agentcore: ${{ steps.filter.outputs.integrations-agentcore }}
      integrations-smolagents: ${{ steps.filter.outputs.integrations-smolagents }}
+      integrations-claude-agent-sdk: ${{ steps.filter.outputs.integrations-claude-agent-sdk }}
      integrations-dify: ${{ steps.filter.outputs.integrations-dify }}
      integrations-gemini-spark: ${{ steps.filter.outputs.integrations-gemini-spark }}
      integrations-vapi: ${{ steps.filter.outputs.integrations-vapi }}
      integrations-flowise: ${{ steps.filter.outputs.integrations-flowise }}
+      integrations-google-adk: ${{ steps.filter.outputs.integrations-google-adk }}
      tools-agent-sdk: ${{ steps.filter.outputs.tools-agent-sdk }}
      integrations-roo-code: ${{ steps.filter.outputs.integrations-roo-code }}
      dev: ${{ steps.filter.outputs.dev }}
@@ -124,6 +129,10 @@ jobs:
            - 'hindsight-integrations/pydantic-ai/**'
          integrations-ag2:
            - 'hindsight-integrations/ag2/**'
+          integrations-autogen:
+            - 'hindsight-integrations/autogen/**'
+          integrations-langgraph:
+            - 'hindsight-integrations/langgraph/**'
          integrations-llamaindex:
            - 'hindsight-integrations/llamaindex/**'
          integrations-paperclip:
@@ -134,6 +143,8 @@ jobs:
            - 'hindsight-integrations/n8n/**'
          integrations-cloudflare-oauth-proxy:
            - 'hindsight-integrations/cloudflare-oauth-proxy/**'
+          integrations-superagent:
+            - 'hindsight-integrations/superagent/**'
          integrations-lockfiles:
            - 'hindsight-integrations/*/package-lock.json'
            - 'hindsight-integrations/*/package.json'
@@ -146,6 +157,8 @@ jobs:
            - 'hindsight-integrations/agentcore/**'
          integrations-smolagents:
            - 'hindsight-integrations/smolagents/**'
+          integrations-claude-agent-sdk:
+            - 'hindsight-integrations/claude-agent-sdk/**'
          integrations-dify:
            - 'hindsight-integrations/dify/**'
          integrations-gemini-spark:
@@ -154,6 +167,8 @@ jobs:
            - 'hindsight-integrations/vapi/**'
          integrations-flowise:
            - 'hindsight-integrations/flowise/**'
+          integrations-google-adk:
+            - 'hindsight-integrations/google-adk/**'
          tools-agent-sdk:
            - 'hindsight-tools/hindsight-agent-sdk/**'
          integrations-roo-code:
@@ -728,6 +743,43 @@ jobs:
      working-directory: ./hindsight-integrations/pipecat
      run: uv run pytest tests -v

+  test-google-adk-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-google-adk == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build google-adk integration
+      working-directory: ./hindsight-integrations/google-adk
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/google-adk
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/google-adk
+      run: uv run pytest tests -v
+
  test-gemini-spark-integration:
    needs: [detect-changes]
    if: >-
@@ -2761,6 +2813,45 @@ jobs:
      working-directory: ./hindsight-integrations/ag2
      run: uv run pytest tests -v

+  test-autogen-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-autogen == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build autogen integration
+      working-directory: ./hindsight-integrations/autogen
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/autogen
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/autogen
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
  test-smolagents-integration:
    needs: [detect-changes]
    if: >-
@@ -2939,6 +3030,45 @@ jobs:
      working-directory: ./hindsight-integrations/vapi
      run: uv run pytest tests -v

+  test-superagent-integration:
+    needs: [detect-changes]
+    if: >-
+      github.event_name != 'pull_request_review' &&
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-superagent == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build superagent integration
+      working-directory: ./hindsight-integrations/superagent
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/superagent
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/superagent
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs live Hindsight + provider keys and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
  test-litellm-integration:
    needs: [detect-changes]
    if: >-
@@ -2974,7 +3104,9 @@ jobs:

    - name: Run tests
      working-directory: ./hindsight-integrations/litellm
-      run: uv run pytest tests -v
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs live Hindsight + provider keys and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"

  test-pydantic-ai-integration:
    needs: [detect-changes]
@@ -3013,6 +3145,45 @@ jobs:
      working-directory: ./hindsight-integrations/pydantic-ai
      run: uv run pytest tests -v

+  test-langgraph-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-langgraph == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build langgraph integration
+      working-directory: ./hindsight-integrations/langgraph
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/langgraph
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/langgraph
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
  test-llamaindex-integration:
    needs: [detect-changes]
    if: >-
@@ -3048,7 +3219,9 @@ jobs:

    - name: Run tests
      working-directory: ./hindsight-integrations/llamaindex
-      run: uv run pytest tests -v
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"

  test-openai-agents-integration:
    needs: [detect-changes]
@@ -3085,7 +3258,47 @@ jobs:

    - name: Run tests
      working-directory: ./hindsight-integrations/openai-agents
-      run: uv run pytest tests -v
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
+  test-claude-agent-sdk-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-claude-agent-sdk == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build claude-agent-sdk integration
+      working-directory: ./hindsight-integrations/claude-agent-sdk
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/claude-agent-sdk
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/claude-agent-sdk
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"

  test-agentcore-integration:
    needs: [detect-changes]
@@ -3960,6 +4173,7 @@ jobs:
      - test-pipecat-integration
      - test-gemini-spark-integration
      - test-vapi-integration
+      - test-google-adk-integration
      - test-roo-code-integration
      - build-control-plane
      - build-docs
@@ -3980,19 +4194,24 @@ jobs:
      - test-openclaw-integration
      - test-integration
      - test-ag2-integration
+      - test-autogen-integration
      - test-smolagents-integration
      - test-dify-integration
      - test-flowise-integration
      - test-crewai-integration
+      - test-langgraph-integration
+      - test-superagent-integration
      - test-litellm-integration
      - test-pydantic-ai-integration
      - test-llamaindex-integration
+      - test-openai-agents-integration
      - test-agentcore-integration
      - test-pip-slim
      - test-embed
      - test-embed-windows
      - test-hindsight-all
      - test-hindsight-agent-sdk
+      - test-claude-agent-sdk-integration
      - test-doc-examples
      - test-upgrade
      - verify-generated-files
@@ -220,6 +220,30 @@ migration file dispatches through `run_for_dialect`, which calls either

 **MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.

+### Testing
+
+Most tests are deterministic (MockLLM, pure functions) — assert directly.
+
+**Tests that verify LLM behaviour use a real LLM + an LLM-as-judge.** When the thing under test is *how the model interprets a prompt* (classification, attribution, dimension preservation, instruction-following), MockLLM can't simulate it and exact string/enum asserts flake across providers and runs. Use this pattern instead:
+
+1. Mark the test module `pytestmark = pytest.mark.hs_llm_core` (single-provider; CI runs it in the core-LLM job). Use `hs_llm_mat` only for provider-matrix acceptance tests.
+2. Call the real pipeline (`LLMConfig.from_env()`, `_get_raw_config()`), e.g. `extract_facts_from_text(...)`.
+3. Assert with the judge, not string matching:
+   ```python
+   from tests.llm_judge import assert_meets_criteria
+   facts_summary = "\n".join(f"- [{f.fact_type}] {f.fact}" for f in facts)
+   await assert_meets_criteria(
+       response=facts_summary,
+       criteria="The first-person user statements are classified 'world' and attributed to the user, not the agent.",
+       context="What the input said and who was speaking.",
+   )
+   ```
+
+Rules of thumb:
+- **Judge anything non-deterministic** — including `fact_type` classification and speaker attribution. Do NOT hard-assert `fact_type == "..."`; pass a `[fact_type] fact` summary to the judge instead. Structural facts that ARE deterministic (counts, presence of a field, that a substring was injected into a prompt) stay as direct asserts in fast unit tests.
+- **Split the test surface**: cover the deterministic mechanics (prompt assembly, suppression logic) with fast non-LLM unit tests, and the model-following behaviour with one `hs_llm_core` judge test. (Example pair: `test_narrator_resolution.py` + `test_narrator_context_override.py`.)
+- The judge model is independent of the test provider (defaults to Gemini); never judge with the same call you're testing.
+
 ### Memory Banks
 - Each bank is an isolated memory store (like a "brain" for one user/agent)
 - Banks have dispositions (skepticism, literalism, empathy traits 1-5) affecting reflect
@@ -9,13 +9,36 @@ Thanks for your interest in contributing to Hindsight!
   git clone [email protected]:vectorize-io/hindsight.git
   cd hindsight
   ```
-2. Set up your environment:
+
+2. Bootstrap your dev environment in one shot:
+   ```bash
+   ./scripts/dev/setup.sh
+   ```
+   This is idempotent (safe to re-run) and gets you ready to develop, including
+   offline. It:
+   - installs the required toolchains if missing (uv/Python, Node/npm, Rust/cargo),
+   - creates `.env` from `.env.example` (remember to add your LLM API key),
+   - configures git hooks,
+   - installs all Python and Node workspace dependencies,
+   - pre-downloads the local ML models + tokenizer so the API runs offline,
+   - builds the TypeScript SDK and the Rust CLI.
+
+   Useful flags: `--skip-build` (deps only), `--skip-models` (skip ML model
+   download), `--with-docs` (also build the docs site), `--force` (rebuild
+   artifacts). Docker image builds are out of scope. Run
+   `./scripts/dev/setup.sh --help` for details.
+
+### Manual setup
+
+If you'd rather set things up by hand instead of running the script above:
+
+1. Set up your environment:
   ```bash
   cp .env.example .env
   ```
   Edit the .env to add LLM API key and config as required

-3. Install dependencies:
+2. Install dependencies:
   ```bash
   # Python dependencies
   uv sync --directory hindsight-api/
@@ -62,9 +62,9 @@ If you need more control over how and when your agent stores and recalls memorie
 ```bash
 export OPENAI_API_KEY=sk-xxx

-docker run --rm -it --pull always -p 8888:8888 -p 9999:9999 \
+docker run -it --pull always --name hindsight --restart unless-stopped -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
-  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
+  -v hindsight-data:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest
 ```

@@ -50,6 +50,8 @@ WORKDIR /app/api
 # Sync dependencies using appropriate extras based on INCLUDE_LOCAL_MODELS
 # local-ml: torch, sentence-transformers, transformers, einops, flashrank, mlx (optional)
 # embedded-db: pg0-embedded (always included for embedded PostgreSQL support)
+# ONNX Runtime embeddings are intentionally not bundled into the official
+# standalone image; install the local-onnx extra in custom images when needed.
 RUN if [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
        uv sync --extra local-ml --extra embedded-db; \
    else \
@@ -43,11 +43,56 @@ check_pg0_data_integrity() {
    return 0
 }

+# =============================================================================
+# Embedded pg0 writability pre-check (#1483)
+#
+# The container runs as the unprivileged `hindsight` user (UID 1000). When the
+# pg0 data directory is a host bind mount (e.g. `-v $HOME/dir:/home/hindsight/.pg0`)
+# that is not owned by UID 1000 — the default on macOS Docker Desktop and most
+# non-1000 Linux hosts — pg0 fails with the opaque "Permission denied (os error
+# 13)". We cannot chown it ourselves without root (and the image is deliberately
+# rootless), so we surface an actionable message up front instead.
+#
+# Docker *named* volumes are seeded with the image directory's ownership (UID
+# 1000) on first use, so they avoid this entirely — hence the named-volume
+# recommendation below and in the README.
+# =============================================================================
+check_pg0_writable() {
+    local pg0_data_dir="$1"
+
+    # Only relevant for embedded pg0; an external database doesn't use this dir.
+    if [ -n "${HINDSIGHT_API_DATABASE_URL:-}" ]; then
+        return 0
+    fi
+
+    mkdir -p "$pg0_data_dir" 2>/dev/null || true
+    if touch "$pg0_data_dir/.hindsight-write-test" 2>/dev/null; then
+        rm -f "$pg0_data_dir/.hindsight-write-test" 2>/dev/null || true
+        return 0
+    fi
+
+    echo "❌ The embedded database directory $pg0_data_dir is not writable by this container (UID $(id -u))."
+    echo ""
+    echo "   A host directory was bind-mounted but is not owned by the container user (UID 1000)."
+    echo "   Hindsight runs rootless and cannot fix this for you. Choose one:"
+    echo ""
+    echo "   • Recommended — use a Docker named volume (auto-owned by the container):"
+    echo "       -v hindsight-data:/home/hindsight/.pg0"
+    echo ""
+    echo "   • Or keep the host path and run as your host user, chowning it to match:"
+    echo "       sudo chown -R \$(id -u):\$(id -g) <host-directory>"
+    echo "       docker run --user \$(id -u):\$(id -g) -e HOME=/home/hindsight ..."
+    echo ""
+    echo "   See https://github.com/vectorize-io/hindsight/issues/1483"
+    return 1
+}
+
 if [ "${HINDSIGHT_START_ALL_SOURCE_ONLY:-false}" = "true" ]; then
    return 0 2>/dev/null || exit 0
 fi

 check_pg0_data_integrity "${HOME}/.pg0"
+check_pg0_writable "${HOME}/.pg0" || exit 1

 # Service flags (default to true if not set)
 ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"
@@ -8,7 +8,7 @@ source "$SCRIPT_DIR/start-all.sh"
 unset HINDSIGHT_START_ALL_SOURCE_ONLY

 TMP_DIR="$(mktemp -d)"
-trap 'rm -rf "$TMP_DIR"' EXIT
+trap 'chmod -R u+rwx "$TMP_DIR" 2>/dev/null || true; rm -rf "$TMP_DIR"' EXIT

 assert_contains() {
    local output="$1"
@@ -71,3 +71,51 @@ nonempty_output="$(check_pg0_data_integrity "$TMP_DIR/nonempty")"
 assert_contains "$nonempty_output" "WARNING: pg0 data directory exists"

 echo "start-all pg0 integrity checks passed"
+
+# =============================================================================
+# check_pg0_writable (#1483)
+# These rely on filesystem permissions, which root bypasses; skip under root.
+# =============================================================================
+if [ "$(id -u)" != "0" ]; then
+    # Writable directory: returns 0, prints nothing, leaves no artifact behind.
+    mkdir -p "$TMP_DIR/writable"
+    writable_output="$(check_pg0_writable "$TMP_DIR/writable")"
+    assert_empty "$writable_output"
+    if [ -e "$TMP_DIR/writable/.hindsight-write-test" ]; then
+        echo "check_pg0_writable left its write-test file behind"
+        exit 1
+    fi
+
+    # Non-writable directory: returns 1 with actionable guidance.
+    mkdir -p "$TMP_DIR/readonly"
+    chmod 000 "$TMP_DIR/readonly"
+    set +e
+    readonly_output="$(check_pg0_writable "$TMP_DIR/readonly" 2>&1)"
+    readonly_rc=$?
+    set -e
+    chmod 755 "$TMP_DIR/readonly"
+    if [ "$readonly_rc" -eq 0 ]; then
+        echo "check_pg0_writable should fail on a non-writable directory"
+        exit 1
+    fi
+    assert_contains "$readonly_output" "not writable"
+    assert_contains "$readonly_output" "hindsight-data:/home/hindsight/.pg0"
+    assert_contains "$readonly_output" "--user"
+
+    # External database configured: skip the check regardless of dir perms.
+    mkdir -p "$TMP_DIR/extdb"
+    chmod 000 "$TMP_DIR/extdb"
+    set +e
+    HINDSIGHT_API_DATABASE_URL="postgres://x" check_pg0_writable "$TMP_DIR/extdb" >/dev/null 2>&1
+    extdb_rc=$?
+    set -e
+    chmod 755 "$TMP_DIR/extdb"
+    if [ "$extdb_rc" -ne 0 ]; then
+        echo "check_pg0_writable should skip when an external database is configured"
+        exit 1
+    fi
+
+    echo "start-all pg0 writability checks passed"
+else
+    echo "⚠️  Running as root; skipping pg0 writability checks (permissions are bypassed)."
+fi
@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.7.1
-appVersion: "0.7.1"
+version: 0.7.2
+appVersion: "0.7.2"
 keywords:
  - ai
  - memory
@@ -1,6 +1,6 @@
 {
  "name": "@vectorize-io/hindsight-all",
-  "version": "0.7.1",
+  "version": "0.7.2",
  "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
  "main": "dist/index.js",
  "types": "dist/index.d.ts",
@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "hindsight-all-slim"
-version = "0.7.1"
+version = "0.7.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim==0.7.1",
+    "hindsight-api-slim==0.7.2",
    "hindsight-client>=0.0.7",
    "hindsight-embed>=0.1.0",
 ]
@@ -4,12 +4,12 @@ build-backend = "hatchling.build"

 [project]
 name = "hindsight-all"
-version = "0.7.1"
+version = "0.7.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim[all]==0.7.1",
+    "hindsight-api-slim[all]==0.7.2",
    "hindsight-client>=0.0.7",
    "hindsight-embed>=0.1.0",
 ]
@@ -21,7 +21,7 @@ hindsight-embed = { workspace = true }

 [project.optional-dependencies]
 local-llm = [
-    "hindsight-api-slim[local-llm]==0.7.1",
+    "hindsight-api-slim[local-llm]==0.7.2",
 ]
 test = [
    "pytest>=7.0.0",
@@ -99,7 +99,7 @@ hindsight-api
 ## Docker

 ```bash
-docker run --rm -it -p 8888:8888 \
+docker run -it --name hindsight --restart unless-stopped -p 8888:8888 \
  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest
@@ -53,4 +53,4 @@ __all__ = [
    "RemoteTEICrossEncoder",
    "LLMConfig",
 ]
-__version__ = "0.7.1"
+__version__ = "0.7.2"
@@ -17,7 +17,9 @@ import asyncpg
 import typer

 from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig
+from ..engine.memory_engine import _current_schema
 from ..engine.schema import fq_table_explicit as _fq_table
+from ..engine.transfer import export_bank
 from ..extensions import TenantExtension, load_extension
 from ..pg0 import parse_pg0_url, resolve_database_url

@@ -50,18 +52,38 @@ BACKUP_TABLES = [
    "unit_entities",
    "entity_cooccurrences",
    "memory_links",
+    "observation_history",
    "mental_models",
+    "mental_model_history",
    "directives",
    "async_operations",
    "webhooks",
    "file_storage",
    "audit_log",
+    "llm_requests",
    "graph_maintenance_queue",
 ]

 MANIFEST_VERSION = "1"


+async def _admin_connect(db_url: str) -> asyncpg.Connection:
+    """Open a raw asyncpg connection to an admin DB URL.
+
+    ``resolve_database_url`` handles both plain ``postgres://`` (passthrough) and
+    ``pg0://`` (boots the embedded server and returns its real libpq URL), so this
+    is the only step needed to connect. JSON codecs are registered so ``jsonb``
+    columns decode to Python objects (used by the export row dumps).
+    """
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    conn = await asyncpg.connect(await resolve_database_url(db_url))
+    for type_name in ("json", "jsonb"):
+        await conn.set_type_codec(type_name, encoder=json.dumps, decoder=json.loads, schema="pg_catalog")
+    return conn
+
+
 async def _backup(database_url: str, output_path: Path, schema: str = "public") -> dict[str, Any]:
    """Backup all tables to a zip file using binary COPY protocol."""
    conn = await asyncpg.connect(database_url)
@@ -330,6 +352,123 @@ def run_db_migration(
    typer.echo(f"Database migrations completed successfully for {len(schemas)} schema(s)")


+async def _run_export_bank(db_url: str, bank_id: str, output: Path, schema: str, include_history: bool) -> int:
+    """Export a whole bank to a ZIP archive."""
+    conn = await _admin_connect(db_url)
+    try:
+        # export_bank resolves table names via fq_table (the _current_schema
+        # contextvar); set it so the raw connection targets the right schema.
+        _current_schema.set(schema)
+        data = await export_bank(conn, bank_id, include_history=include_history)
+    finally:
+        await conn.close()
+
+    output.write_bytes(data)
+    return len(data)
+
+
+@app.command(name="export-bank")
+def export_bank_command(
+    bank_id: str = typer.Option(..., "--bank", "-b", help="Bank id to export."),
+    output: Path = typer.Option(..., "--output", "-o", help="Path to write the .zip archive."),
+    schema: str | None = typer.Option(
+        None,
+        "--schema",
+        "-s",
+        help="Database schema the bank lives in. Defaults to the configured base schema.",
+    ),
+    include_history: bool = typer.Option(
+        False,
+        "--include-history",
+        help="Also export operational history (audit_log, llm_requests). Off by default.",
+    ),
+):
+    """Export an entire bank to a portable ZIP (no embeddings — regenerated on import).
+
+    Carries documents, facts, observations, bank config, mental models, directives
+    and webhooks so the bank can be imported into a new instance configured with a
+    different embedding model / vector / text-search backend.
+    """
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    target_schema = schema or config.database_schema or DEFAULT_DATABASE_SCHEMA
+    typer.echo(f"Exporting bank '{bank_id}' from schema '{target_schema}'...")
+
+    size = asyncio.run(_run_export_bank(config.database_url, bank_id, output, target_schema, include_history))
+
+    typer.echo(f"Exported bank '{bank_id}' to {output} ({size} bytes)")
+
+
+async def _run_import_bank(archive_path: Path, schema: str, target_bank_id: str | None, include_history: bool):
+    """Boot a MemoryEngine (for the target's embedding model) and restore a bank archive."""
+    # MemoryEngine is heavy (loads embeddings); import it lazily so other admin
+    # commands don't pay for it. _current_schema is imported at module top.
+    from ..engine.memory_engine import MemoryEngine
+    from ..models import RequestContext
+
+    archive_bytes = archive_path.read_bytes()
+    # run_migrations=True so a fresh target instance is provisioned at this
+    # instance's embedding dimension / vector / text-search backend before restore.
+    engine = MemoryEngine(run_migrations=True)
+    await engine.initialize()
+    try:
+        _current_schema.set(schema)
+        context = RequestContext(internal=True, user_initiated=True)
+        return await engine.import_bank_async(
+            archive_bytes,
+            context,
+            target_bank_id=target_bank_id,
+            include_history=include_history,
+        )
+    finally:
+        await engine.close()
+
+
+@app.command(name="import-bank")
+def import_bank_command(
+    archive: Path = typer.Option(..., "--archive", "-a", help="Path to the .zip produced by export-bank."),
+    schema: str | None = typer.Option(
+        None, "--schema", "-s", help="Target schema. Defaults to the configured base schema."
+    ),
+    target_bank: str | None = typer.Option(
+        None, "--target-bank", help="Override the bank id (defaults to the archive's source bank)."
+    ),
+    include_history: bool = typer.Option(
+        False, "--include-history", help="Also restore operational history if present in the archive."
+    ),
+):
+    """Restore a whole bank from an export-bank archive into THIS instance.
+
+    Re-embeds facts with this instance's configured embedding model and rebuilds
+    links and indexes — the import half of a cross-instance migration. Run against
+    an instance configured with the desired embedding / vector / text-search backend.
+    The target bank must not already exist (import restores a whole bank, not a merge).
+    """
+    config = HindsightConfig.from_env()
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    target_schema = schema or config.database_schema or DEFAULT_DATABASE_SCHEMA
+    typer.echo(f"Importing bank archive '{archive}' into schema '{target_schema}'...")
+
+    result = asyncio.run(_run_import_bank(archive, target_schema, target_bank, include_history))
+
+    typer.echo(
+        f"Imported bank '{result.bank_id}': {result.documents_imported} doc(s), "
+        f"{result.facts_imported} fact(s), {result.observations_imported} observation(s), "
+        f"{result.mental_models_imported} mental model(s), "
+        f"{result.mental_model_history_imported} mm-history row(s), {result.directives_imported} directive(s), "
+        f"{result.webhooks_imported} webhook(s), {result.history_rows_imported} history row(s)"
+    )
+
+
 async def _decommission_worker(db_url: str, worker_id: str, schema: str = "public") -> int:
    """Release all tasks owned by a worker, setting them back to pending status."""
    is_pg0, instance_name, _ = parse_pg0_url(db_url)
@@ -0,0 +1,253 @@
+"""Move mental-model and observation history into dedicated tables.
+
+Both histories were accumulated in a single JSONB/CLOB ``history`` column
+(``mental_models.history`` and ``memory_units.history``), appended to on every
+update. That design has two problems:
+
+1. **Unbounded growth on observations.** The observation write path appended a
+   snapshot on every update with no cap at all, so a frequently-reinforced
+   observation grew its ``history`` array until it crossed Postgres's hard 256MB
+   jsonb limit (SQLSTATE 54000), after which every further UPDATE failed and the
+   row was stuck.
+2. **Wrong-axis cap on mental models.** The mental-model cap bounded the *number*
+   of entries (50), not their *size* — a single large reflect snapshot could
+   still blow the budget — and rewrote the whole array (plus TOAST) on every
+   refresh, defeating HOT updates.
+
+This migration creates one row per history entry in two dedicated tables, with
+an index that makes "most recent N for this item" cheap, then drops the old
+columns. The cap is now enforced at write time as a bounded DELETE of the
+oldest over-cap rows (see config ``*_HISTORY_MAX_ENTRIES``).
+
+Revision ID: a7b8c9d0e1f2
+Revises: d3e4f5a6b7c8
+Create Date: 2026-06-05
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "a7b8c9d0e1f2"
+down_revision: str | Sequence[str] | None = "d3e4f5a6b7c8"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+# ---------------------------------------------------------------------------
+# PostgreSQL
+# ---------------------------------------------------------------------------
+def _pg_upgrade() -> None:
+    schema = _get_schema_prefix()
+
+    # Both tables share the same shape: surrogate id, FK to the parent, bank_id,
+    # the snapshot payload as a single JSONB ``content`` blob, and changed_at.
+    # The payload is per-row (one change per row) so it stays small — this is NOT
+    # the old single-column-grows-forever design; growth is bounded by row count
+    # plus the write-time cap. Folding the previous_* fields into one JSONB keeps
+    # the schema dialect-simple (no array columns) and flexible.
+
+    # --- mental_model_history -------------------------------------------------
+    # content: {"previous_content": ..., "previous_reflect_response": {...}}
+    op.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS {schema}mental_model_history (
+            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+            mental_model_id VARCHAR(64) NOT NULL,
+            bank_id VARCHAR(64) NOT NULL,
+            content JSONB NOT NULL,
+            changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
+            FOREIGN KEY (mental_model_id, bank_id)
+                REFERENCES {schema}mental_models(id, bank_id) ON DELETE CASCADE
+        )
+        """
+    )
+    op.execute(
+        "CREATE INDEX IF NOT EXISTS idx_mm_history_model "
+        f"ON {schema}mental_model_history (bank_id, mental_model_id, changed_at DESC, id DESC)"
+    )
+
+    # --- observation_history --------------------------------------------------
+    # content: {"previous_text", "previous_tags", "previous_occurred_start",
+    #           "previous_occurred_end", "previous_mentioned_at", "new_source_memory_ids"}
+    op.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS {schema}observation_history (
+            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
+            observation_id UUID NOT NULL,
+            bank_id VARCHAR(64) NOT NULL,
+            content JSONB NOT NULL,
+            changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
+            FOREIGN KEY (observation_id)
+                REFERENCES {schema}memory_units(id) ON DELETE CASCADE
+        )
+        """
+    )
+    op.execute(
+        "CREATE INDEX IF NOT EXISTS idx_observation_history_obs "
+        f"ON {schema}observation_history (observation_id, changed_at DESC, id DESC)"
+    )
+
+    # --- backfill mental models ----------------------------------------------
+    # Explode each row's history array into rows, preserving chronological order
+    # via WITH ORDINALITY so the IDENTITY id tie-breaks oldest->newest correctly.
+    # changed_at is promoted to its own column; the rest of the element becomes
+    # ``content`` (the ``- 'changed_at'`` strips the now-redundant key).
+    op.execute(
+        f"""
+        INSERT INTO {schema}mental_model_history (mental_model_id, bank_id, content, changed_at)
+        SELECT mm.id, mm.bank_id,
+               e - 'changed_at',
+               COALESCE(NULLIF(e->>'changed_at', '')::timestamptz, now())
+        FROM {schema}mental_models mm
+             CROSS JOIN LATERAL jsonb_array_elements(mm.history) WITH ORDINALITY a(e, ord)
+        WHERE mm.history IS NOT NULL
+          AND jsonb_typeof(mm.history) = 'array'
+          AND jsonb_array_length(mm.history) > 0
+        ORDER BY mm.id, mm.bank_id, ord
+        """
+    )
+
+    # --- backfill observations -----------------------------------------------
+    op.execute(
+        f"""
+        INSERT INTO {schema}observation_history (observation_id, bank_id, content, changed_at)
+        SELECT mu.id, mu.bank_id,
+               e - 'changed_at',
+               COALESCE(NULLIF(e->>'changed_at', '')::timestamptz, now())
+        FROM {schema}memory_units mu
+             CROSS JOIN LATERAL jsonb_array_elements(mu.history) WITH ORDINALITY a(e, ord)
+        WHERE mu.fact_type = 'observation'
+          AND mu.history IS NOT NULL
+          AND jsonb_typeof(mu.history) = 'array'
+          AND jsonb_array_length(mu.history) > 0
+        ORDER BY mu.id, ord
+        """
+    )
+
+    # --- drop the legacy columns ---------------------------------------------
+    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS history")
+    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS history")
+
+
+def _pg_downgrade() -> None:
+    schema = _get_schema_prefix()
+
+    # Re-add the columns (empty — historical content is not reconstructed back
+    # into the array form; the dedicated tables are dropped below).
+    op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
+    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
+
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_observation_history_obs")
+    op.execute(f"DROP TABLE IF EXISTS {schema}observation_history")
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mm_history_model")
+    op.execute(f"DROP TABLE IF EXISTS {schema}mental_model_history")
+
+
+# ---------------------------------------------------------------------------
+# Oracle 23ai
+# ---------------------------------------------------------------------------
+def _oracle_upgrade() -> None:
+    # Same single-JSONB shape as PG: ``content`` holds the snapshot payload as a
+    # CLOB IS JSON. The legacy per-element JSON object (minus changed_at, promoted
+    # to its own column) is carried through verbatim on backfill — the array
+    # columns the previous design needed are gone.
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS mental_model_history (
+            id NUMBER GENERATED BY DEFAULT AS IDENTITY,
+            mental_model_id VARCHAR2(256) NOT NULL,
+            bank_id VARCHAR2(256) NOT NULL,
+            content CLOB NOT NULL
+                CONSTRAINT mmh_content_json CHECK (content IS JSON),
+            changed_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+            CONSTRAINT pk_mental_model_history PRIMARY KEY (id),
+            CONSTRAINT fk_mmh_model FOREIGN KEY (mental_model_id, bank_id)
+                REFERENCES mental_models(id, bank_id) ON DELETE CASCADE
+        )
+        """
+    )
+    op.execute(
+        "CREATE INDEX idx_mm_history_model ON mental_model_history (bank_id, mental_model_id, changed_at DESC, id DESC)"
+    )
+
+    op.execute(
+        """
+        CREATE TABLE IF NOT EXISTS observation_history (
+            id NUMBER GENERATED BY DEFAULT AS IDENTITY,
+            observation_id RAW(16) NOT NULL,
+            bank_id VARCHAR2(256) NOT NULL,
+            content CLOB NOT NULL
+                CONSTRAINT oh_content_json CHECK (content IS JSON),
+            changed_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+            CONSTRAINT pk_observation_history PRIMARY KEY (id),
+            CONSTRAINT fk_oh_obs FOREIGN KEY (observation_id)
+                REFERENCES memory_units(id) ON DELETE CASCADE
+        )
+        """
+    )
+    op.execute(
+        "CREATE INDEX idx_observation_history_obs ON observation_history (observation_id, changed_at DESC, id DESC)"
+    )
+
+    bind = op.get_bind()
+
+    # Backfill via JSON_TABLE. ``content`` is the whole element (FORMAT JSON PATH
+    # '$'); changed_at is also promoted to its own column. Backfilled content may
+    # therefore still carry a redundant changed_at key, which the read path
+    # ignores in favour of the column — harmless, and avoids JSON surgery here.
+    bind.exec_driver_sql(
+        """
+        INSERT INTO mental_model_history (mental_model_id, bank_id, content, changed_at)
+        SELECT mm.id, mm.bank_id, jt.content, NVL(jt.changed_at, SYSTIMESTAMP)
+        FROM mental_models mm,
+             JSON_TABLE(mm.history, '$[*]' COLUMNS (
+                 seq FOR ORDINALITY,
+                 content CLOB FORMAT JSON PATH '$',
+                 changed_at TIMESTAMP WITH TIME ZONE PATH '$.changed_at'
+             )) jt
+        WHERE mm.history IS NOT NULL
+        ORDER BY mm.id, mm.bank_id, jt.seq
+        """
+    )
+
+    bind.exec_driver_sql(
+        """
+        INSERT INTO observation_history (observation_id, bank_id, content, changed_at)
+        SELECT mu.id, mu.bank_id, jt.content, NVL(jt.changed_at, SYSTIMESTAMP)
+        FROM memory_units mu,
+             JSON_TABLE(mu.history, '$[*]' COLUMNS (
+                 seq FOR ORDINALITY,
+                 content CLOB FORMAT JSON PATH '$',
+                 changed_at TIMESTAMP WITH TIME ZONE PATH '$.changed_at'
+             )) jt
+        WHERE mu.fact_type = 'observation' AND mu.history IS NOT NULL
+        ORDER BY mu.id, jt.seq
+        """
+    )
+
+    op.execute("ALTER TABLE mental_models DROP COLUMN history")
+    op.execute("ALTER TABLE memory_units DROP COLUMN history")
+
+
+def _oracle_downgrade() -> None:
+    op.execute("ALTER TABLE mental_models ADD history CLOB DEFAULT '[]' NOT NULL")
+    op.execute("ALTER TABLE memory_units ADD history CLOB DEFAULT '[]'")
+    op.execute("DROP TABLE observation_history CASCADE CONSTRAINTS")
+    op.execute("DROP TABLE mental_model_history CASCADE CONSTRAINTS")
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)
@@ -0,0 +1,96 @@
+"""Add llm_requests table for per-bank LLM request tracing.
+
+Stores one row per logical LLM call Hindsight makes (success and failure),
+capturing the input messages, model output, token usage (input/output/cached/
+total), finish reason, and caller metadata. Disabled by default at the
+application layer (HINDSIGHT_API_LLM_TRACE_ENABLED); this migration only
+creates the table.
+
+PostgreSQL only — the tracing subsystem is not wired for Oracle, so the Oracle
+slot is intentionally absent (mirrors the audit_log table).
+
+Revision ID: d3e4f5a6b7c8
+Revises: c1d2e3f4a5b6
+Create Date: 2026-06-01
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "d3e4f5a6b7c8"
+down_revision: str | Sequence[str] | None = "c1d2e3f4a5b6"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    schema = _get_schema_prefix()
+
+    op.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS {schema}llm_requests (
+            id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+            bank_id TEXT,
+            operation TEXT,
+            scope TEXT,
+            -- OTel-style grouping: trace_id is shared by every LLM call of one
+            -- operation invocation (e.g. all calls of a single reflect run);
+            -- parent_span_id is that operation span; span_id is this call.
+            trace_id TEXT,
+            span_id TEXT,
+            parent_span_id TEXT,
+            provider TEXT,
+            model TEXT,
+            status TEXT NOT NULL,
+            started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+            ended_at TIMESTAMPTZ,
+            duration_ms INTEGER,
+            input_tokens INTEGER,
+            output_tokens INTEGER,
+            cached_tokens INTEGER,
+            total_tokens INTEGER,
+            input JSONB,
+            output JSONB,
+            error TEXT,
+            llm_info JSONB DEFAULT '{{}}'::jsonb,
+            metadata JSONB DEFAULT '{{}}'::jsonb
+        )
+        """
+    )
+
+    op.execute(
+        f"CREATE INDEX IF NOT EXISTS idx_llm_requests_bank_started ON {schema}llm_requests (bank_id, started_at DESC)"
+    )
+    op.execute(
+        f"CREATE INDEX IF NOT EXISTS idx_llm_requests_status_started ON {schema}llm_requests (status, started_at DESC)"
+    )
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_llm_requests_started ON {schema}llm_requests (started_at DESC)")
+    op.execute(
+        f"CREATE INDEX IF NOT EXISTS idx_llm_requests_trace ON {schema}llm_requests (bank_id, trace_id, started_at)"
+    )
+
+
+def _pg_downgrade() -> None:
+    schema = _get_schema_prefix()
+
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_llm_requests_started")
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_llm_requests_status_started")
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_llm_requests_bank_started")
+    op.execute(f"DROP TABLE IF EXISTS {schema}llm_requests")
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)
@@ -0,0 +1,153 @@
+"""Add server-side routines for background maintenance sweeps.
+
+Installs two PL/pgSQL discovery routines in the ``public`` schema. Both loop
+over every schema that actually holds the relevant table (via ``pg_class``), so
+a single function call covers all tenants in one round-trip instead of the
+per-tenant query storm that a client-side loop would create at thousands of
+tenants.
+
+- ``public.banks_needing_consolidation()`` -> (schema_name, bank_id) for banks
+  that have eligible-but-unscheduled facts (``consolidated_at IS NULL AND
+  consolidation_failed_at IS NULL`` for consolidatable fact types), have
+  auto-consolidation not explicitly disabled at the bank level, and have no
+  consolidation operation already pending/processing. Drives the periodic
+  reconcile that re-schedules consolidation after a terminal failure left facts
+  stranded (see HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS).
+
+- ``public.schemas_with_expired_rows(p_table, p_ts_col, p_days)`` -> schema
+  names that hold at least one ``p_table`` row older than ``p_days``. Drives the
+  cross-tenant retention sweeps for ``audit_log`` and ``llm_requests``; the loop
+  then issues a DELETE only against the returned schemas.
+
+These are read-only (STABLE) discovery routines — the caller performs the
+enqueue/DELETE — so installing them never mutates data.
+
+PostgreSQL only — the worker poller and these tables are not wired for Oracle,
+so the Oracle slot is intentionally absent (mirrors the audit_log / llm_requests
+table migrations). The routines live in ``public`` and are CREATE OR REPLACE, so
+running this migration once per tenant schema is idempotent.
+
+Revision ID: e5f6a7b8c9d0
+Revises: a7b8c9d0e1f2
+Create Date: 2026-06-05
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "e5f6a7b8c9d0"
+down_revision: str | Sequence[str] | None = "a7b8c9d0e1f2"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _is_base_schema_run() -> bool:
+    """True only for the base-schema migration (no per-tenant target_schema).
+
+    These routines live in the shared ``public`` schema, so they must be created
+    exactly once. Running ``CREATE OR REPLACE FUNCTION public....`` again from each
+    concurrent per-tenant migration aborts with ``tuple concurrently updated`` on
+    the ``pg_proc`` catalog row, so tenant runs skip it (the base run already
+    created the function for every tenant to use).
+    """
+    return not context.config.get_main_option("target_schema")
+
+
+def _pg_upgrade() -> None:
+    if not _is_base_schema_run():
+        return
+    # Banks with eligible-but-unscheduled facts and no in-flight consolidation.
+    # Auto-consolidation is filtered here only at the bank level (cheap prune);
+    # the full hierarchical resolution (global -> tenant -> bank, plus
+    # enable_observations) is done by the caller for the small returned set.
+    op.execute(
+        """
+        CREATE OR REPLACE FUNCTION public.banks_needing_consolidation()
+        RETURNS TABLE(schema_name text, bank_id text)
+        LANGUAGE plpgsql STABLE
+        AS $fn$
+        DECLARE
+            sch text;
+        BEGIN
+            FOR sch IN
+                SELECT n.nspname
+                FROM pg_class c
+                JOIN pg_namespace n ON n.oid = c.relnamespace
+                WHERE c.relname = 'memory_units' AND c.relkind = 'r'
+            LOOP
+                RETURN QUERY EXECUTE format($q$
+                    SELECT %1$L::text, m.bank_id
+                    FROM %1$I.memory_units m
+                    JOIN %1$I.banks b ON b.bank_id = m.bank_id
+                    WHERE m.consolidated_at IS NULL
+                      AND m.consolidation_failed_at IS NULL
+                      AND m.fact_type IN ('experience', 'world')
+                      AND COALESCE(b.config -> 'enable_auto_consolidation', 'true'::jsonb) <> 'false'::jsonb
+                      AND NOT EXISTS (
+                          SELECT 1 FROM %1$I.async_operations o
+                          WHERE o.bank_id = m.bank_id
+                            AND o.operation_type = 'consolidation'
+                            AND o.status IN ('pending', 'processing')
+                      )
+                    GROUP BY m.bank_id
+                $q$, sch);
+            END LOOP;
+        END;
+        $fn$;
+        """
+    )
+
+    # Schemas holding at least one row of p_table older than p_days. p_ts_col is
+    # the timestamp column to compare. Returns nothing when p_days <= 0
+    # (retention disabled).
+    op.execute(
+        """
+        CREATE OR REPLACE FUNCTION public.schemas_with_expired_rows(
+            p_table text, p_ts_col text, p_days int
+        )
+        RETURNS SETOF text
+        LANGUAGE plpgsql STABLE
+        AS $fn$
+        DECLARE
+            sch text;
+            has_expired boolean;
+        BEGIN
+            IF p_days IS NULL OR p_days <= 0 THEN
+                RETURN;
+            END IF;
+            FOR sch IN
+                SELECT n.nspname
+                FROM pg_class c
+                JOIN pg_namespace n ON n.oid = c.relnamespace
+                WHERE c.relname = p_table AND c.relkind = 'r'
+            LOOP
+                EXECUTE format(
+                    'SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I < NOW() - make_interval(days => $1))',
+                    sch, p_table, p_ts_col
+                ) INTO has_expired USING p_days;
+                IF has_expired THEN
+                    RETURN NEXT sch;
+                END IF;
+            END LOOP;
+        END;
+        $fn$;
+        """
+    )
+
+
+def _pg_downgrade() -> None:
+    if not _is_base_schema_run():
+        return
+    op.execute("DROP FUNCTION IF EXISTS public.banks_needing_consolidation()")
+    op.execute("DROP FUNCTION IF EXISTS public.schemas_with_expired_rows(text, text, int)")
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)
@@ -17,7 +17,13 @@ from typing import Any, Literal
 from fastapi import Depends, FastAPI, File, Form, Header, HTTPException, Query, Request, UploadFile
 from fastapi.middleware.gzip import GZipMiddleware

-from hindsight_api.engine.audit import AuditEntry, AuditLogger
+from hindsight_api.engine.audit import (
+    AuditEntry,
+    AuditLogger,
+    AuditLogListResponse,
+    AuditLogStatsResponse,
+)
+from hindsight_api.engine.llm_trace import LLMRequestListResponse, LLMRequestStatsResponse
 from hindsight_api.extensions import AuthenticationError


@@ -1444,6 +1450,18 @@ class ReprocessDocumentResponse(BaseModel):
    items_count: int


+class DocumentImportSubmitResponse(BaseModel):
+    """Response for the async document-import endpoint (202).
+
+    The import runs in the background; poll the operations endpoint for status.
+    The imported/skipped counts (documents_imported, facts_imported,
+    observations_imported, etc.) are written to the operation's result_metadata.
+    """
+
+    operation_id: str
+    status: str = "pending"
+
+
 class DeleteResponse(BaseModel):
    """Response model for delete operations."""

@@ -2150,6 +2168,28 @@ async def apply_bank_template_manifest(
    )


+class OperationProgress(BaseModel):
+    """Last-known progress snapshot for a long-running async operation.
+
+    Written at coarse phase/batch boundaries by the worker (consolidation, batch
+    retain). Lets an operator polling the operation status API distinguish a healthy
+    long-running job (``processed`` advancing across polls) from a frozen one (same
+    numbers, no movement in ``at``). Absent (``null``) on operations that never
+    reached a checkpoint — completed-instantly or pre-feature rows.
+    """
+
+    stage: str = Field(description="Coarse phase the operation last reported (e.g. 'processing_batch').")
+    at: str = Field(description="ISO-8601 timestamp when this snapshot was written.")
+    processed: int | None = Field(
+        default=None, description="Units of work finished so far (sub-batches, memories), when known."
+    )
+    total: int | None = Field(default=None, description="Total units of work for the operation, when known.")
+    detail: dict[str, int] | None = Field(
+        default=None,
+        description="Operation-specific counters (e.g. observations_created, round, items_in_sub_batch).",
+    )
+
+
 class OperationResponse(BaseModel):
    """Response model for a single async operation."""

@@ -2174,6 +2214,10 @@ class OperationResponse(BaseModel):
    items_count: int
    document_id: str | None = None
    created_at: str
+    updated_at: str | None = Field(
+        default=None,
+        description="When this operation's row last changed (claim, progress heartbeat, or completion).",
+    )
    status: str
    error_message: str | None
    retry_count: int | None = Field(
@@ -2190,6 +2234,10 @@ class OperationResponse(BaseModel):
            "some backpressure window opens. Always null for completed tasks."
        ),
    )
+    progress: OperationProgress | None = Field(
+        default=None,
+        description="Last-known progress snapshot for a running operation; null if none was recorded.",
+    )


 class ConsolidationRequest(BaseModel):
@@ -2325,6 +2373,10 @@ class OperationStatusResponse(BaseModel):
            "immediate pickup."
        ),
    )
+    progress: OperationProgress | None = Field(
+        default=None,
+        description="Last-known progress snapshot for a running operation; null if none was recorded.",
+    )
    result_metadata: dict[str, Any] | None = Field(
        default=None,
        description="Internal metadata for debugging. Structure may change without notice. Not for production use.",
@@ -2362,6 +2414,10 @@ class FeaturesInfo(BaseModel):
    worker: bool = Field(description="Whether the background worker is enabled")
    bank_config_api: bool = Field(description="Whether per-bank configuration API is enabled")
    file_upload_api: bool = Field(description="Whether file upload/conversion API is enabled")
+    document_export_api: bool = Field(description="Whether the document export endpoint is enabled")
+    document_import_api: bool = Field(description="Whether the document import endpoint is enabled")
+    audit_log: bool = Field(description="Whether audit logging is enabled")
+    llm_trace: bool = Field(description="Whether per-bank LLM request tracing is enabled")


 class VersionResponse(BaseModel):
@@ -2377,6 +2433,8 @@ class VersionResponse(BaseModel):
                    "worker": True,
                    "bank_config_api": False,
                    "file_upload_api": True,
+                    "document_export_api": True,
+                    "document_import_api": True,
                },
            }
        }
@@ -3023,6 +3081,10 @@ def _register_routes(app: FastAPI):
                worker=config.worker_enabled,
                bank_config_api=config.enable_bank_config_api,
                file_upload_api=config.enable_file_upload_api,
+                document_export_api=config.enable_document_export_api,
+                document_import_api=config.enable_document_import_api,
+                audit_log=config.audit_log_enabled,
+                llm_trace=config.llm_trace_enabled,
            ),
        )

@@ -5265,6 +5327,124 @@ def _register_routes(app: FastAPI):
            logger.error(f"Error in GET /v1/default/banks/{bank_id}/export: {error_detail}")
            raise HTTPException(status_code=500, detail=str(e))

+    # =====================================================================
+    # Document Transfer (Export / Import between banks — no LLM re-extraction)
+    # =====================================================================
+
+    @app.get(
+        # Dedicated path (not under /documents/) to avoid colliding with the
+        # greedy GET /documents/{document_id:path} route, which would otherwise
+        # capture "export"/"import" as a document id.
+        "/v1/default/banks/{bank_id}/document-transfer",
+        summary="Export documents",
+        description="Export documents (extracted facts, entity names, causal links, chunks) from a bank as a "
+        "transfer ZIP archive. Embeddings and database ids are not included — importing re-embeds with the target "
+        "bank's model and re-resolves entities. Consolidated observations are excluded unless include_observations=true. "
+        "Pass document_id query params to export specific documents, or omit to export the whole bank.",
+        operation_id="export_documents",
+        tags=["Document Transfer"],
+        responses={200: {"content": {"application/zip": {}}, "description": "Transfer archive"}},
+    )
+    async def api_export_documents(
+        bank_id: str,
+        document_id: list[str] | None = Query(default=None, description="Document id(s) to export; omit for all"),
+        include_observations: bool = Query(
+            default=False, description="Also export consolidated observations (restored on import)"
+        ),
+        request_context: RequestContext = Depends(get_request_context),
+    ):
+        """Export documents from a bank into a transfer ZIP archive."""
+        from fastapi.responses import Response
+
+        try:
+            if not get_config().enable_document_export_api:
+                raise HTTPException(
+                    status_code=404,
+                    detail="Document export API is disabled. "
+                    "Set HINDSIGHT_API_ENABLE_DOCUMENT_EXPORT_API=true to enable.",
+                )
+            profile = await app.state.memory.get_bank_profile(
+                bank_id, request_context=request_context, create_if_missing=False
+            )
+            if profile is None:
+                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
+
+            try:
+                archive = await app.state.memory.export_documents_async(
+                    bank_id,
+                    request_context,
+                    list(document_id) if document_id else None,
+                    include_observations=include_observations,
+                )
+            except ValueError as e:
+                # e.g. include_observations combined with a document_id subset.
+                raise HTTPException(status_code=400, detail=str(e))
+            return Response(
+                content=archive,
+                media_type="application/zip",
+                headers={"Content-Disposition": f'attachment; filename="{bank_id}-documents.zip"'},
+            )
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            logger.error(f"Error in GET /v1/default/banks/{bank_id}/document-transfer: {traceback.format_exc()}")
+            raise HTTPException(status_code=500, detail=str(e))
+
+    @app.post(
+        "/v1/default/banks/{bank_id}/document-transfer",
+        response_model=DocumentImportSubmitResponse,
+        status_code=202,
+        summary="Import documents (async)",
+        description="Submit a transfer archive (produced by the export endpoint) for import into a bank. Runs as a "
+        "background operation: facts are re-embedded with the target bank's embedding model and entities are "
+        "re-resolved — no LLM extraction. Returns an operation_id; poll "
+        "GET /v1/default/banks/{bank_id}/operations/{operation_id} for status and the imported/skipped counts in "
+        "result_metadata. Use on_conflict to control existing document ids: skip (default), replace, or new-id.",
+        operation_id="import_documents",
+        tags=["Document Transfer"],
+    )
+    @audited("import_documents", request_param=None)
+    async def api_import_documents(
+        bank_id: str,
+        file: UploadFile = File(..., description="Transfer ZIP archive"),
+        on_conflict: str = Query(default="skip", description="skip | replace | new-id"),
+        request_context: RequestContext = Depends(get_request_context),
+    ):
+        """Submit a transfer archive for async import into a bank."""
+        try:
+            if not get_config().enable_document_import_api:
+                raise HTTPException(
+                    status_code=404,
+                    detail="Document import API is disabled. "
+                    "Set HINDSIGHT_API_ENABLE_DOCUMENT_IMPORT_API=true to enable.",
+                )
+            if on_conflict not in ("skip", "replace", "new-id"):
+                raise HTTPException(
+                    status_code=400, detail=f"Invalid on_conflict '{on_conflict}' (expected skip|replace|new-id)"
+                )
+            archive_bytes = await file.read()
+            try:
+                submission = await app.state.memory.import_documents_async(
+                    bank_id, archive_bytes, request_context, on_conflict
+                )
+            except ValueError as e:
+                # Invalid archive / unsupported schema version — fail fast.
+                raise HTTPException(status_code=400, detail=str(e))
+            return DocumentImportSubmitResponse(operation_id=submission["operation_id"])
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            logger.error(f"Error in POST /v1/default/banks/{bank_id}/document-transfer: {traceback.format_exc()}")
+            raise HTTPException(status_code=500, detail=str(e))
+
    @app.get(
        "/v1/bank-template-schema",
        summary="Get bank template JSON Schema",
@@ -6202,48 +6382,13 @@ def _register_routes(app: FastAPI):
            raise HTTPException(status_code=500, detail=str(e))

    # ---- Audit Logs ----
+    # Response models live in engine/audit.py so the MemoryEngine read methods
+    # (list_audit_logs / audit_log_stats) can build and return them directly.

-    class AuditLogEntry(BaseModel):
-        """A single audit log entry."""
-
-        id: str
-        action: str
-        transport: str
-        bank_id: str | None
-        started_at: str | None
-        ended_at: str | None
-        duration_ms: int | None = Field(
-            default=None,
-            description="Server-computed duration in milliseconds (started_at → ended_at). Null if not yet completed.",
-        )
-        request: dict[str, Any] | None
-        response: dict[str, Any] | None
-        metadata: dict[str, Any]
-
-    class AuditLogListResponse(BaseModel):
-        """Response model for list audit logs endpoint."""
-
-        bank_id: str
-        total: int
-        limit: int
-        offset: int
-        items: list[AuditLogEntry]
-
-    class AuditLogStatsBucket(BaseModel):
-        """A single time bucket in audit log stats."""
-
-        time: str
-        actions: dict[str, int]
-        total: int
-
-    class AuditLogStatsResponse(BaseModel):
-        """Response model for audit log stats endpoint."""
-
-        bank_id: str
-        period: str
-        trunc: str
-        start: str
-        buckets: list[AuditLogStatsBucket]
+    # ---- LLM Request Traces ----
+    # Response models + queries live in the engine (engine/llm_trace.py and
+    # MemoryEngine.list_llm_requests / llm_request_stats). The handlers below
+    # only parse params, delegate to the engine, and map a missing bank to 404.

    @app.get(
        "/v1/default/banks/{bank_id}/audit-logs",
@@ -6265,120 +6410,19 @@ def _register_routes(app: FastAPI):
    ):
        """List audit log entries for a bank."""
        try:
-            from hindsight_api.engine.memory_engine import fq_table
-
-            pool = await app.state.memory._get_backend()
-
-            # Read endpoint: verify bank exists without auto-creating it.
-            if (
-                await app.state.memory.get_bank_profile(
-                    bank_id, request_context=request_context, create_if_missing=False
-                )
-                is None
-            ):
+            result = await app.state.memory.list_audit_logs(
+                bank_id,
+                request_context=request_context,
+                action=action,
+                transport=transport,
+                start_date=datetime.fromisoformat(start_date.replace("Z", "+00:00")) if start_date else None,
+                end_date=datetime.fromisoformat(end_date.replace("Z", "+00:00")) if end_date else None,
+                limit=limit,
+                offset=offset,
+            )
+            if result is None:
                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
-
-            from hindsight_api.engine.db_utils import acquire_with_retry
-
-            async with acquire_with_retry(pool) as conn:
-                where_clauses = ["bank_id = $1"]
-                params: list[Any] = [bank_id]
-                idx = 2
-
-                if action:
-                    where_clauses.append(f"action = ${idx}")
-                    params.append(action)
-                    idx += 1
-
-                if transport:
-                    where_clauses.append(f"transport = ${idx}")
-                    params.append(transport)
-                    idx += 1
-
-                if start_date:
-                    parsed_start = datetime.fromisoformat(start_date.replace("Z", "+00:00"))
-                    where_clauses.append(f"started_at >= ${idx}")
-                    params.append(parsed_start)
-                    idx += 1
-
-                if end_date:
-                    parsed_end = datetime.fromisoformat(end_date.replace("Z", "+00:00"))
-                    where_clauses.append(f"started_at < ${idx}")
-                    params.append(parsed_end)
-                    idx += 1
-
-                where_sql = " AND ".join(where_clauses)
-                table = fq_table("audit_log")
-
-                # Get total count
-                count_row = await conn.fetchrow(
-                    f"SELECT COUNT(*) as total FROM {table} WHERE {where_sql}",
-                    *params,
-                )
-                total = count_row["total"] if count_row else 0
-
-                # Get paginated results
-                params.append(limit)
-                params.append(offset)
-                rows = await conn.fetch(
-                    f"""
-                    SELECT id, action, transport, bank_id, started_at, ended_at,
-                           request, response, metadata
-                    FROM {table}
-                    WHERE {where_sql}
-                    ORDER BY started_at DESC
-                    LIMIT ${idx} OFFSET ${idx + 1}
-                    """,
-                    *params,
-                )
-
-                items = []
-                for row in rows:
-                    duration_ms = None
-                    started = row["started_at"]
-                    ended = row["ended_at"]
-                    if started and ended and hasattr(started, "total_seconds"):
-                        duration_ms = int((ended - started).total_seconds() * 1000)
-                    elif started and ended:
-                        try:
-                            duration_ms = int((ended - started).total_seconds() * 1000)
-                        except (TypeError, AttributeError):
-                            pass
-
-                    def _safe_iso(val):
-                        if val is None:
-                            return None
-                        return val.isoformat() if hasattr(val, "isoformat") else str(val)
-
-                    def _safe_json(val):
-                        if val is None:
-                            return None
-                        if isinstance(val, dict):
-                            return val
-                        return json.loads(val) if isinstance(val, str) else val
-
-                    items.append(
-                        {
-                            "id": str(row["id"]),
-                            "action": row["action"],
-                            "transport": row["transport"],
-                            "bank_id": row["bank_id"],
-                            "started_at": _safe_iso(started),
-                            "ended_at": _safe_iso(ended),
-                            "duration_ms": duration_ms,
-                            "request": _safe_json(row["request"]),
-                            "response": _safe_json(row["response"]),
-                            "metadata": _safe_json(row["metadata"]) or {},
-                        }
-                    )
-
-                return {
-                    "bank_id": bank_id,
-                    "total": total,
-                    "limit": limit,
-                    "offset": offset,
-                    "items": items,
-                }
+            return result
        except OperationValidationError as e:
            raise HTTPException(status_code=e.status_code, detail=e.reason)
        except (AuthenticationError, HTTPException):
@@ -6405,72 +6449,15 @@ def _register_routes(app: FastAPI):
    ):
        """Get audit log counts grouped by time bucket."""
        try:
-            from hindsight_api.engine.db_utils import acquire_with_retry
-            from hindsight_api.engine.memory_engine import fq_table
-
-            pool = await app.state.memory._get_backend()
-            # Read endpoint: verify bank exists without auto-creating it.
-            if (
-                await app.state.memory.get_bank_profile(
-                    bank_id, request_context=request_context, create_if_missing=False
-                )
-                is None
-            ):
+            result = await app.state.memory.audit_log_stats(
+                bank_id,
+                request_context=request_context,
+                action=action,
+                period=period,
+            )
+            if result is None:
                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
-
-            # Determine time range (always per-day buckets)
-            from datetime import timedelta as _td
-
-            now = datetime.now(timezone.utc)
-            trunc = "day"
-            if period == "1d":
-                start = now - _td(days=1)
-            elif period == "30d":
-                start = now - _td(days=30)
-            else:  # 7d default
-                start = now - _td(days=7)
-
-            table = fq_table("audit_log")
-
-            async with acquire_with_retry(pool) as conn:
-                where_clauses = ["bank_id = $1", "started_at >= $2"]
-                params: list[Any] = [bank_id, start]
-                idx = 3
-
-                if action:
-                    where_clauses.append(f"action = ${idx}")
-                    params.append(action)
-                    idx += 1
-
-                where_sql = " AND ".join(where_clauses)
-
-                rows = await conn.fetch(
-                    f"""
-                    SELECT date_trunc('{trunc}', started_at) AS bucket,
-                           action,
-                           COUNT(*) AS count
-                    FROM {table}
-                    WHERE {where_sql}
-                    GROUP BY bucket, action
-                    ORDER BY bucket ASC
-                    """,
-                    *params,
-                )
-
-                buckets: dict[str, dict[str, int]] = {}
-                for row in rows:
-                    bucket_key = row["bucket"].isoformat()
-                    if bucket_key not in buckets:
-                        buckets[bucket_key] = {}
-                    buckets[bucket_key][row["action"]] = row["count"]
-
-                return {
-                    "bank_id": bank_id,
-                    "period": period,
-                    "trunc": trunc,
-                    "start": start.isoformat(),
-                    "buckets": [{"time": k, "actions": v, "total": sum(v.values())} for k, v in buckets.items()],
-                }
+            return result
        except OperationValidationError as e:
            raise HTTPException(status_code=e.status_code, detail=e.reason)
        except (AuthenticationError, HTTPException):
@@ -6480,3 +6467,98 @@ def _register_routes(app: FastAPI):

            logger.error(f"Error getting audit log stats: {traceback.format_exc()}")
            raise HTTPException(status_code=500, detail=str(e))
+
+    @app.get(
+        "/v1/default/banks/{bank_id}/llm-requests",
+        summary="List LLM request traces",
+        description="List traced LLM requests for a bank, ordered by most recent first. "
+        "Requires LLM request tracing to be enabled (HINDSIGHT_API_LLM_TRACE_ENABLED).",
+        operation_id="list_llm_requests",
+        tags=["LLM Traces"],
+        response_model=LLMRequestListResponse,
+    )
+    async def api_list_llm_requests(
+        bank_id: str,
+        status: str | None = Query(None, description="Filter by status (success, error)"),
+        operation: str | None = Query(None, description="Filter by operation (retain, reflect, consolidation)"),
+        scope: str | None = Query(None, description="Filter by call scope"),
+        provider: str | None = Query(None, description="Filter by LLM provider"),
+        trace_id: str | None = Query(None, description="Filter to one operation run (all LLM calls sharing a trace)"),
+        document_id: str | None = Query(None, description="Filter to LLM calls that processed a given document"),
+        memory_id: str | None = Query(
+            None, description="Filter to the operation run(s) that produced or consumed a given memory_unit"
+        ),
+        group: bool = Query(
+            False, description="Paginate by operation run (trace) instead of by call; returns whole runs"
+        ),
+        start_date: str | None = Query(None, description="Filter from this ISO datetime (inclusive)"),
+        end_date: str | None = Query(None, description="Filter until this ISO datetime (exclusive)"),
+        limit: int = Query(50, ge=1, le=500, description="Max items to return"),
+        offset: int = Query(0, ge=0, description="Offset for pagination"),
+        request_context: RequestContext = Depends(get_request_context),
+    ):
+        """List traced LLM requests for a bank."""
+        try:
+            result = await app.state.memory.list_llm_requests(
+                bank_id,
+                request_context=request_context,
+                status=status,
+                operation=operation,
+                scope=scope,
+                provider=provider,
+                trace_id=trace_id,
+                document_id=document_id,
+                memory_id=memory_id,
+                group=group,
+                start_date=datetime.fromisoformat(start_date.replace("Z", "+00:00")) if start_date else None,
+                end_date=datetime.fromisoformat(end_date.replace("Z", "+00:00")) if end_date else None,
+                limit=limit,
+                offset=offset,
+            )
+            if result is None:
+                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
+            return result
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            logger.error(f"Error listing LLM requests: {traceback.format_exc()}")
+            raise HTTPException(status_code=500, detail=str(e))
+
+    @app.get(
+        "/v1/default/banks/{bank_id}/llm-requests/stats",
+        summary="LLM request statistics",
+        description="Get LLM request counts grouped by time bucket and status for charting.",
+        operation_id="llm_request_stats",
+        tags=["LLM Traces"],
+        response_model=LLMRequestStatsResponse,
+    )
+    async def api_llm_request_stats(
+        bank_id: str,
+        operation: str | None = Query(None, description="Filter by operation"),
+        period: str = Query("7d", description="Time period: 1d, 7d, or 30d"),
+        request_context: RequestContext = Depends(get_request_context),
+    ):
+        """Get LLM request counts grouped by time bucket and status."""
+        try:
+            result = await app.state.memory.llm_request_stats(
+                bank_id,
+                request_context=request_context,
+                operation=operation,
+                period=period,
+            )
+            if result is None:
+                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
+            return result
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            logger.error(f"Error getting LLM request stats: {traceback.format_exc()}")
+            raise HTTPException(status_code=500, detail=str(e))
@@ -143,6 +143,7 @@ ENV_LLM_GROQ_SERVICE_TIER = "HINDSIGHT_API_LLM_GROQ_SERVICE_TIER"
 ENV_LLM_OPENAI_SERVICE_TIER = "HINDSIGHT_API_LLM_OPENAI_SERVICE_TIER"
 ENV_LLM_EXTRA_BODY = "HINDSIGHT_API_LLM_EXTRA_BODY"
 ENV_LLM_DEFAULT_HEADERS = "HINDSIGHT_API_LLM_DEFAULT_HEADERS"
+ENV_LLM_STRICT_SCHEMA = "HINDSIGHT_API_LLM_STRICT_SCHEMA"

 # LiteLLM Router chain — provider-specific config consumed by the "litellmrouter"
 # provider. Each entry is a deployment; the Router tries them in declared order and
@@ -209,6 +210,17 @@ ENV_EMBEDDINGS_PROVIDER = "HINDSIGHT_API_EMBEDDINGS_PROVIDER"
 ENV_EMBEDDINGS_LOCAL_MODEL = "HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL"
 ENV_EMBEDDINGS_LOCAL_FORCE_CPU = "HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"
 ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE = "HINDSIGHT_API_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE"
+ENV_EMBEDDINGS_ONNX_MODEL_ID = "HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_ID"
+ENV_EMBEDDINGS_ONNX_MODEL_PATH = "HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH"
+ENV_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH = "HINDSIGHT_API_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH"
+ENV_EMBEDDINGS_ONNX_FILE = "HINDSIGHT_API_EMBEDDINGS_ONNX_FILE"
+ENV_EMBEDDINGS_ONNX_DIMENSIONS = "HINDSIGHT_API_EMBEDDINGS_ONNX_DIMENSIONS"
+ENV_EMBEDDINGS_ONNX_MAX_TOKENS = "HINDSIGHT_API_EMBEDDINGS_ONNX_MAX_TOKENS"
+ENV_EMBEDDINGS_ONNX_POOLING = "HINDSIGHT_API_EMBEDDINGS_ONNX_POOLING"
+ENV_EMBEDDINGS_ONNX_NORMALIZE = "HINDSIGHT_API_EMBEDDINGS_ONNX_NORMALIZE"
+ENV_EMBEDDINGS_ONNX_QUERY_PREFIX = "HINDSIGHT_API_EMBEDDINGS_ONNX_QUERY_PREFIX"
+ENV_EMBEDDINGS_ONNX_PASSAGE_PREFIX = "HINDSIGHT_API_EMBEDDINGS_ONNX_PASSAGE_PREFIX"
+ENV_EMBEDDINGS_ONNX_OUTPUT_NAME = "HINDSIGHT_API_EMBEDDINGS_ONNX_OUTPUT_NAME"
 ENV_EMBEDDINGS_TEI_URL = "HINDSIGHT_API_EMBEDDINGS_TEI_URL"
 ENV_EMBEDDINGS_OPENAI_API_KEY = "HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY"
 ENV_EMBEDDINGS_OPENAI_MODEL = "HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL"
@@ -297,6 +309,7 @@ ENV_RERANKER_LITELLM_TIMEOUT = "HINDSIGHT_API_RERANKER_LITELLM_TIMEOUT"
 ENV_RERANKER_LITELLM_SDK_TIMEOUT = "HINDSIGHT_API_RERANKER_LITELLM_SDK_TIMEOUT"
 ENV_RERANKER_GOOGLE_TIMEOUT = "HINDSIGHT_API_RERANKER_GOOGLE_TIMEOUT"
 ENV_RERANKER_MAX_CANDIDATES = "HINDSIGHT_API_RERANKER_MAX_CANDIDATES"
+ENV_SEMANTIC_MIN_SIMILARITY = "HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY"
 ENV_RERANKER_FLASHRANK_MODEL = "HINDSIGHT_API_RERANKER_FLASHRANK_MODEL"
 ENV_RERANKER_FLASHRANK_CACHE_DIR = "HINDSIGHT_API_RERANKER_FLASHRANK_CACHE_DIR"
 ENV_RERANKER_FLASHRANK_CPU_MEM_ARENA = "HINDSIGHT_API_RERANKER_FLASHRANK_CPU_MEM_ARENA"
@@ -346,6 +359,8 @@ ENV_RECALL_MAX_QUERY_TOKENS = "HINDSIGHT_API_RECALL_MAX_QUERY_TOKENS"
 ENV_MENTAL_MODEL_REFRESH_CONCURRENCY = "HINDSIGHT_API_MENTAL_MODEL_REFRESH_CONCURRENCY"
 ENV_LINK_EXPANSION_PER_ENTITY_LIMIT = "HINDSIGHT_API_LINK_EXPANSION_PER_ENTITY_LIMIT"
 ENV_LINK_EXPANSION_TIMEOUT = "HINDSIGHT_API_LINK_EXPANSION_TIMEOUT"
+ENV_BANK_STATS_CACHE_TTL_SECONDS = "HINDSIGHT_API_BANK_STATS_CACHE_TTL_SECONDS"
+ENV_BANK_STATS_CACHE_MAX_ENTRIES = "HINDSIGHT_API_BANK_STATS_CACHE_MAX_ENTRIES"

 # OpenTelemetry tracing configuration
 ENV_OTEL_TRACES_ENABLED = "HINDSIGHT_API_OTEL_TRACES_ENABLED"
@@ -363,6 +378,16 @@ ENV_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY = "HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOU
 # Gemini safety settings
 ENV_LLM_GEMINI_SAFETY_SETTINGS = "HINDSIGHT_API_LLM_GEMINI_SAFETY_SETTINGS"

+# Gemini prompt caching. When enabled, retain fact-extraction reuses a
+# CachedContent prefix for the static system_instruction + response_schema,
+# cutting per-call input cost on workloads with many small documents.
+# Provider-agnostic prompt-prefix caching. Providers that support it (currently
+# Gemini/Vertex via CachedContent) reuse the large, fixed, bank-agnostic system
+# prefix at the cached-input rate; providers that don't simply ignore it. On by
+# default — the prefix is bank-agnostic so a single cache is shared across all
+# banks, and creation soft-fails to an uncached call, so it never breaks a request.
+ENV_LLM_PROMPT_CACHE_ENABLED = "HINDSIGHT_API_LLM_PROMPT_CACHE_ENABLED"
+
 # Retain settings
 ENV_RETAIN_MAX_COMPLETION_TOKENS = "HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS"
 ENV_RETAIN_CHUNK_SIZE = "HINDSIGHT_API_RETAIN_CHUNK_SIZE"
@@ -400,12 +425,17 @@ ENV_FILE_CONVERSION_MAX_BATCH_SIZE = "HINDSIGHT_API_FILE_CONVERSION_MAX_BATCH_SI
 ENV_ENABLE_FILE_UPLOAD_API = "HINDSIGHT_API_ENABLE_FILE_UPLOAD_API"
 ENV_FILE_DELETE_AFTER_RETAIN = "HINDSIGHT_API_FILE_DELETE_AFTER_RETAIN"

+# Document transfer (export/import documents between banks without re-running the LLM)
+ENV_ENABLE_DOCUMENT_EXPORT_API = "HINDSIGHT_API_ENABLE_DOCUMENT_EXPORT_API"
+ENV_ENABLE_DOCUMENT_IMPORT_API = "HINDSIGHT_API_ENABLE_DOCUMENT_IMPORT_API"
+
 # Observations settings (consolidated knowledge from facts)
 ENV_ENABLE_OBSERVATIONS = "HINDSIGHT_API_ENABLE_OBSERVATIONS"
 ENV_ENABLE_AUTO_CONSOLIDATION = "HINDSIGHT_API_ENABLE_AUTO_CONSOLIDATION"
 ENV_CONSOLIDATION_BATCH_SIZE = "HINDSIGHT_API_CONSOLIDATION_BATCH_SIZE"
 ENV_CONSOLIDATION_MAX_MEMORIES_PER_ROUND = "HINDSIGHT_API_CONSOLIDATION_MAX_MEMORIES_PER_ROUND"
 ENV_CONSOLIDATION_LLM_BATCH_SIZE = "HINDSIGHT_API_CONSOLIDATION_LLM_BATCH_SIZE"
+ENV_CONSOLIDATION_DEDUP_THRESHOLD = "HINDSIGHT_API_CONSOLIDATION_DEDUP_THRESHOLD"
 ENV_CONSOLIDATION_LLM_PARALLELISM = "HINDSIGHT_API_CONSOLIDATION_LLM_PARALLELISM"
 ENV_CONSOLIDATION_MAX_TOKENS = "HINDSIGHT_API_CONSOLIDATION_MAX_TOKENS"
 ENV_CONSOLIDATION_SOURCE_FACTS_MAX_TOKENS = "HINDSIGHT_API_CONSOLIDATION_SOURCE_FACTS_MAX_TOKENS"
@@ -417,6 +447,7 @@ ENV_CONSOLIDATION_MAX_ATTEMPTS = "HINDSIGHT_API_CONSOLIDATION_MAX_ATTEMPTS"
 ENV_OBSERVATIONS_MISSION = "HINDSIGHT_API_OBSERVATIONS_MISSION"
 ENV_MAX_OBSERVATIONS_PER_SCOPE = "HINDSIGHT_API_MAX_OBSERVATIONS_PER_SCOPE"
 ENV_ENABLE_OBSERVATION_HISTORY = "HINDSIGHT_API_ENABLE_OBSERVATION_HISTORY"
+ENV_OBSERVATION_HISTORY_MAX_ENTRIES = "HINDSIGHT_API_OBSERVATION_HISTORY_MAX_ENTRIES"
 ENV_ENABLE_MENTAL_MODEL_HISTORY = "HINDSIGHT_API_ENABLE_MENTAL_MODEL_HISTORY"
 ENV_MENTAL_MODEL_HISTORY_MAX_ENTRIES = "HINDSIGHT_API_MENTAL_MODEL_HISTORY_MAX_ENTRIES"

@@ -448,6 +479,11 @@ ENV_DB_COMMAND_TIMEOUT = "HINDSIGHT_API_DB_COMMAND_TIMEOUT"
 ENV_DB_ACQUIRE_TIMEOUT = "HINDSIGHT_API_DB_ACQUIRE_TIMEOUT"
 ENV_DB_STATEMENT_TIMEOUT = "HINDSIGHT_API_DB_STATEMENT_TIMEOUT"

+# Wall-clock cap on model/connection initialization at startup. If embeddings,
+# cross-encoder, or LLM verification hang (e.g. an offline HuggingFace download
+# or an unreachable provider), the daemon fails fast instead of hanging forever.
+ENV_MODEL_INIT_TIMEOUT = "HINDSIGHT_API_MODEL_INIT_TIMEOUT"
+
 # Worker configuration (distributed task processing)
 ENV_WORKER_ENABLED = "HINDSIGHT_API_WORKER_ENABLED"
 ENV_WORKER_ID = "HINDSIGHT_API_WORKER_ID"
@@ -468,6 +504,7 @@ WORKER_SLOT_RESERVATION_TYPES: dict[str, tuple[str, int]] = {
    "file_convert_retain": ("HINDSIGHT_API_WORKER_FILE_CONVERT_RETAIN_MAX_SLOTS", 0),
    "refresh_mental_model": ("HINDSIGHT_API_WORKER_REFRESH_MENTAL_MODEL_MAX_SLOTS", 0),
    "graph_maintenance": ("HINDSIGHT_API_WORKER_GRAPH_MAINTENANCE_MAX_SLOTS", 0),
+    "import_documents": ("HINDSIGHT_API_WORKER_IMPORT_DOCUMENTS_MAX_SLOTS", 0),
 }
 ENV_WORKER_CONSOLIDATION_BANK_PRIORITY = "HINDSIGHT_API_WORKER_CONSOLIDATION_BANK_PRIORITY"
 ENV_RETAIN_MAX_CONCURRENT = "HINDSIGHT_API_RETAIN_MAX_CONCURRENT"
@@ -493,11 +530,32 @@ ENV_RECALL_BUDGET_ADAPTIVE_HIGH = "HINDSIGHT_API_RECALL_BUDGET_ADAPTIVE_HIGH"
 ENV_RECALL_BUDGET_MIN = "HINDSIGHT_API_RECALL_BUDGET_MIN"
 ENV_RECALL_BUDGET_MAX = "HINDSIGHT_API_RECALL_BUDGET_MAX"

+# Recall candidate gating (per-source cap + BM25 score floor)
+ENV_BM25_MIN_SCORE = "HINDSIGHT_API_BM25_MIN_SCORE"
+ENV_RECALL_MAX_CANDIDATES_PER_SOURCE = "HINDSIGHT_API_RECALL_MAX_CANDIDATES_PER_SOURCE"
+# Per-strategy recall boost. Prioritises specific retrieval arms (semantic,
+# bm25, graph, temporal) on recall via a human priority level — e.g.
+# "graph:high" to strongly favour graph hits, or "graph:high,semantic:low".
+# Valid levels: low | medium | high. The level (not a raw number) is the knob
+# because the boost is applied on two different score scales — see
+# engine/search/recall_boost.py for the level -> magnitude mapping and rationale.
+# Empty disables the feature.
+ENV_RECALL_STRATEGY_BOOSTS = "HINDSIGHT_API_RECALL_STRATEGY_BOOSTS"
+
 # Audit log settings
 ENV_AUDIT_LOG_ENABLED = "HINDSIGHT_API_AUDIT_LOG_ENABLED"
 ENV_AUDIT_LOG_ACTIONS = "HINDSIGHT_API_AUDIT_LOG_ACTIONS"
 ENV_AUDIT_LOG_RETENTION_DAYS = "HINDSIGHT_API_AUDIT_LOG_RETENTION_DAYS"

+# LLM request tracing settings
+ENV_LLM_TRACE_ENABLED = "HINDSIGHT_API_LLM_TRACE_ENABLED"
+ENV_LLM_TRACE_SCOPES = "HINDSIGHT_API_LLM_TRACE_SCOPES"
+ENV_LLM_TRACE_RETENTION_DAYS = "HINDSIGHT_API_LLM_TRACE_RETENTION_DAYS"
+ENV_LLM_TRACE_MAX_CHARS = "HINDSIGHT_API_LLM_TRACE_MAX_CHARS"
+
+# Background maintenance settings
+ENV_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS = "HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS"
+
 # Disposition settings
 ENV_DISPOSITION_SKEPTICISM = "HINDSIGHT_API_DISPOSITION_SKEPTICISM"
 ENV_DISPOSITION_LITERALISM = "HINDSIGHT_API_DISPOSITION_LITERALISM"
@@ -513,9 +571,9 @@ DEFAULT_LLM_PROVIDER = "openai"
 PROVIDER_DEFAULT_MODELS = {
    "openai": "gpt-4o-mini",
    "anthropic": "claude-haiku-4-5",
-    "gemini": "gemini-2.5-flash",
+    "gemini": "gemini-3.5-flash",
    "groq": "openai/gpt-oss-120b",
-    "minimax": "MiniMax-M2.7",
+    "minimax": "MiniMax-M3",
    "deepseek": "deepseek-v4-flash",
    "zai": "glm-4.5-flash",
    "opencode-go": "deepseek-v4-flash",
@@ -523,7 +581,7 @@ PROVIDER_DEFAULT_MODELS = {
    "ollama-cloud": "gemma3:12b",
    "llamacpp": "gemma-4-e2b-it",
    "lmstudio": "local-model",
-    "vertexai": "google/gemini-2.5-flash-lite",
+    "vertexai": "google/gemini-3.1-flash-lite",
    "openai-codex": "gpt-5.4-mini",
    "claude-code": "claude-sonnet-4-5-20250929",
    "mock": "mock-model",
@@ -542,6 +600,14 @@ DEFAULT_LLAMACPP_CHAT_FORMAT = None  # None = auto-detect from GGUF metadata
 DEFAULT_LLAMACPP_NO_GRAMMAR = False  # True = disable JSON grammar enforcement (faster but less reliable)
 DEFAULT_LLAMACPP_EXTRA_ARGS = None  # Space-separated extra CLI args for llama.cpp server

+# True = ask schema-capable backends to grammar-enforce structured output via
+# json_schema strict (OpenAI-compatible, LiteLLM; Gemini already enforces its
+# native response_schema). Default False keeps the soft "schema-in-prompt +
+# json_object" path, which weaker self-hosted instruction-followers can violate
+# (prose preambles, markdown fences, invalid JSON) — wedging retain/consolidation
+# on parse retries.
+DEFAULT_LLM_STRICT_SCHEMA = False
+
 DEFAULT_LLM_MAX_CONCURRENT = 32
 DEFAULT_LLM_MAX_RETRIES = 3  # Max retry attempts for LLM API calls
 DEFAULT_LLM_INITIAL_BACKOFF = 1.0  # Initial backoff in seconds for retry exponential backoff
@@ -561,6 +627,13 @@ DEFAULT_EMBEDDINGS_PROVIDER = "local"
 DEFAULT_EMBEDDINGS_LOCAL_MODEL = "BAAI/bge-small-en-v1.5"
 DEFAULT_EMBEDDINGS_LOCAL_FORCE_CPU = False  # Force CPU mode for local embeddings
 DEFAULT_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE = False  # Security: disabled by default, required for some models
+DEFAULT_EMBEDDINGS_ONNX_MODEL_ID = "intfloat/multilingual-e5-small"
+DEFAULT_EMBEDDINGS_ONNX_FILE = "onnx/model.onnx"
+DEFAULT_EMBEDDINGS_ONNX_MAX_TOKENS = 512
+DEFAULT_EMBEDDINGS_ONNX_POOLING = "mean"
+DEFAULT_EMBEDDINGS_ONNX_NORMALIZE = True
+DEFAULT_EMBEDDINGS_ONNX_QUERY_PREFIX = "query: "
+DEFAULT_EMBEDDINGS_ONNX_PASSAGE_PREFIX = "passage: "
 DEFAULT_EMBEDDINGS_OPENAI_MODEL = "text-embedding-3-small"
 DEFAULT_EMBEDDINGS_OPENAI_BATCH_SIZE = 100
 DEFAULT_EMBEDDINGS_GEMINI_MODEL = "gemini-embedding-001"
@@ -592,6 +665,65 @@ DEFAULT_RERANKER_LITELLM_TIMEOUT = 60.0
 DEFAULT_RERANKER_LITELLM_SDK_TIMEOUT = 60.0
 DEFAULT_RERANKER_GOOGLE_TIMEOUT = 60.0
 DEFAULT_RERANKER_MAX_CANDIDATES = 300
+DEFAULT_SEMANTIC_MIN_SIMILARITY = 0.3
+# Minimum BM25 score a row must exceed to enter fusion. 0.0 gates out
+# zero-score (non-matching) rows on backends — notably VectorChord — whose
+# operator ranks every document rather than pre-filtering to term matches.
+DEFAULT_BM25_MIN_SCORE = 0.0
+# Per-source candidate cap applied to each retrieval arm (semantic, BM25, graph,
+# temporal) before RRF, so a single over-expanding backend cannot fill the
+# reranker's global candidate budget on its own. 0 disables the cap.
+DEFAULT_RECALL_MAX_CANDIDATES_PER_SOURCE = 0
+# Per-strategy recall boost, as a comma-separated "strategy:level" list (e.g.
+# "graph:high,semantic:low"). Empty disables the feature. See
+# ENV_RECALL_STRATEGY_BOOSTS for the full rationale.
+DEFAULT_RECALL_STRATEGY_BOOSTS = ""
+# Retrieval arms that can be boosted; mirrors fusion.py source_names.
+RECALL_STRATEGY_NAMES = ("semantic", "bm25", "graph", "temporal")
+# User-facing priority levels. Kept in sync with recall_boost.BOOST_LEVELS by a
+# guard test; defined here (not imported) so config stays free of the heavy
+# engine.search import graph.
+RECALL_BOOST_LEVELS = ("low", "medium", "high")
+# Level applied when a strategy is listed without one (e.g. "graph" or "graph:").
+DEFAULT_RECALL_BOOST_LEVEL = "medium"
+
+
+def _parse_strategy_boosts(raw: str | None) -> dict[str, str]:
+    """Parse a "strategy:level,strategy:level" string into a boost map.
+
+    A strategy listed without a level (``"graph"`` or ``"graph:"``) defaults to
+    ``medium``. Only the strategies you list are boosted; any strategy you omit
+    keeps its normal, unboosted weight. Unknown strategy names, unknown levels,
+    and malformed entries are skipped with a warning so a typo degrades to a
+    no-op boost rather than breaking recall.
+    """
+    if not raw or not raw.strip():
+        return {}
+    boosts: dict[str, str] = {}
+    for entry in raw.split(","):
+        entry = entry.strip()
+        if not entry:
+            continue
+        name, _sep, level = entry.partition(":")
+        name = name.strip().lower()
+        level = level.strip().lower() or DEFAULT_RECALL_BOOST_LEVEL
+        if name not in RECALL_STRATEGY_NAMES:
+            logger.warning(
+                "Ignoring unknown recall strategy %r in boost (valid: %s)", name, ", ".join(RECALL_STRATEGY_NAMES)
+            )
+            continue
+        if level not in RECALL_BOOST_LEVELS:
+            logger.warning(
+                "Ignoring unknown recall boost level %r for %r (valid: %s)",
+                level,
+                name,
+                ", ".join(RECALL_BOOST_LEVELS),
+            )
+            continue
+        boosts[name] = level
+    return boosts
+
+
 DEFAULT_RERANKER_FLASHRANK_MODEL = "ms-marco-MiniLM-L-12-v2"  # Best balance of speed and quality
 DEFAULT_RERANKER_FLASHRANK_CACHE_DIR = None  # Use default cache directory
 DEFAULT_RERANKER_FLASHRANK_CPU_MEM_ARENA = False  # Disable ONNX CPU memory arena to bound RSS
@@ -667,6 +799,8 @@ DEFAULT_RECALL_MAX_QUERY_TOKENS = 500  # Maximum tokens allowed in recall query
 DEFAULT_MENTAL_MODEL_REFRESH_CONCURRENCY = 8  # Max concurrent mental model refreshes
 DEFAULT_LINK_EXPANSION_PER_ENTITY_LIMIT = 200  # Max target units per entity in graph expansion
 DEFAULT_LINK_EXPANSION_TIMEOUT = 10.0  # Timeout (seconds) for entity expansion query
+DEFAULT_BANK_STATS_CACHE_TTL_SECONDS = 60.0  # TTL for get_bank_stats result cache; 0 disables
+DEFAULT_BANK_STATS_CACHE_MAX_ENTRIES = 1024  # LRU bound across (schema, bank) keys

 # Retain settings
 DEFAULT_RETAIN_MAX_COMPLETION_TOKENS = 64000  # Max tokens for fact extraction LLM call
@@ -685,6 +819,7 @@ DEFAULT_RETAIN_BATCH_TOKENS = 10_000  # ~40KB of text  # Max chars per sub-batch
 DEFAULT_RETAIN_ENTITY_LOOKUP = "trigram"  # "full" or "trigram"
 DEFAULT_RETAIN_ENTITY_RESOLUTION_BATCH_SIZE = 100  # Unique entity names per pg_trgm candidate lookup query
 DEFAULT_RETAIN_BATCH_ENABLED = False  # Use LLM Batch API for fact extraction (only when async=True)
+DEFAULT_LLM_PROMPT_CACHE_ENABLED = True  # Reuse the fixed system prefix via provider prompt caching
 DEFAULT_RETAIN_BATCH_POLL_INTERVAL_SECONDS = 60  # Batch API polling interval in seconds

 # File storage defaults
@@ -696,23 +831,36 @@ DEFAULT_FILE_CONVERSION_MAX_BATCH_SIZE = 10  # Max files per batch upload
 DEFAULT_ENABLE_FILE_UPLOAD_API = True  # Enable file upload endpoint
 DEFAULT_FILE_DELETE_AFTER_RETAIN = True  # Delete file bytes after retain (saves storage)

+# Document transfer defaults (export/import enabled by default; gated independently)
+DEFAULT_ENABLE_DOCUMENT_EXPORT_API = True
+DEFAULT_ENABLE_DOCUMENT_IMPORT_API = True
+
 # Observations defaults (consolidated knowledge from facts)
 DEFAULT_ENABLE_OBSERVATIONS = True  # Observations enabled by default
 DEFAULT_ENABLE_AUTO_CONSOLIDATION = True  # Auto-consolidation after retain enabled by default
 DEFAULT_ENABLE_OBSERVATION_HISTORY = True  # Observation history tracking enabled by default
 DEFAULT_ENABLE_MENTAL_MODEL_HISTORY = True  # Mental model history tracking enabled by default
-# Each history entry snapshots previous_content + previous_reflect_response. Without
-# a cap, sustained mental-model refresh load grows the jsonb array unboundedly until
-# it crosses Postgres's hard 256MB jsonb limit and subsequent UPDATEs fail with
-# SQLSTATE 54000. 50 keeps the array well under 100MB even with large reflect
-# responses, while preserving enough recent history for meaningful audit / rollback.
+# History (mental-model refresh snapshots and observation update snapshots) lives in
+# the dedicated mental_model_history / observation_history tables, one row per change.
+# On every write we insert the new entry and delete the oldest rows beyond the cap,
+# so the per-item history can never grow unboundedly (the old single-JSONB-column
+# design hit Postgres's hard 256MB jsonb limit -> SQLSTATE 54000 and stuck rows).
+# 50 preserves enough recent history for meaningful audit / rollback per item.
+# A cap <= 0 removes the trim (unbounded growth) — to turn history OFF use the
+# enable_* flag, not a zero cap.
 DEFAULT_MENTAL_MODEL_HISTORY_MAX_ENTRIES = 50
+DEFAULT_OBSERVATION_HISTORY_MAX_ENTRIES = 50
 DEFAULT_CONSOLIDATION_MAX_ATTEMPTS = 3  # Outer retry attempts for consolidation LLM batch calls
 DEFAULT_CONSOLIDATION_BATCH_SIZE = 50  # Memories to load per batch (internal memory optimization)
 DEFAULT_CONSOLIDATION_MAX_MEMORIES_PER_ROUND = (
    100  # Max memories per consolidation round (0 = unlimited). Limits how long one bank holds a worker slot.
 )
 DEFAULT_CONSOLIDATION_LLM_BATCH_SIZE = 8  # Facts per LLM call (1 = no batching; >1 = batch mode)
+# Cosine >= this between a newly-created or freshly-updated observation and an existing one
+# triggers a focused 1-by-1 LLM "merge or keep" pass (the LLM reads both, so numbers/negation/
+# entities are respected). Enabled by default; set to 1.0 to disable. Postgres only — the merge
+# path uses Postgres-only SQL, so consolidation skips it on Oracle regardless of this value.
+DEFAULT_CONSOLIDATION_DEDUP_THRESHOLD = 0.97
 DEFAULT_CONSOLIDATION_LLM_PARALLELISM = (
    4  # Max tag groups consolidated concurrently per op. Locks on overlapping write
    # scopes degrade to sequential automatically; matches retain_max_concurrent.
@@ -737,6 +885,7 @@ DEFAULT_DB_POOL_MAX_SIZE = 100
 DEFAULT_DB_COMMAND_TIMEOUT = 60  # seconds
 DEFAULT_DB_ACQUIRE_TIMEOUT = 30  # seconds
 DEFAULT_DB_STATEMENT_TIMEOUT = 600  # seconds (Postgres statement_timeout applied on every pool connection; 0 disables)
+DEFAULT_MODEL_INIT_TIMEOUT = 300  # seconds (cap on startup model/connection init; covers first-time downloads)

 # Worker configuration (distributed task processing)
 DEFAULT_WORKER_ENABLED = True  # API runs worker by default (standalone mode)
@@ -789,6 +938,18 @@ DEFAULT_AUDIT_LOG_ENABLED = False  # Disabled by default
 DEFAULT_AUDIT_LOG_ACTIONS = ""  # Empty = audit all eligible actions
 DEFAULT_AUDIT_LOG_RETENTION_DAYS = -1  # -1 = keep forever

+# LLM request tracing defaults
+DEFAULT_LLM_TRACE_ENABLED = True  # Enabled by default
+DEFAULT_LLM_TRACE_SCOPES = ""  # Empty = trace all call scopes
+DEFAULT_LLM_TRACE_RETENTION_DAYS = 1  # Retain trace rows for 1 day by default
+DEFAULT_LLM_TRACE_MAX_CHARS = 50000  # Truncate stored input/output beyond this many chars
+
+# Background maintenance defaults
+# Periodic reconcile that re-schedules consolidation for banks with eligible-but-unscheduled
+# facts (e.g. after a consolidation operation failed terminally and left them unscheduled).
+# 0 disables the reconcile sweep.
+DEFAULT_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS = 300
+
 # Default MCP tool descriptions (can be customized via env vars)
 DEFAULT_MCP_RETAIN_DESCRIPTION = """Store important information to long-term memory.

@@ -1052,6 +1213,7 @@ class HindsightConfig:
    llm_default_headers: (
        dict | None
    )  # Custom headers passed as default_headers to provider SDK clients (e.g. {"X-Component-Id": "hindsight"} for proxies / request tracing)
+    llm_strict_schema: bool  # Grammar-enforce structured output via the provider's strongest schema mode (see DEFAULT_LLM_STRICT_SCHEMA)

    # LiteLLM Router chain (provider-specific; consumed by the "litellmrouter" provider).
    # List of deployment dicts evaluated in order with fallback on transient errors.
@@ -1067,6 +1229,10 @@ class HindsightConfig:
    # Gemini safety settings (None = use Gemini defaults; list of dicts with category/threshold)
    llm_gemini_safety_settings: list | None

+    # Gemini prompt caching toggle. When True, retain extraction reuses a
+    # CachedContent prefix for its system prompt + response schema.
+    llm_prompt_cache_enabled: bool
+
    # Built-in llama.cpp configuration (for provider=llamacpp)
    llamacpp_model_path: str | None  # Path to GGUF file (None = auto-download default)
    llamacpp_gpu_layers: int  # -1 = all layers on GPU, 0 = CPU only
@@ -1119,6 +1285,17 @@ class HindsightConfig:
    embeddings_local_model: str
    embeddings_local_force_cpu: bool
    embeddings_local_trust_remote_code: bool
+    embeddings_onnx_model_id: str
+    embeddings_onnx_model_path: str | None
+    embeddings_onnx_tokenizer_name_or_path: str | None
+    embeddings_onnx_file: str
+    embeddings_onnx_dimensions: int | None
+    embeddings_onnx_max_tokens: int
+    embeddings_onnx_pooling: str
+    embeddings_onnx_normalize: bool
+    embeddings_onnx_query_prefix: str
+    embeddings_onnx_passage_prefix: str
+    embeddings_onnx_output_name: str | None
    embeddings_tei_url: str | None
    embeddings_openai_base_url: str | None
    embeddings_cohere_api_key: str | None
@@ -1158,6 +1335,10 @@ class HindsightConfig:
    reranker_tei_max_concurrent: int
    reranker_tei_http_timeout: float
    reranker_max_candidates: int
+    semantic_min_similarity: float
+    bm25_min_score: float
+    recall_max_candidates_per_source: int
+    recall_strategy_boosts: dict[str, str]
    reranker_cohere_api_key: str | None
    reranker_cohere_model: str
    reranker_cohere_base_url: str | None
@@ -1213,6 +1394,8 @@ class HindsightConfig:
    mental_model_refresh_concurrency: int
    link_expansion_per_entity_limit: int
    link_expansion_timeout: float
+    bank_stats_cache_ttl_seconds: float
+    bank_stats_cache_max_entries: int

    # Retain settings
    retain_max_completion_tokens: int
@@ -1251,14 +1434,18 @@ class HindsightConfig:
    file_conversion_max_batch_size: int  # Max files per request
    enable_file_upload_api: bool
    file_delete_after_retain: bool
+    enable_document_export_api: bool
+    enable_document_import_api: bool

    # Observations settings (consolidated knowledge from facts)
    enable_observations: bool
    enable_auto_consolidation: bool
    enable_observation_history: bool
+    observation_history_max_entries: int
    enable_mental_model_history: bool
    mental_model_history_max_entries: int
    consolidation_batch_size: int
+    consolidation_dedup_threshold: float
    consolidation_max_memories_per_round: int
    consolidation_llm_batch_size: int
    consolidation_llm_parallelism: int
@@ -1318,6 +1505,7 @@ class HindsightConfig:
    db_command_timeout: int
    db_acquire_timeout: int
    db_statement_timeout: int
+    model_init_timeout: float

    # Worker configuration (distributed task processing)
    worker_enabled: bool
@@ -1349,6 +1537,17 @@ class HindsightConfig:
    audit_log_actions: list[str]  # Allowlist of action types (empty = all)
    audit_log_retention_days: int  # -1 = keep forever, >0 = delete after N days

+    # LLM request tracing configuration (static - server-level only)
+    llm_trace_enabled: bool  # Master switch for per-bank LLM request tracing
+    llm_trace_scopes: list[str]  # Allowlist of call scopes to trace (empty = all)
+    llm_trace_retention_days: int  # -1 = keep forever, >0 = delete after N days
+    llm_trace_max_chars: int  # Truncate stored input/output beyond this many chars
+
+    # Background maintenance configuration (static - server-level only)
+    # Interval for the periodic sweep that re-schedules consolidation for banks with
+    # eligible-but-unscheduled facts. 0 = disabled.
+    consolidation_reconcile_interval_seconds: int
+
    # Webhook configuration (static - server-level only, not per-bank)
    webhook_url: str | None  # Global webhook URL (None = disabled)
    webhook_secret: str | None  # HMAC signing secret (None = unsigned)
@@ -1549,6 +1748,11 @@ class HindsightConfig:
            self.text_search_extension_pg_search_tokenizer
        )

+        if not 0.0 <= self.semantic_min_similarity <= 1.0:
+            raise ValueError(
+                f"Invalid semantic_min_similarity: {self.semantic_min_similarity}. Must be between 0.0 and 1.0"
+            )
+
        # When LLM provider is "none", force chunks-only mode and disable LLM-dependent features
        if self.llm_provider == "none":
            self.retain_extraction_mode = "chunks"
@@ -1597,6 +1801,21 @@ class HindsightConfig:
                    " and ".join(missing),
                )

+        if self.embeddings_provider == "onnx":
+            try:
+                import importlib
+
+                importlib.import_module("onnxruntime")
+                importlib.import_module("transformers")
+            except ImportError:
+                logger.warning(
+                    "ONNX embeddings provider configured, but 'onnxruntime' and/or "
+                    "'transformers' is not installed. The API will fail at model init time. Either:\n"
+                    "  1. Install ONNX deps: pip install hindsight-api-slim[local-onnx]\n"
+                    "  2. Use a different embeddings provider, e.g. HINDSIGHT_API_EMBEDDINGS_PROVIDER=local "
+                    "or openai"
+                )
+
        # Validate that sum of per-operation slot reservations does not exceed max_slots
        total_reserved = sum(self.worker_slot_reservations.values())
        if total_reserved > self.worker_max_slots:
@@ -1649,6 +1868,7 @@ class HindsightConfig:
            llm_openai_service_tier=os.getenv(ENV_LLM_OPENAI_SERVICE_TIER, DEFAULT_LLM_OPENAI_SERVICE_TIER),
            llm_extra_body=json.loads(os.getenv(ENV_LLM_EXTRA_BODY, "null")),
            llm_default_headers=json.loads(os.getenv(ENV_LLM_DEFAULT_HEADERS, "null")),
+            llm_strict_schema=os.getenv(ENV_LLM_STRICT_SCHEMA, str(DEFAULT_LLM_STRICT_SCHEMA)).lower() in ("true", "1"),
            llm_litellmrouter_config=_parse_llm_router_config(ENV_LLM_LITELLMROUTER_CONFIG),
            # Vertex AI
            llm_vertexai_project_id=os.getenv(ENV_LLM_VERTEXAI_PROJECT_ID) or DEFAULT_LLM_VERTEXAI_PROJECT_ID,
@@ -1657,6 +1877,10 @@ class HindsightConfig:
            or DEFAULT_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY,
            # Gemini safety settings (JSON-encoded list of {category, threshold} dicts)
            llm_gemini_safety_settings=json.loads(os.getenv(ENV_LLM_GEMINI_SAFETY_SETTINGS, "null")),
+            llm_prompt_cache_enabled=os.getenv(
+                ENV_LLM_PROMPT_CACHE_ENABLED, str(DEFAULT_LLM_PROMPT_CACHE_ENABLED)
+            ).lower()
+            in ("1", "true", "yes", "on"),
            # Built-in llama.cpp configuration
            llamacpp_model_path=os.getenv(ENV_LLAMACPP_MODEL_PATH) or None,
            llamacpp_gpu_layers=int(os.getenv(ENV_LLAMACPP_GPU_LAYERS, str(DEFAULT_LLAMACPP_GPU_LAYERS))),
@@ -1755,6 +1979,36 @@ class HindsightConfig:
                ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE, str(DEFAULT_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE)
            ).lower()
            in ("true", "1"),
+            embeddings_onnx_model_id=os.getenv(ENV_EMBEDDINGS_ONNX_MODEL_ID, DEFAULT_EMBEDDINGS_ONNX_MODEL_ID),
+            embeddings_onnx_model_path=os.getenv(ENV_EMBEDDINGS_ONNX_MODEL_PATH) or None,
+            embeddings_onnx_tokenizer_name_or_path=os.getenv(ENV_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH) or None,
+            embeddings_onnx_file=os.getenv(ENV_EMBEDDINGS_ONNX_FILE, DEFAULT_EMBEDDINGS_ONNX_FILE),
+            embeddings_onnx_dimensions=_parse_optional_positive_int(
+                ENV_EMBEDDINGS_ONNX_DIMENSIONS,
+                os.getenv(ENV_EMBEDDINGS_ONNX_DIMENSIONS),
+            ),
+            embeddings_onnx_max_tokens=_parse_positive_int(
+                ENV_EMBEDDINGS_ONNX_MAX_TOKENS,
+                os.getenv(ENV_EMBEDDINGS_ONNX_MAX_TOKENS),
+                DEFAULT_EMBEDDINGS_ONNX_MAX_TOKENS,
+            ),
+            embeddings_onnx_pooling=_parse_optional_choice(
+                ENV_EMBEDDINGS_ONNX_POOLING,
+                os.getenv(ENV_EMBEDDINGS_ONNX_POOLING),
+                frozenset({"mean", "cls"}),
+            )
+            or DEFAULT_EMBEDDINGS_ONNX_POOLING,
+            embeddings_onnx_normalize=os.getenv(
+                ENV_EMBEDDINGS_ONNX_NORMALIZE, str(DEFAULT_EMBEDDINGS_ONNX_NORMALIZE)
+            ).lower()
+            in ("true", "1"),
+            embeddings_onnx_query_prefix=os.getenv(
+                ENV_EMBEDDINGS_ONNX_QUERY_PREFIX, DEFAULT_EMBEDDINGS_ONNX_QUERY_PREFIX
+            ),
+            embeddings_onnx_passage_prefix=os.getenv(
+                ENV_EMBEDDINGS_ONNX_PASSAGE_PREFIX, DEFAULT_EMBEDDINGS_ONNX_PASSAGE_PREFIX
+            ),
+            embeddings_onnx_output_name=os.getenv(ENV_EMBEDDINGS_ONNX_OUTPUT_NAME) or None,
            embeddings_tei_url=os.getenv(ENV_EMBEDDINGS_TEI_URL),
            embeddings_openai_base_url=os.getenv(ENV_EMBEDDINGS_OPENAI_BASE_URL) or None,
            embeddings_openai_batch_size=_parse_positive_int(
@@ -1876,6 +2130,14 @@ class HindsightConfig:
                os.getenv(ENV_RERANKER_TEI_HTTP_TIMEOUT, str(DEFAULT_RERANKER_TEI_HTTP_TIMEOUT))
            ),
            reranker_max_candidates=int(os.getenv(ENV_RERANKER_MAX_CANDIDATES, str(DEFAULT_RERANKER_MAX_CANDIDATES))),
+            semantic_min_similarity=float(os.getenv(ENV_SEMANTIC_MIN_SIMILARITY, str(DEFAULT_SEMANTIC_MIN_SIMILARITY))),
+            bm25_min_score=float(os.getenv(ENV_BM25_MIN_SCORE, str(DEFAULT_BM25_MIN_SCORE))),
+            recall_max_candidates_per_source=int(
+                os.getenv(ENV_RECALL_MAX_CANDIDATES_PER_SOURCE, str(DEFAULT_RECALL_MAX_CANDIDATES_PER_SOURCE))
+            ),
+            recall_strategy_boosts=_parse_strategy_boosts(
+                os.getenv(ENV_RECALL_STRATEGY_BOOSTS, DEFAULT_RECALL_STRATEGY_BOOSTS)
+            ),
            # Cohere reranker (with backward-compatible fallback to shared API key)
            reranker_cohere_api_key=os.getenv(ENV_RERANKER_COHERE_API_KEY) or os.getenv(ENV_COHERE_API_KEY),
            reranker_cohere_model=os.getenv(ENV_RERANKER_COHERE_MODEL, DEFAULT_RERANKER_COHERE_MODEL),
@@ -1965,6 +2227,12 @@ class HindsightConfig:
                os.getenv(ENV_LINK_EXPANSION_PER_ENTITY_LIMIT, str(DEFAULT_LINK_EXPANSION_PER_ENTITY_LIMIT))
            ),
            link_expansion_timeout=float(os.getenv(ENV_LINK_EXPANSION_TIMEOUT, str(DEFAULT_LINK_EXPANSION_TIMEOUT))),
+            bank_stats_cache_ttl_seconds=float(
+                os.getenv(ENV_BANK_STATS_CACHE_TTL_SECONDS, str(DEFAULT_BANK_STATS_CACHE_TTL_SECONDS))
+            ),
+            bank_stats_cache_max_entries=int(
+                os.getenv(ENV_BANK_STATS_CACHE_MAX_ENTRIES, str(DEFAULT_BANK_STATS_CACHE_MAX_ENTRIES))
+            ),
            # Optimization flags
            skip_llm_verification=os.getenv(ENV_SKIP_LLM_VERIFICATION, "false").lower() == "true",
            lazy_reranker=os.getenv(ENV_LAZY_RERANKER, "false").lower() == "true",
@@ -2028,6 +2296,14 @@ class HindsightConfig:
                ENV_FILE_DELETE_AFTER_RETAIN, str(DEFAULT_FILE_DELETE_AFTER_RETAIN)
            ).lower()
            == "true",
+            enable_document_export_api=os.getenv(
+                ENV_ENABLE_DOCUMENT_EXPORT_API, str(DEFAULT_ENABLE_DOCUMENT_EXPORT_API)
+            ).lower()
+            == "true",
+            enable_document_import_api=os.getenv(
+                ENV_ENABLE_DOCUMENT_IMPORT_API, str(DEFAULT_ENABLE_DOCUMENT_IMPORT_API)
+            ).lower()
+            == "true",
            # Observations settings (consolidated knowledge from facts)
            enable_observations=os.getenv(ENV_ENABLE_OBSERVATIONS, str(DEFAULT_ENABLE_OBSERVATIONS)).lower() == "true",
            enable_auto_consolidation=os.getenv(
@@ -2038,6 +2314,12 @@ class HindsightConfig:
                ENV_ENABLE_OBSERVATION_HISTORY, str(DEFAULT_ENABLE_OBSERVATION_HISTORY)
            ).lower()
            == "true",
+            observation_history_max_entries=int(
+                os.getenv(
+                    ENV_OBSERVATION_HISTORY_MAX_ENTRIES,
+                    str(DEFAULT_OBSERVATION_HISTORY_MAX_ENTRIES),
+                )
+            ),
            enable_mental_model_history=os.getenv(
                ENV_ENABLE_MENTAL_MODEL_HISTORY, str(DEFAULT_ENABLE_MENTAL_MODEL_HISTORY)
            ).lower()
@@ -2057,6 +2339,9 @@ class HindsightConfig:
                    str(DEFAULT_CONSOLIDATION_MAX_MEMORIES_PER_ROUND),
                )
            ),
+            consolidation_dedup_threshold=float(
+                os.getenv(ENV_CONSOLIDATION_DEDUP_THRESHOLD, str(DEFAULT_CONSOLIDATION_DEDUP_THRESHOLD))
+            ),
            consolidation_llm_batch_size=int(
                os.getenv(ENV_CONSOLIDATION_LLM_BATCH_SIZE, str(DEFAULT_CONSOLIDATION_LLM_BATCH_SIZE))
            ),
@@ -2099,6 +2384,7 @@ class HindsightConfig:
            db_command_timeout=int(os.getenv(ENV_DB_COMMAND_TIMEOUT, str(DEFAULT_DB_COMMAND_TIMEOUT))),
            db_acquire_timeout=int(os.getenv(ENV_DB_ACQUIRE_TIMEOUT, str(DEFAULT_DB_ACQUIRE_TIMEOUT))),
            db_statement_timeout=int(os.getenv(ENV_DB_STATEMENT_TIMEOUT, str(DEFAULT_DB_STATEMENT_TIMEOUT))),
+            model_init_timeout=float(os.getenv(ENV_MODEL_INIT_TIMEOUT, str(DEFAULT_MODEL_INIT_TIMEOUT))),
            # Worker configuration
            worker_enabled=os.getenv(ENV_WORKER_ENABLED, str(DEFAULT_WORKER_ENABLED)).lower() == "true",
            worker_id=os.getenv(ENV_WORKER_ID) or DEFAULT_WORKER_ID,
@@ -2183,6 +2469,22 @@ class HindsightConfig:
            audit_log_retention_days=int(
                os.getenv(ENV_AUDIT_LOG_RETENTION_DAYS, str(DEFAULT_AUDIT_LOG_RETENTION_DAYS))
            ),
+            # LLM request tracing configuration (static, server-level only)
+            llm_trace_enabled=os.getenv(ENV_LLM_TRACE_ENABLED, str(DEFAULT_LLM_TRACE_ENABLED)).lower() == "true",
+            llm_trace_scopes=[
+                s.strip() for s in os.getenv(ENV_LLM_TRACE_SCOPES, DEFAULT_LLM_TRACE_SCOPES).split(",") if s.strip()
+            ],
+            llm_trace_retention_days=int(
+                os.getenv(ENV_LLM_TRACE_RETENTION_DAYS, str(DEFAULT_LLM_TRACE_RETENTION_DAYS))
+            ),
+            llm_trace_max_chars=int(os.getenv(ENV_LLM_TRACE_MAX_CHARS, str(DEFAULT_LLM_TRACE_MAX_CHARS))),
+            # Background maintenance configuration (static, server-level only)
+            consolidation_reconcile_interval_seconds=int(
+                os.getenv(
+                    ENV_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS,
+                    str(DEFAULT_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS),
+                )
+            ),
            # Webhook configuration (static, server-level only)
            webhook_url=os.getenv(ENV_WEBHOOK_URL) or DEFAULT_WEBHOOK_URL,
            webhook_secret=os.getenv(ENV_WEBHOOK_SECRET) or DEFAULT_WEBHOOK_SECRET,
@@ -266,12 +266,20 @@ class ConfigResolver:
        # Validate recall budget fields
        _validate_recall_budget_updates(normalized_updates)

-        # Merge with existing config (JSONB || operator)
+        # Persist the override. Banks are created lazily (on first retain), so a
+        # PATCH that precedes any ingestion would otherwise UPDATE zero rows and
+        # silently no-op while returning 200. Ensure the bank row exists first
+        # (this also creates its per-bank vector indexes), then merge defensively:
+        # COALESCE guards against a NULL config column (NULL || jsonb is NULL),
+        # which would drop the override even when a row is updated.
+        from .engine.retain.fact_storage import ensure_bank_exists
+
        async with self._backend.acquire() as conn:
+            await ensure_bank_exists(conn, bank_id, ops=self._backend.ops)
            await conn.execute(
                f"""
                UPDATE {fq_table("banks")}
-                SET config = config || $1::jsonb,
+                SET config = COALESCE(config, '{{}}'::jsonb) || $1::jsonb,
                    updated_at = now()
                WHERE bank_id = $2
                """,
@@ -16,11 +16,59 @@ from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Any

+from pydantic import BaseModel, Field
+
 from ..engine.db_utils import acquire_with_retry

 logger = logging.getLogger(__name__)


+class AuditLogEntry(BaseModel):
+    """A single audit log entry."""
+
+    id: str
+    action: str
+    transport: str
+    bank_id: str | None
+    started_at: str | None
+    ended_at: str | None
+    duration_ms: int | None = Field(
+        default=None,
+        description="Server-computed duration in milliseconds (started_at → ended_at). Null if not yet completed.",
+    )
+    request: dict[str, Any] | None
+    response: dict[str, Any] | None
+    metadata: dict[str, Any]
+
+
+class AuditLogListResponse(BaseModel):
+    """Response model for list audit logs endpoint."""
+
+    bank_id: str
+    total: int
+    limit: int
+    offset: int
+    items: list[AuditLogEntry]
+
+
+class AuditLogStatsBucket(BaseModel):
+    """A single time bucket in audit log stats."""
+
+    time: str
+    actions: dict[str, int]
+    total: int
+
+
+class AuditLogStatsResponse(BaseModel):
+    """Response model for audit log stats endpoint."""
+
+    bank_id: str
+    period: str
+    trunc: str
+    start: str
+    buckets: list[AuditLogStatsBucket]
+
+
@dataclass
 class AuditEntry:
    """A single audit log entry."""
@@ -59,11 +107,11 @@ def _safe_json(data: Any) -> str | None:
        return None


-_SWEEP_INTERVAL_SECONDS = 3600  # Run retention sweep every hour
-
-
 class AuditLogger:
-    """Fire-and-forget audit log writer with optional retention sweep."""
+    """Fire-and-forget audit log writer.
+
+    Retention of old rows is handled by the background :class:`MaintenanceLoop`.
+    """

    def __init__(
        self,
@@ -71,14 +119,11 @@ class AuditLogger:
        schema_getter: Callable[[], str],
        enabled: bool,
        allowed_actions: list[str],
-        retention_days: int = -1,
    ) -> None:
        self._pool_getter = pool_getter
        self._schema_getter = schema_getter
        self._enabled = enabled
        self._allowed_actions: frozenset[str] | None = frozenset(allowed_actions) if allowed_actions else None
-        self._retention_days = retention_days
-        self._sweep_task: asyncio.Task | None = None

    def is_enabled(self, action: str) -> bool:
        """Check if audit logging is enabled for this action."""
@@ -128,48 +173,6 @@ class AuditLogger:
        except Exception as e:
            logger.warning(f"Audit log write failed for action={entry.action}: {e}")

-    def start_retention_sweep(self) -> None:
-        """Start the periodic retention sweep if retention is configured."""
-        if self._retention_days <= 0 or not self._enabled:
-            return
-        try:
-            self._sweep_task = asyncio.create_task(self._sweep_loop())
-        except RuntimeError:
-            logger.debug("Cannot start retention sweep: no running event loop")
-
-    async def stop_retention_sweep(self) -> None:
-        """Stop the periodic retention sweep."""
-        if self._sweep_task and not self._sweep_task.done():
-            self._sweep_task.cancel()
-            try:
-                await self._sweep_task
-            except asyncio.CancelledError:
-                pass
-            self._sweep_task = None
-
-    async def _sweep_loop(self) -> None:
-        """Periodically delete audit log entries older than retention_days."""
-        while True:
-            await self._run_sweep()
-            await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
-
-    async def _run_sweep(self) -> None:
-        """Delete expired audit log entries. Concurrent-safe via row-level deletes."""
-        pool = self._pool_getter()
-        if pool is None:
-            return
-        try:
-            schema = self._schema_getter()
-            table = f"{schema}.audit_log"
-            async with acquire_with_retry(pool, max_retries=1) as conn:
-                result = await conn.execute(
-                    f"DELETE FROM {table} WHERE started_at < NOW() - INTERVAL '{self._retention_days} days'"
-                )
-                if result and result != "DELETE 0":
-                    logger.info(f"Audit log retention sweep: {result}")
-        except Exception as e:
-            logger.warning(f"Audit log retention sweep failed: {e}")
-

@asynccontextmanager
 async def audit_context(
@@ -0,0 +1,122 @@
+"""TTL + coalescing cache for `get_bank_stats`.
+
+`get_bank_stats` aggregates over `memory_links` (and joins to `memory_units`),
+which can be a multi-second parallel sequential scan on banks with millions of
+rows. The result is intentionally approximate (it powers a UI widget and a
+freshness hint inside `reflect`), so caching it for a few tens of seconds is
+safe and dramatically reduces planner-driven thrash from clients that poll.
+
+The cache also coalesces concurrent misses on the same key onto a single
+in-flight task so that N concurrent callers produce one query rather than N.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import time
+from collections import OrderedDict
+from typing import Any, Awaitable, Callable
+
+
+class BankStatsCache:
+    """Per-process TTL cache keyed on (schema, bank_id).
+
+    `ttl_seconds <= 0` disables caching: each call passes straight through to
+    the loader. `max_entries` bounds memory in environments with many banks.
+    """
+
+    def __init__(self, *, ttl_seconds: float, max_entries: int) -> None:
+        self._ttl = float(ttl_seconds)
+        self._max_entries = int(max_entries) if max_entries and max_entries > 0 else 0
+        self._entries: OrderedDict[tuple[str, str], tuple[float, dict[str, Any]]] = OrderedDict()
+        self._in_flight: dict[tuple[str, str], asyncio.Future[dict[str, Any]]] = {}
+        self._lock = asyncio.Lock()
+
+    @property
+    def enabled(self) -> bool:
+        return self._ttl > 0
+
+    def _now(self) -> float:
+        return time.monotonic()
+
+    def _get_fresh_unlocked(self, key: tuple[str, str]) -> dict[str, Any] | None:
+        entry = self._entries.get(key)
+        if entry is None:
+            return None
+        expires_at, value = entry
+        if expires_at <= self._now():
+            # Expired — drop so the loader runs again.
+            self._entries.pop(key, None)
+            return None
+        # Mark as recently used for LRU eviction.
+        self._entries.move_to_end(key)
+        return value
+
+    def _store_unlocked(self, key: tuple[str, str], value: dict[str, Any]) -> None:
+        if not self.enabled:
+            return
+        self._entries[key] = (self._now() + self._ttl, value)
+        self._entries.move_to_end(key)
+        if self._max_entries:
+            while len(self._entries) > self._max_entries:
+                self._entries.popitem(last=False)
+
+    async def get_or_load(
+        self,
+        schema: str,
+        bank_id: str,
+        loader: Callable[[], Awaitable[dict[str, Any]]],
+    ) -> dict[str, Any]:
+        """Return cached stats for `(schema, bank_id)` or call `loader()`.
+
+        Concurrent misses on the same key are coalesced onto a single
+        in-flight loader.
+        """
+        if not self.enabled:
+            return await loader()
+
+        key = (schema, bank_id)
+
+        async with self._lock:
+            cached = self._get_fresh_unlocked(key)
+            if cached is not None:
+                return cached
+            in_flight = self._in_flight.get(key)
+            if in_flight is None:
+                in_flight = asyncio.get_running_loop().create_future()
+                self._in_flight[key] = in_flight
+                is_owner = True
+            else:
+                is_owner = False
+
+        if not is_owner:
+            return await asyncio.shield(in_flight)
+
+        try:
+            value = await loader()
+        except BaseException as exc:
+            async with self._lock:
+                self._in_flight.pop(key, None)
+            if not in_flight.done():
+                in_flight.set_exception(exc)
+            # Suppress "Future exception was never retrieved" when no other
+            # caller was waiting on this loader — we re-raise to the owner
+            # immediately and the future is a no-op in that case.
+            in_flight.exception()
+            raise
+
+        async with self._lock:
+            self._store_unlocked(key, value)
+            self._in_flight.pop(key, None)
+        if not in_flight.done():
+            in_flight.set_result(value)
+        return value
+
+    async def invalidate(self, schema: str, bank_id: str) -> None:
+        """Drop any cached stats for `(schema, bank_id)`."""
+        async with self._lock:
+            self._entries.pop((schema, bank_id), None)
+
+    async def clear(self) -> None:
+        async with self._lock:
+            self._entries.clear()
@@ -22,19 +22,30 @@ import time
 import uuid
 from collections import defaultdict
 from contextlib import AsyncExitStack
-from dataclasses import dataclass, field
+from dataclasses import asdict, dataclass, field
 from datetime import datetime, timezone
 from itertools import combinations
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING, Any, Literal

 from pydantic import BaseModel, field_validator

 from ...config import get_config
+from ...worker.stage import set_stage
 from ..db_utils import acquire_with_retry
+from ..llm_trace import (
+    record_created_memory_ids,
+    record_source_memory_ids,
+    reset_trace_context,
+    set_trace_context,
+    trace_context_of,
+)
 from ..llm_wrapper import sanitize_llm_output
 from ..memory_engine import Budget, fq_table
 from ..retain import embedding_utils
-from .prompts import build_batch_consolidation_prompt
+from .prompts import (
+    build_consolidation_input,
+    build_consolidation_system_prompt,
+)

 if TYPE_CHECKING:
    from asyncpg import Connection
@@ -46,6 +57,254 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)


+def _norm_obs_text(text: str) -> str:
+    """Whitespace-normalised observation text for exact-duplicate matching.
+
+    Collapses runs of whitespace only; case is preserved. The reconciliation guard
+    drops a CREATE on the premise that an exact-text match loses no information — but
+    case-folding would also drop a create differing only in case (e.g. "TLS" vs "tls"),
+    which *does* lose information, so we match case-sensitively.
+    """
+    return " ".join((text or "").split()).strip()
+
+
+def _duplicate_create_target(
+    create_text: str,
+    shown_obs_by_text: "dict[str, MemoryFact]",
+    update_texts: set[str],
+) -> str | None:
+    """Return a human label for what ``create_text`` duplicates, or None if novel.
+
+    A CREATE is a duplicate when its normalised text matches an observation that was
+    already shown to the LLM, or the text of an UPDATE issued in the same response
+    (the model occasionally UPDATEs the twin to text X and also CREATEs X). Exact-text
+    match means no information is lost by dropping the CREATE.
+    """
+    norm = _norm_obs_text(create_text)
+    matched = shown_obs_by_text.get(norm)
+    if matched is not None:
+        return f"shown observation {str(matched.id)[:8]}"
+    if norm in update_texts:
+        return "an UPDATE in this response"
+    return None
+
+
+# Top-K existing observations probed (by the new observation's own embedding) when
+# semantic dedup is enabled. Small: we only need the nearest few candidates.
+_DEDUP_TOP_K = 5
+
+
+class _DedupDecision(BaseModel):
+    """Focused 1-by-1 verdict for whether a new observation duplicates an existing one."""
+
+    action: Literal["merge", "keep"]
+    text: str = ""  # the synthesized merged observation (when action == "merge")
+    reason: str = ""
+
+
+_DEDUP_PROMPT = """You reconcile long-term memory observations. A NEW observation is about to be \
+stored, and it is highly similar to an EXISTING one:
+
+[NEW] {new}
+[EXISTING] {existing}
+
+If they assert the SAME fact (wording aside), respond action="merge" and provide `text`: a single \
+observation that preserves EVERY detail from both. If they differ in ANY important detail — a \
+number/quantity, a named entity or language, a negation, or a condition — respond action="keep"."""
+
+
+def _dedup_active(config: Any) -> bool:
+    """Whether create/update semantic dedup runs for this consolidation.
+
+    Enabled when the resolved threshold is < 1.0, EXCEPT on Oracle: the merge path uses
+    Postgres-only SQL (``unnest``/``array_agg``, ``UPDATE ... FROM``), so on Oracle dedup is
+    skipped — it behaves exactly as it did before this feature, regardless of the configured
+    threshold. This is why the feature can ship enabled-by-default without breaking Oracle.
+    """
+    if config is None or getattr(config, "consolidation_dedup_threshold", 1.0) >= 1.0:
+        return False
+    return get_config().database_backend != "oracle"
+
+
+@dataclass
+class _DedupOutcome:
+    """Result of probing one observation against its in-scope neighbours.
+
+    ``best_id`` is the nearest observation at/above the threshold (None if none),
+    ``merged_text`` is the LLM-synthesized union text (set only when ``should_merge``).
+    """
+
+    best_id: str | None
+    merged_text: str
+    should_merge: bool
+
+
+async def _dedup_adjudicate(
+    conn: "Connection",
+    memory_engine: "MemoryEngine",
+    bank_id: str,
+    config: Any,
+    dedup_llm_config: Any,
+    anchor_text: str,
+    anchor_emb_str: str | None,
+    tags: list[str] | None,
+    exclude_id: str | None,
+) -> _DedupOutcome:
+    """Probe one observation's embedding against in-scope observations and adjudicate a merge.
+
+    Anchored on the observation text — the correct obs<->obs comparison, unlike consolidation
+    recall which is anchored on the raw fact. Returns the nearest observation at/above
+    ``consolidation_dedup_threshold`` and, when found, the LLM's focused 1-by-1 merge-or-keep
+    verdict (scope ``consolidation_dedup``): the LLM reads both texts, so a word-level difference
+    (number / negation / entity) is respected. ``exclude_id`` skips the anchor observation itself
+    (used by the UPDATE path, where the anchor row already exists and would self-match at 1.0).
+    ``anchor_emb_str`` reuses an already-computed embedding (the UPDATE path just embedded it);
+    pass None to embed ``anchor_text`` here (the CREATE path).
+    """
+    from ..search.retrieval import retrieve_semantic_bm25_combined
+
+    threshold = config.consolidation_dedup_threshold
+    if anchor_emb_str is None:
+        embs = await embedding_utils.generate_embeddings_batch(memory_engine.embeddings, [anchor_text])
+        if not embs:
+            return _DedupOutcome(best_id=None, merged_text="", should_merge=False)
+        anchor_emb_str = str(embs[0])
+    tags_match = "all_strict" if tags else "any"
+    grouped = await retrieve_semantic_bm25_combined(
+        conn, anchor_emb_str, anchor_text, bank_id, ["observation"], _DEDUP_TOP_K, tags=tags, tags_match=tags_match
+    )
+    results = grouped.get("observation", ([], []))[0]
+    best_id: str | None = None
+    best_text = ""
+    best_sim = threshold  # only candidates at/above the threshold are considered
+    for r in results:
+        rid = str(r.id)
+        if exclude_id is not None and rid == exclude_id:
+            continue  # never match the anchor observation against itself
+        sim = r.similarity or 0.0
+        if sim >= best_sim:
+            best_id, best_text, best_sim = rid, r.text, sim
+
+    if best_id is None:
+        return _DedupOutcome(best_id=None, merged_text="", should_merge=False)
+
+    decision: _DedupDecision = await dedup_llm_config.call(
+        messages=[{"role": "user", "content": _DEDUP_PROMPT.format(new=anchor_text, existing=best_text)}],
+        response_format=_DedupDecision,
+        scope="consolidation_dedup",
+    )
+    if decision.action != "merge":
+        return _DedupOutcome(best_id=best_id, merged_text="", should_merge=False)
+    return _DedupOutcome(best_id=best_id, merged_text=decision.text.strip() or best_text, should_merge=True)
+
+
+async def _dedup_reconcile_create(
+    conn: "Connection",
+    memory_engine: "MemoryEngine",
+    bank_id: str,
+    config: Any,
+    dedup_llm_config: Any,
+    create_text: str,
+    create_source_ids: list[uuid.UUID],
+    tags: list[str] | None,
+) -> str | None:
+    """Semantic dedup for a single CREATE (create-time, focused 1-by-1).
+
+    On "merge", folds the new source facts + the synthesized text into the existing
+    observation and returns its id (caller skips the CREATE). Returns None when there is
+    no near twin or the LLM keeps them distinct.
+    """
+    outcome = await _dedup_adjudicate(
+        conn, memory_engine, bank_id, config, dedup_llm_config, create_text, None, tags, exclude_id=None
+    )
+    if not outcome.should_merge or outcome.best_id is None:
+        return None
+
+    # Fold the new source facts into the twin and persist the merged text. We keep the twin's
+    # existing embedding: the merged text is >= threshold similar, so the stored vector stays
+    # representative and we avoid a re-embed + a dialect-specific vector UPDATE.
+    await conn.execute(
+        f"""
+        UPDATE {fq_table("memory_units")}
+        SET text = $1,
+            source_memory_ids = (SELECT array_agg(DISTINCT e) FROM unnest(source_memory_ids || $2::uuid[]) e),
+            proof_count = (SELECT count(DISTINCT e) FROM unnest(source_memory_ids || $2::uuid[]) e),
+            updated_at = now()
+        WHERE id = $3::uuid
+        """,
+        outcome.merged_text,
+        create_source_ids,
+        uuid.UUID(outcome.best_id),
+    )
+    return outcome.best_id
+
+
+async def _dedup_reconcile_update(
+    conn: "Connection",
+    memory_engine: "MemoryEngine",
+    bank_id: str,
+    config: Any,
+    dedup_llm_config: Any,
+    updated_id: str,
+    updated_text: str,
+    updated_emb_str: str | None,
+    tags: list[str] | None,
+) -> None:
+    """Semantic dedup for an UPDATE (after the observation was rewritten + re-embedded).
+
+    An UPDATE rewrites an observation's text and re-embeds it, so its vector can drift to
+    within threshold of a DIFFERENT existing observation. The create-time guard never sees
+    this (it only runs on CREATE), so without this the two persist as a near-duplicate pair —
+    the measured residual-duplicate source. Probe the updated observation's new embedding
+    against the others (excluding itself); on "merge", fold the just-updated observation's
+    sources into the twin, persist the merged text, and DELETE the updated row. Unlike the
+    CREATE path the row already exists, so reconciliation is a fold-and-delete, not a skip.
+    """
+    outcome = await _dedup_adjudicate(
+        conn,
+        memory_engine,
+        bank_id,
+        config,
+        dedup_llm_config,
+        updated_text,
+        updated_emb_str,
+        tags,
+        exclude_id=updated_id,
+    )
+    if not outcome.should_merge or outcome.best_id is None:
+        return
+
+    # Fold the updated observation's sources into the twin (keeping the twin's embedding, as in
+    # the create path) then delete the now-redundant updated row. The all_strict/any tag match
+    # guarantees twin and updated share scope, so dropping the updated row's tags loses no
+    # visibility. Temporal fields follow the surviving twin (minimal scope; matches create).
+    await conn.execute(
+        f"""
+        UPDATE {fq_table("memory_units")} t
+        SET text = $1,
+            source_memory_ids = (
+                SELECT array_agg(DISTINCT e) FROM unnest(t.source_memory_ids || u.source_memory_ids) e
+            ),
+            proof_count = (
+                SELECT count(DISTINCT e) FROM unnest(t.source_memory_ids || u.source_memory_ids) e
+            ),
+            updated_at = now()
+        FROM {fq_table("memory_units")} u
+        WHERE t.id = $2::uuid AND u.id = $3::uuid
+        """,
+        outcome.merged_text,
+        uuid.UUID(outcome.best_id),
+        uuid.UUID(updated_id),
+    )
+    await _execute_delete_action(conn, bank_id, updated_id)
+    logger.info(
+        "[CONSOLIDATION] dedup-merged updated observation %s into %s (cosine>=%.2f)",
+        updated_id[:8],
+        outcome.best_id[:8],
+        config.consolidation_dedup_threshold,
+    )
+
+
@dataclass
 class _BatchDeltas:
    """Per-LLM-batch deltas, merged into the job's running stats after dispatch.
@@ -167,6 +426,9 @@ async def _filter_live_source_memories(
 class _CreateAction(BaseModel):
    text: str
    source_fact_ids: list[str]  # memory UUIDs from the NEW FACTS list
+    # One-sentence justification from the LLM (why CREATE vs UPDATE). Diagnostic
+    # only — surfaced in the consolidation trace to explain duplicate creates.
+    reason: str = ""

    @field_validator("text", mode="before")
    @classmethod
@@ -178,6 +440,7 @@ class _UpdateAction(BaseModel):
    text: str
    observation_id: str  # UUID of the existing observation to update
    source_fact_ids: list[str]  # memory UUIDs from the NEW FACTS list
+    reason: str = ""  # LLM's one-sentence justification (diagnostic only)

    @field_validator("text", mode="before")
    @classmethod
@@ -187,6 +450,7 @@ class _UpdateAction(BaseModel):

 class _DeleteAction(BaseModel):
    observation_id: str  # UUID of the observation to remove
+    reason: str = ""  # LLM's one-sentence justification (diagnostic only)


 class _ConsolidationBatchResponse(BaseModel):
@@ -361,8 +625,36 @@ async def run_consolidation_job(

    # Build a configured LLM wrapper that applies per-bank settings (e.g. safety settings)
    # to every call without leaking across operations.
-    llm_config = memory_engine._consolidation_llm_config.with_config(config)
+    llm_config = memory_engine._consolidation_llm_config.with_config(config, bank_id=bank_id, operation="consolidation")

+    # Bind the operation trace context for the whole run so the create/update DB
+    # sites (deep inside _process_memory_batch) can accumulate the observations
+    # this consolidation produced and the source memories it consumed onto the
+    # trace — flushed onto every trace row on exit by attach_memory_ids.
+    trace_ctx = trace_context_of(llm_config)
+    trace_token = set_trace_context(trace_ctx) if trace_ctx is not None else None
+    try:
+        return await _run_consolidation_job(
+            memory_engine, bank_id, request_context, config, llm_config, operation_id, observation_scopes
+        )
+    finally:
+        if trace_token is not None:
+            reset_trace_context(trace_token)
+            # Fire-and-forget: patched on a background task, off the consolidation
+            # critical path.
+            memory_engine._llm_recorder.attach_memory_ids(trace_ctx)
+
+
+async def _run_consolidation_job(
+    memory_engine: "MemoryEngine",
+    bank_id: str,
+    request_context: "RequestContext",
+    config: Any,
+    llm_config: Any,
+    operation_id: str | None = None,
+    observation_scopes: list[list[str]] | None = None,
+) -> dict[str, Any]:
+    """Core consolidation flow. See ``run_consolidation_job`` for the public entrypoint."""
    perf = ConsolidationPerfLog(bank_id)
    max_memories_per_batch = config.consolidation_batch_size
    max_memories_per_round = config.consolidation_max_memories_per_round
@@ -426,6 +718,44 @@ async def run_consolidation_job(
    logger.info(f"[CONSOLIDATION] bank={bank_id} total_unconsolidated={total_count}")
    perf.log(f"[1] Found {total_count} pending memories to consolidate")

+    # Initial durable progress snapshot so an operator polling the operation status
+    # API sees the job has started and how much work it found, before the first batch
+    # of LLM work completes (which can take minutes on a dense bank). Uses the same
+    # "consolidating" stage as the per-batch heartbeat so the operator sees a single
+    # phase advancing 0/N -> N/N rather than an opaque "scanning" -> "processing" hop.
+    set_stage("consolidation.consolidating")
+    await memory_engine._write_operation_progress(operation_id, stage="consolidating", processed=0, total=total_count)
+
+    async def _count_unconsolidated() -> int:
+        """Re-count memories still pending consolidation in this job's scope.
+
+        ``total_count`` is a point-in-time estimate from job start; memories retained
+        while consolidation runs get picked up by later fetches, so processed can pass
+        it. When that happens we re-count to report a real total (processed + remaining)
+        instead of pinning the bar at 100%."""
+        async with acquire_with_retry(pool) as count_conn:
+            pending = await count_conn.fetchval(
+                f"""
+                SELECT COUNT(*)
+                FROM {fq_table("memory_units")}
+                WHERE bank_id = $1
+                  AND consolidated_at IS NULL
+                  AND consolidation_failed_at IS NULL
+                  AND fact_type IN ('experience', 'world')
+                  {scope_clause}
+                """,
+                *scope_params,
+            )
+        return pending or 0
+
+    async def _progress_total(processed: int) -> int:
+        # Cheap path: while we're still within the start-of-job estimate it's exact, so
+        # no extra query. Only re-count once the estimate is exhausted (≈the final batch
+        # normally, or repeatedly only if memories keep arriving mid-run).
+        if processed < total_count:
+            return total_count
+        return processed + await _count_unconsolidated()
+
    # Process each memory with individual commits for crash recovery
    stats: dict[str, int] = {
        "memories_processed": 0,
@@ -446,10 +776,18 @@ async def run_consolidation_job(
    hit_round_limit = False

    llm_batch_num = 0
-    # Cumulative count of memories processed across the whole job, shared by
-    # the per-batch log so it can still report processed/total under parallelism.
-    # Mutable container so the inner closure can update without a `nonlocal`.
-    cumulative_progress = {"processed": 0}
+    # Cumulative counters across the whole job, shared by the per-batch log and the
+    # durable progress snapshot so both report processed/total (and observation
+    # tallies) under parallelism. Mutable container so the inner closure can update
+    # without a `nonlocal`.
+    cumulative_progress = {
+        "processed": 0,
+        "observations_created": 0,
+        "observations_updated": 0,
+        "observations_merged": 0,
+        "observations_deleted": 0,
+        "memories_failed": 0,
+    }
    while True:
        # Cap fetch size by remaining round budget
        fetch_limit = (
@@ -670,12 +1008,18 @@ async def run_consolidation_job(
                    local_stats["memories_failed"] += 1

            # Maintain the cumulative-progress indicator under parallelism:
-            # increment a shared counter and snapshot under the same statement
-            # so the snapshot includes this batch. No await between the read
-            # and write, so single-threaded asyncio gives us atomicity for free
-            # — no lock needed.
+            # increment shared counters and snapshot under the same statements so
+            # the snapshot includes this batch. No await between the reads and
+            # writes, so single-threaded asyncio gives us atomicity for free —
+            # no lock needed.
            cumulative_progress["processed"] += local_stats["memories_processed"]
+            cumulative_progress["observations_created"] += local_stats["observations_created"]
+            cumulative_progress["observations_updated"] += local_stats["observations_updated"]
+            cumulative_progress["observations_merged"] += local_stats["observations_merged"]
+            cumulative_progress["observations_deleted"] += local_stats["observations_deleted"]
+            cumulative_progress["memories_failed"] += local_stats["memories_failed"]
            cum_processed = cumulative_progress["processed"]
+            cum_snapshot = dict(cumulative_progress)

            # Per-batch log uses batch_perf so timings/llm-calls/tokens reflect
            # only this batch's own work, even when other batches are running
@@ -703,6 +1047,27 @@ async def run_consolidation_job(
                f" | avg={llm_batch_time / max(1, len(llm_batch_local)):.3f}s/memory"
            )

+            # Durable progress snapshot per LLM batch — this is the heartbeat an
+            # operator polls. The whole fetched batch is processed inside one outer
+            # round, so a round-boundary write would sit at the pre-round count for
+            # the entire (often minutes-long) LLM phase; writing here advances
+            # processed/total as each batch commits. set_stage mirrors it for the
+            # live worker log.
+            set_stage(f"consolidation.llm_batch.{batch_num_local}")
+            await memory_engine._write_operation_progress(
+                operation_id,
+                stage="consolidating",
+                processed=cum_processed,
+                total=await _progress_total(cum_processed),
+                detail={
+                    "observations_created": cum_snapshot["observations_created"],
+                    "observations_updated": cum_snapshot["observations_updated"],
+                    "observations_merged": cum_snapshot["observations_merged"],
+                    "observations_deleted": cum_snapshot["observations_deleted"],
+                    "memories_failed": cum_snapshot["memories_failed"],
+                },
+            )
+
            # Fold batch counters into the job-level perf so the final summary
            # (perf.flush) totals every batch correctly. Safe without a lock —
            # ConsolidationPerfLog.merge_from is a series of += on Python ints
@@ -841,6 +1206,13 @@ async def run_consolidation_job(
        stats["mental_models_refreshed"] = 0
        logger.info(f"[CONSOLIDATION] bank={bank_id} skipping mental model refresh (round limit hit, re-queued)")
    else:
+        set_stage("consolidation.refreshing_mental_models")
+        await memory_engine._write_operation_progress(
+            operation_id,
+            stage="refreshing_mental_models",
+            processed=stats["memories_processed"],
+            total=await _progress_total(stats["memories_processed"]),
+        )
        # SECURITY: Only refresh mental models with matching tags (or all if no tags were consolidated)
        mental_models_refreshed = await _trigger_mental_model_refreshes(
            memory_engine=memory_engine,
@@ -984,6 +1356,9 @@ async def _process_memory_batch(
    """
    import asyncio

+    # Map the source memories this batch consumes onto the consolidation trace.
+    record_source_memory_ids([str(m["id"]) for m in memories])
+
    # 1. Parallel recalls — one per fact
    # When obs_tags_override is set, use it as the observation scope for all facts.
    t0 = time.time()
@@ -1063,6 +1438,20 @@ async def _process_memory_batch(

    mem_by_id = {str(m["id"]): m for m in memories}

+    # Semantic dedup: when enabled, an observation that is >= the threshold cosine to a DIFFERENT
+    # existing observation is reconciled by a focused 1-by-1 LLM merge (anchored on the observation
+    # text, not the source fact). It runs on both CREATE (a near-dup emitted despite the twin being
+    # in context — weak-model failure mode) and UPDATE (a rewrite+re-embed that drifts an existing
+    # observation into a twin — the create-time guard can't see this). The trace operation/scope is
+    # "consolidation_dedup" (routes through the consolidation concurrency bucket via llm_wrapper's
+    # "consolidation" prefix; recorded distinctly in llm_requests).
+    dedup_enabled = _dedup_active(config)
+    dedup_llm_config = (
+        memory_engine._consolidation_llm_config.with_config(config, bank_id=bank_id, operation="consolidation_dedup")
+        if dedup_enabled
+        else None
+    )
+
    # Execute deletes first to free observation slots before creates consume them
    deleted_count = 0
    for delete in llm_result.deletes:
@@ -1087,7 +1476,7 @@ async def _process_memory_batch(
            )
            continue
        agg = _aggregate_source_fields(source_mems, tags=fact_tags)
-        await _execute_update_action(
+        updated_emb_str = await _execute_update_action(
            conn=conn,
            memory_engine=memory_engine,
            bank_id=bank_id,
@@ -1103,17 +1492,76 @@ async def _process_memory_batch(
        )
        for m in source_mems:
            per_memory_updated.add(str(m["id"]))
+        # Reconcile the rewritten observation against its neighbours: the re-embed may have
+        # drifted it into a near-twin of another existing observation (the residual-duplicate
+        # source). updated_emb_str is None when the update was skipped — nothing to reconcile.
+        if dedup_enabled and updated_emb_str is not None:
+            await _dedup_reconcile_update(
+                conn,
+                memory_engine,
+                bank_id,
+                config,
+                dedup_llm_config,
+                update.observation_id,
+                update.text,
+                updated_emb_str,
+                agg.tags,
+            )
+
+    # Deterministic dedup guard: map the observations the LLM was SHOWN by their
+    # normalised text. The model intermittently emits a CREATE whose text is identical
+    # to an observation already in its context (over-aggregation / incoherence — it even
+    # UPDATEs the twin and creates a sibling). When that happens we drop the duplicate
+    # CREATE instead of inserting a redundant row. No extra LLM/embedding cost — the
+    # match is exact text against the in-memory set.
+    shown_obs_by_text = {_norm_obs_text(o.text): o for o in union_observations}
+    # Also collapse a CREATE that reproduces the text of an UPDATE issued in the SAME
+    # response (the model occasionally UPDATEs the twin to text X and also CREATEs X).
+    update_texts = {_norm_obs_text(u.text) for u in llm_result.updates if u.text}

    for create in llm_result.creates:
        source_mems = [mem_by_id[fid] for fid in create.source_fact_ids if fid in mem_by_id]
        if not source_mems:
            continue
        agg = _aggregate_source_fields(source_mems, tags=fact_tags)
+        create_source_ids = [m["id"] for m in source_mems]
+
+        # Reconcile against observations shown to the LLM: an exact-text match means
+        # this CREATE reproduces verbatim an observation the model already had in context.
+        # Since that observation already carries this exact text, drop the duplicate CREATE
+        # — no row is inserted, nothing is lost. We deliberately do NOT also UPDATE the twin
+        # here: the LLM frequently UPDATEd it earlier in this same batch, and a second update
+        # would run off the pre-LLM snapshot and clobber that change (see _dedupe_updates).
+        duplicate_of = _duplicate_create_target(create.text, shown_obs_by_text, update_texts)
+        if duplicate_of is not None:
+            logger.warning(
+                "[CONSOLIDATION] dropped duplicate observation CREATE — verbatim match of %s; llm_reason=%r",
+                duplicate_of,
+                create.reason or "(none given)",
+            )
+            continue
+
+        # Semantic near-duplicate reconciliation: merge this CREATE into an existing
+        # near-identical observation (LLM-adjudicated, 1-by-1) instead of inserting a dup.
+        if dedup_enabled:
+            merged_into = await _dedup_reconcile_create(
+                conn, memory_engine, bank_id, config, dedup_llm_config, create.text, create_source_ids, agg.tags
+            )
+            if merged_into is not None:
+                logger.info(
+                    "[CONSOLIDATION] dedup-merged observation CREATE into %s (cosine>=%.2f)",
+                    merged_into[:8],
+                    config.consolidation_dedup_threshold,
+                )
+                for m in source_mems:
+                    per_memory_created.add(str(m["id"]))
+                continue
+
        await _execute_create_action(
            conn=conn,
            memory_engine=memory_engine,
            bank_id=bank_id,
-            source_memory_ids=[m["id"] for m in source_mems],
+            source_memory_ids=create_source_ids,
            text=create.text,
            source_fact_tags=agg.tags,
            event_date=agg.event_date,
@@ -1153,6 +1601,64 @@ def _max_date(dates: "Any") -> "datetime | None":
    return max((d for d in dates if d is not None), default=None)


+@dataclass(frozen=True)
+class _ObservationHistorySnapshot:
+    """Pre-update state of an observation, persisted as the ``content`` JSON blob
+    of one observation_history row.
+
+    Temporal fields are the ISO strings carried on MemoryFact; new_source_memory_ids
+    are the ids added by the update.
+    """
+
+    previous_text: str | None
+    previous_tags: list[str]
+    previous_occurred_start: str | None
+    previous_occurred_end: str | None
+    previous_mentioned_at: str | None
+    new_source_memory_ids: list[str]
+
+
+async def _append_observation_history(
+    conn: "Connection",
+    bank_id: str,
+    observation_id: str,
+    snapshot: _ObservationHistorySnapshot,
+    max_entries: int,
+) -> None:
+    """Insert one pre-update snapshot into ``observation_history``, then delete the
+    oldest rows beyond ``max_entries`` for this observation.
+
+    The snapshot is stored as a single JSONB ``content`` blob (per-row, so it stays
+    small). Bounding by row count keeps a frequently-reinforced observation's
+    history from growing without bound.
+    """
+    obs_uuid = uuid.UUID(observation_id)
+    await conn.execute(
+        f"""
+        INSERT INTO {fq_table("observation_history")} (observation_id, bank_id, content, changed_at)
+        VALUES ($1, $2, $3::jsonb, now())
+        """,
+        obs_uuid,
+        bank_id,
+        json.dumps(asdict(snapshot)),
+    )
+    if max_entries and max_entries > 0:
+        await conn.execute(
+            f"""
+            DELETE FROM {fq_table("observation_history")}
+            WHERE observation_id = $1
+              AND id NOT IN (
+                  SELECT id FROM {fq_table("observation_history")}
+                  WHERE observation_id = $1
+                  ORDER BY changed_at DESC, id DESC
+                  LIMIT $2
+              )
+            """,
+            obs_uuid,
+            max_entries,
+        )
+
+
 async def _execute_update_action(
    conn: "Connection",
    memory_engine: "MemoryEngine",
@@ -1166,12 +1672,15 @@ async def _execute_update_action(
    source_occurred_end: datetime | None = None,
    source_mentioned_at: datetime | None = None,
    perf: ConsolidationPerfLog | None = None,
-) -> None:
+) -> str | None:
    """
    Update an existing observation.

    Extends source_memory_ids with all contributing memories, updates temporal fields
    (LEAST for occurred_start, GREATEST for occurred_end / mentioned_at), and merges tags.
+
+    Returns the observation's freshly-computed embedding (pgvector literal) so the caller can
+    run UPDATE-path dedup without re-embedding, or None when the update was skipped.
    """
    model = next((m for m in observations if str(m.id) == observation_id), None)
    if not model:
@@ -1189,15 +1698,14 @@ async def _execute_update_action(

    from ...config import get_config

-    history_entry = {
-        "previous_text": model.text,
-        "previous_tags": list(model.tags or []),
-        "previous_occurred_start": model.occurred_start,
-        "previous_occurred_end": model.occurred_end,
-        "previous_mentioned_at": model.mentioned_at,
-        "changed_at": datetime.now(timezone.utc).isoformat(),
-        "new_source_memory_ids": [str(mid) for mid in source_memory_ids],
-    }
+    history_entry = _ObservationHistorySnapshot(
+        previous_text=model.text,
+        previous_tags=list(model.tags or []),
+        previous_occurred_start=model.occurred_start,
+        previous_occurred_end=model.occurred_end,
+        previous_mentioned_at=model.mentioned_at,
+        new_source_memory_ids=[str(mid) for mid in source_memory_ids],
+    )

    source_ids = list(model.source_fact_ids or []) + source_memory_ids

@@ -1213,9 +1721,6 @@ async def _execute_update_action(
        perf.record_timing("embedding", time.time() - t0)

    config = get_config()
-    history_clause = (
-        "history = COALESCE(history, '[]'::jsonb) || $3::jsonb," if config.enable_observation_history else ""
-    )

    t0 = time.time()
    await conn.execute(
@@ -1223,19 +1728,17 @@ async def _execute_update_action(
        UPDATE {fq_table("memory_units")}
        SET text = $1,
            embedding = $2::vector,
-            {history_clause}
-            source_memory_ids = $4,
-            proof_count = $5,
-            tags = $10,
+            source_memory_ids = $3,
+            proof_count = $4,
+            tags = $9,
            updated_at = now(),
-            occurred_start = LEAST(occurred_start, COALESCE($7, occurred_start)),
-            occurred_end = GREATEST(occurred_end, COALESCE($8, occurred_end)),
-            mentioned_at = GREATEST(mentioned_at, COALESCE($9, mentioned_at))
-        WHERE id = $6
+            occurred_start = LEAST(occurred_start, COALESCE($6, occurred_start)),
+            occurred_end = GREATEST(occurred_end, COALESCE($7, occurred_end)),
+            mentioned_at = GREATEST(mentioned_at, COALESCE($8, mentioned_at))
+        WHERE id = $5
        """,
        new_text,
        embedding_str,
-        json.dumps([history_entry]),
        source_ids,
        len(source_ids),
        uuid.UUID(observation_id),
@@ -1245,6 +1748,15 @@ async def _execute_update_action(
        merged_tags,
    )

+    # Record the pre-update snapshot in the dedicated observation_history table
+    # (one row per change), then trim to the configured cap. History lived in a
+    # single unbounded JSONB column before; an often-reinforced observation grew
+    # it until it crossed Postgres's 256MB jsonb limit and got stuck.
+    if config.enable_observation_history:
+        await _append_observation_history(
+            conn, bank_id, observation_id, history_entry, config.observation_history_max_entries
+        )
+
    # Sync observation_sources junction table (Oracle only — PG uses native array ops).
    if memory_engine._backend.ops.uses_observation_sources_table:
        obs_uuid = uuid.UUID(observation_id)
@@ -1265,7 +1777,10 @@ async def _execute_update_action(
    if perf:
        perf.record_timing("db_write", time.time() - t0)

+    # Map the updated observation onto the consolidation trace as a produced memory.
+    record_created_memory_ids([observation_id])
    logger.debug(f"Updated observation {observation_id} from {len(source_memory_ids)} source memories")
+    return embedding_str


 async def _execute_create_action(
@@ -1287,7 +1802,7 @@ async def _execute_create_action(
    Tags are inherited from the source facts (determined algorithmically, not by LLM)
    to maintain visibility scope.
    """
-    await _create_observation_directly(
+    created = await _create_observation_directly(
        conn=conn,
        memory_engine=memory_engine,
        bank_id=bank_id,
@@ -1300,6 +1815,10 @@ async def _execute_create_action(
        mentioned_at=mentioned_at,
        perf=perf,
    )
+    # Map the new observation onto the consolidation trace as a produced memory.
+    new_id = created.get("observation_id")
+    if new_id:
+        record_created_memory_ids([new_id])
    logger.debug(f"Created observation from {len(source_memory_ids)} source memories")


@@ -1398,6 +1917,13 @@ async def _find_related_observations(
            include_source_facts=True,  # Embed source facts so we avoid a separate DB fetch
            max_source_facts_tokens=config.consolidation_source_facts_max_tokens,
            max_source_facts_tokens_per_observation=config.consolidation_source_facts_max_tokens_per_observation,
+            # Round-robin interleave fusion (no cross-encoder): consolidation is looking
+            # for an existing near-identical observation to merge into. Both the
+            # cross-encoder (semantic #1 -> reranked #37) and RRF (semantic #1 -> outside
+            # the 512-token budget) were measured to bury that twin; interleave guarantees
+            # each retrieval arm's top hits a slot, so the semantic-#1 twin is always shown
+            # to the LLM, which then UPDATEs instead of creating a duplicate.
+            reranking="interleave",
            _quiet=True,  # Suppress logging
        )
    finally:
@@ -1532,16 +2058,38 @@ async def _consolidate_batch_with_llm(
                f"(out of {max_observations_per_scope}). Prefer UPDATE over CREATE when possible."
            )

-    prompt_template = build_batch_consolidation_prompt(
-        config.observations_mission,
-        observation_capacity_note,
+    # Split the prompt: a bank-agnostic system instruction (rules + input format +
+    # decision guide + output format) that is byte-identical across batches AND
+    # across banks, and a per-batch user message (mission + capacity note + facts +
+    # existing observations). The split lets the system prefix be served from a
+    # single Gemini context cache shared by every bank — the bank mission, capacity
+    # note, and response_schema (all bank/batch-variable) are kept OUT of the
+    # cached prefix so one cache serves all and it never busts within a run.
+    system_prompt = build_consolidation_system_prompt(
        llm_output_language=getattr(config, "llm_output_language", None),
    )
-    prompt = prompt_template.format(
+    user_content = build_consolidation_input(
        facts_text=facts_lines,
        observations_text=observations_text,
+        observations_mission=config.observations_mission,
+        observation_capacity_note=observation_capacity_note,
    )

+    # Opt into context caching of the stable system prefix when the provider
+    # supports it (gemini/vertexai with the flag on). response_schema is NOT
+    # passed to the fingerprint: it varies per batch (max_creates) but is not
+    # part of the cached prefix, so keying on it would needlessly bust the cache.
+    cached_prefix_name: str | None = None
+    provider_impl = getattr(llm_config, "_provider_impl", None)
+    if provider_impl is not None and provider_impl.supports_prompt_caching():
+        try:
+            cached_prefix_name = await provider_impl.get_or_create_cached_prefix(
+                system_instruction=system_prompt,
+            )
+        except Exception:
+            logger.exception("Consolidation cache prefix lookup failed; falling back to uncached call")
+            cached_prefix_name = None
+
    # Use a constrained response model when observation limit is active
    response_model = _build_response_model(max_creates=remaining_observation_slots)

@@ -1561,12 +2109,17 @@ async def _consolidate_batch_with_llm(
    for attempt in range(1, max_attempts + 1):
        try:
            call_kwargs: dict[str, Any] = {
-                "messages": [{"role": "user", "content": prompt}],
+                "messages": [
+                    {"role": "system", "content": system_prompt},
+                    {"role": "user", "content": user_content},
+                ],
                "response_format": response_model,
                "scope": "consolidation",
            }
            if inner_max_retries is not None:
                call_kwargs["max_retries"] = inner_max_retries
+            if cached_prefix_name is not None:
+                call_kwargs["cached_prefix"] = cached_prefix_name
            response: _ConsolidationBatchResponse = await llm_config.call(**call_kwargs)
            # Defensive truncation: some LLM providers may not enforce JSON schema max_length
            creates = response.creates
@@ -1583,7 +2136,7 @@ async def _consolidate_batch_with_llm(
                updates=updates,
                deletes=response.deletes,
                obs_count=len(union_observations),
-                prompt_chars=len(prompt),
+                prompt_chars=len(system_prompt) + len(user_content),
            )
        except Exception as exc:
            last_exc = exc
@@ -1595,7 +2148,9 @@ async def _consolidate_batch_with_llm(
        f"[CONSOLIDATION] LLM batch call failed after {max_attempts} attempts for {batch_label}, "
        f"skipping batch. Last error: {last_exc}"
    )
-    return _BatchLLMResult(obs_count=len(union_observations), prompt_chars=len(prompt), failed=True)
+    return _BatchLLMResult(
+        obs_count=len(union_observations), prompt_chars=len(system_prompt) + len(user_content), failed=True
+    )


 async def _create_observation_directly(
@@ -1642,10 +2197,10 @@ async def _create_observation_directly(
        # VectorChord: manually tokenize and insert search_vector
        query = f"""
            INSERT INTO {fq_table("memory_units")} (
-                id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
+                id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids,
                tags, event_date, occurred_start, occurred_end, mentioned_at, search_vector
            )
-            VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10,
+            VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, $6, $7, $8, $9, $10,
                    tokenize($3, 'llmlingua2')::bm25_catalog.bm25vector)
            RETURNING id
        """
@@ -1661,10 +2216,10 @@ async def _create_observation_directly(
        # re-ingested. Tracking a separate fix for that gap.
        query = f"""
            INSERT INTO {fq_table("memory_units")} (
-                id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
+                id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids,
                tags, event_date, occurred_start, occurred_end, mentioned_at
            )
-            VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
+            VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, $6, $7, $8, $9, $10)
            RETURNING id
        """

@@ -37,6 +37,33 @@ _PROCESSING_RULES = """## PROCESSING RULES

 9. KEEP DISTINCT TOPICS DISTINCT: do not merge observations about different people, entities, or unrelated topics. Merging is for the same canonical fact recurring — not for related-but-distinct claims."""

+# Stable description of the input shape. For the cached split path this lives in
+# the system prefix (build_consolidation_system_prompt) so it is not re-sent on
+# every batch; the per-batch user message then carries only the actual data.
+_INPUT_FORMAT_NOTE = """## INPUT FORMAT
+
+Each request provides new facts and existing observations:
+- New facts: one per line, each prefixed with its `[uuid]`, followed by the fact text and optional temporal fields.
+- Existing observations: a JSON array pooled from recalls across the new facts. Each entry has:
+  - `id`: unique identifier — copy this exactly when issuing an UPDATE or DELETE
+  - `text`: the observation content
+  - `proof_count`: number of supporting memories
+  - `occurred_start` / `occurred_end`: temporal range of source facts
+  - `source_memories`: array of supporting facts with their text and dates"""
+
+# Per-batch data section for the cached split path — the stable format
+# explanation above is omitted here (it lives in the cached prefix); only the
+# variable facts/observations remain. Placeholders substituted at call time.
+_SPLIT_INPUT_SECTION = """## INPUT
+
+### New facts
+
+{facts_text}
+
+### Existing observations
+
+{observations_text}"""
+
 # Data section — format placeholders {facts_text} and {observations_text} are substituted at call time
 _INPUT_SECTION = """## INPUT

@@ -65,7 +92,7 @@ _DECISION_GUIDE = """## DECISION GUIDE
 # Output format — JSON braces escaped as {{ }} so .format() leaves them literal
 _OUTPUT_SECTION = """## OUTPUT FORMAT

-Return a JSON object with three arrays: `creates`, `updates`, `deletes`.
+Return a JSON object with three arrays: `creates`, `updates`, `deletes`. Every entry must include a `reason`.

 ### Example 1 — Merging recurring claims into an existing observation

@@ -79,7 +106,7 @@ Existing observation:
 Expected output (one UPDATE, no creates — both new facts are additional evidence for the same canonical decision):

 {{"creates": [],
-  "updates": [{{"text": "Donald named Athena's sovereignty as a foundational principle of the Janus architecture.", "observation_id": "11111111-1111-1111-1111-111111111111", "source_fact_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890", "b2c3d4e5-f6a7-8901-bcde-f12345678901"]}}],
+  "updates": [{{"text": "Donald named Athena's sovereignty as a foundational principle of the Janus architecture.", "observation_id": "11111111-1111-1111-1111-111111111111", "source_fact_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890", "b2c3d4e5-f6a7-8901-bcde-f12345678901"], "reason": "Both new facts restate the same sovereignty decision already captured by obs 1111 — merged as evidence rather than creating siblings."}}],
  "deletes": []}}

 ### Example 2 — State change updates one observation; unrelated fact creates a new one
@@ -93,8 +120,8 @@ Existing observation:

 Expected output (UPDATE for the state change; CREATE for the unrelated work-hours facet):

-{{"creates": [{{"text": "Alice works long hours, often past midnight.", "source_fact_ids": ["d4e5f6a7-b8c9-0123-defa-234567890123"]}}],
-  "updates": [{{"text": "Alice owned a 2019 Honda Civic; sold it on March 15, 2025.", "observation_id": "22222222-2222-2222-2222-222222222222", "source_fact_ids": ["c3d4e5f6-a7b8-9012-cdef-123456789012"]}}],
+{{"creates": [{{"text": "Alice works long hours, often past midnight.", "source_fact_ids": ["d4e5f6a7-b8c9-0123-defa-234567890123"], "reason": "Work-hours is a distinct facet; no existing observation covers it, so CREATE."}}],
+  "updates": [{{"text": "Alice owned a 2019 Honda Civic; sold it on March 15, 2025.", "observation_id": "22222222-2222-2222-2222-222222222222", "source_fact_ids": ["c3d4e5f6-a7b8-9012-cdef-123456789012"], "reason": "State change to the existing Honda Civic observation 2222 — UPDATE, not a new sibling."}}],
  "deletes": []}}

 ### Observation text rules
@@ -110,6 +137,7 @@ Expected output (UPDATE for the state change; CREATE for the unrelated work-hour
 - One create or update may reference multiple facts when they jointly support the observation.
 - **AT MOST ONE UPDATE PER `observation_id`**: if several new facts all update the same existing observation, emit a single `updates` entry that lists all contributing `source_fact_ids` and a single consolidated `text`. Never emit two `updates` entries with the same `observation_id` in one response — they would silently overwrite each other.
 - `deletes`: only when an observation is directly superseded or contradicted by new facts.
+- `reason`: REQUIRED on every create/update/delete — one sentence explaining the choice. For a CREATE, state which existing observation(s) you considered and why none matched (a near-identical existing observation means you should UPDATE, not CREATE). This is audited to catch duplicate creates.
 - Do NOT include `tags` — handled automatically.
 - Return `{{"creates": [], "updates": [], "deletes": []}}` if nothing durable is found."""

@@ -145,3 +173,55 @@ def build_batch_consolidation_prompt(
        f"{_DECISION_GUIDE}\n\n"
        f"{_OUTPUT_SECTION}" + output_language_directive(llm_output_language)
    )
+
+
+def build_consolidation_system_prompt(
+    llm_output_language: str | None = None,
+) -> str:
+    """Bank-agnostic, cacheable system instruction for batch consolidation.
+
+    Holds only what is constant across banks: processing rules, input format,
+    decision guide, and output format. The bank's MISSION is deliberately NOT
+    here — baking it in would make the prefix bank-specific and force a separate
+    Gemini context cache per mission. The mission, the per-batch INPUT, and any
+    capacity constraint all ride in the user message (see
+    :func:`build_consolidation_input`), so this prefix is identical for every
+    bank and a single CachedContent serves them all. Returns final text
+    (brace-escaped examples already unescaped) for verbatim use as system message
+    and cached prefix.
+    """
+    template = (
+        "You are a memory consolidation system. Synthesize new facts into "
+        "observations, merging with existing observations when appropriate.\n\n"
+        f"{_MISSION_PRIORITY_NOTE}\n\n"
+        f"{_PROCESSING_RULES}\n\n"
+        f"{_INPUT_FORMAT_NOTE}\n\n"
+        f"{_DECISION_GUIDE}\n\n"
+        f"{_OUTPUT_SECTION}" + output_language_directive(llm_output_language)
+    )
+    # No {facts_text}/{observations_text} placeholders here — the only braces are
+    # the doubled {{ }} in the OUTPUT examples, which .format() unescapes.
+    return template.format()
+
+
+def build_consolidation_input(
+    facts_text: str,
+    observations_text: str,
+    observations_mission: str | None = None,
+    observation_capacity_note: str | None = None,
+) -> str:
+    """Per-batch user message: MISSION + INPUT data + any capacity constraint.
+
+    The MISSION lives here (not in the cached system prefix) so the prefix stays
+    bank-agnostic and one CachedContent serves every bank. The capacity note also
+    lives here since it varies as observation slots fill.
+    """
+    mission = escape_for_prompt(observations_mission or _DEFAULT_MISSION)
+    mission_section = f"## MISSION\n\n{mission}\n\n"
+    capacity_section = ""
+    if observation_capacity_note:
+        capacity_section = f"## CAPACITY CONSTRAINT\n\n{escape_for_prompt(observation_capacity_note)}\n\n"
+    # _SPLIT_INPUT_SECTION omits the stable observation-format explanation (now in
+    # the cached system prefix) — only the variable facts/observations remain.
+    template = mission_section + capacity_section + _SPLIT_INPUT_SECTION
+    return template.format(facts_text=facts_text, observations_text=observations_text)
@@ -46,7 +46,6 @@ from ..config import (
    ENV_RERANKER_FLASHRANK_CPU_MEM_ARENA,
    ENV_RERANKER_FLASHRANK_MODEL,
    ENV_RERANKER_GOOGLE_PROJECT_ID,
-    ENV_RERANKER_LITELLM_SDK_API_KEY,
    ENV_RERANKER_LOCAL_FORCE_CPU,
    ENV_RERANKER_LOCAL_MAX_CONCURRENT,
    ENV_RERANKER_LOCAL_MODEL,
@@ -1199,7 +1198,7 @@ class LiteLLMSDKCrossEncoder(CrossEncoderModel):

    def __init__(
        self,
-        api_key: str,
+        api_key: str | None = None,
        model: str = DEFAULT_RERANKER_LITELLM_SDK_MODEL,
        api_base: str | None = None,
        timeout: float = 60.0,
@@ -1209,7 +1208,8 @@ class LiteLLMSDKCrossEncoder(CrossEncoderModel):
        Initialize LiteLLM SDK cross-encoder client.

        Args:
-            api_key: API key for the reranking provider
+            api_key: API key for the reranking provider (optional — omit for
+                     providers that use ambient credentials, e.g. AWS Bedrock with IAM)
            model: Model name with provider prefix (e.g., "deepinfra/Qwen3-reranker-8B")
            api_base: Custom base URL for API (optional)
            timeout: Request timeout in seconds (default: 60.0)
@@ -1284,8 +1284,9 @@ class LiteLLMSDKCrossEncoder(CrossEncoderModel):
                "model": self.model,
                "query": query,
                "documents": texts,
-                "api_key": self.api_key,
            }
+            if self.api_key:
+                rerank_kwargs["api_key"] = self.api_key
            if self.api_base:
                rerank_kwargs["api_base"] = self.api_base

@@ -1697,13 +1698,8 @@ def create_cross_encoder_from_env() -> CrossEncoderModel:
            timeout=config.reranker_litellm_timeout,
        )
    elif provider == "litellm-sdk":
-        api_key = config.reranker_litellm_sdk_api_key
-        if not api_key:
-            raise ValueError(
-                f"{ENV_RERANKER_LITELLM_SDK_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'litellm-sdk'"
-            )
        return LiteLLMSDKCrossEncoder(
-            api_key=api_key,
+            api_key=config.reranker_litellm_sdk_api_key or None,
            model=config.reranker_litellm_sdk_model,
            api_base=config.reranker_litellm_sdk_api_base,
            max_tokens_per_doc=config.reranker_litellm_max_tokens_per_doc,
@@ -72,6 +72,30 @@ class DataAccessOps(ABC):
        """
        ...

+    @abstractmethod
+    async def lock_document_for_write(
+        self,
+        conn: DatabaseConnection,
+        table: str,
+        doc_id: str,
+        bank_id: str,
+    ) -> str | None:
+        """Ensure the document row exists, take a row lock on it, and return its
+        pre-existing ``content_hash``.
+
+        This serializes all concurrent writers for ``doc_id`` at the DB level
+        (so interleaved same-document retains can't corrupt each other), while
+        creating the row on first write. The returned hash is ``'__pending__'``
+        for a freshly inserted row, the stored hash for an existing one, or
+        ``None`` if the row could not be read back.
+
+        PG does this in a single statement (``INSERT ... ON CONFLICT DO UPDATE
+        ... RETURNING``), which always takes the row lock as part of the upsert.
+        Oracle can't (``MERGE`` doesn't support ``RETURNING``), so it splits the
+        work into an idempotent insert plus a ``SELECT ... FOR UPDATE``.
+        """
+        ...
+
    @abstractmethod
    async def insert_facts_batch(
        self,
@@ -47,6 +47,37 @@ class OracleOps(DataAccessOps):
            column_types=["text[]", "text[]", "text[]", "text[]", "integer[]", "text[]"],
        )

+    async def lock_document_for_write(
+        self,
+        conn: DatabaseConnection,
+        table: str,
+        doc_id: str,
+        bank_id: str,
+    ) -> str | None:
+        # Oracle can't express the PG "INSERT ... ON CONFLICT DO UPDATE ...
+        # RETURNING" upsert in one statement — MERGE doesn't support RETURNING,
+        # so the single-statement form rewrites to a MERGE that returns no rows
+        # (DPY-1003). Split it into two statements instead:
+        #   1. Idempotent insert that silently skips an existing row. The
+        #      IGNORE_ROW_ON_DUPKEY_INDEX hint suppresses ORA-00001 server-side;
+        #      a concurrent uncommitted insert of the same key blocks here until
+        #      the other writer commits, so writers still serialize.
+        #   2. SELECT ... FOR UPDATE to take the row lock and read the hash
+        #      ('__pending__' for a row we just inserted, the stored hash for an
+        #      existing one).
+        await conn.execute(
+            f"INSERT /*+ IGNORE_ROW_ON_DUPKEY_INDEX({table}, pk_documents) */ "
+            f"INTO {table} (id, bank_id, original_text, content_hash) "
+            f"VALUES ($1, $2, '', '__pending__')",
+            doc_id,
+            bank_id,
+        )
+        return await conn.fetchval(
+            f"SELECT content_hash FROM {table} WHERE id = $1 AND bank_id = $2 FOR UPDATE",
+            doc_id,
+            bank_id,
+        )
+
    async def insert_facts_batch(
        self,
        conn: DatabaseConnection,
@@ -49,6 +49,30 @@ class PostgreSQLOps(DataAccessOps):
            content_hashes,
        )

+    async def lock_document_for_write(
+        self,
+        conn: DatabaseConnection,
+        table: str,
+        doc_id: str,
+        bank_id: str,
+    ) -> str | None:
+        # Single upsert that both creates the row (if absent) and locks it (if
+        # present) atomically. ON CONFLICT DO UPDATE always takes the row lock as
+        # part of the statement, so all concurrent same-document writers serialize
+        # on the document row in one consistent step (the earlier two-step form —
+        # DO NOTHING + a separate SELECT FOR UPDATE — could deadlock because
+        # DO NOTHING takes no lock on an existing row). The SET is a no-op
+        # self-assignment used only to acquire the lock; RETURNING yields the
+        # pre-existing hash (or '__pending__' for a freshly inserted row).
+        return await conn.fetchval(
+            f"INSERT INTO {table} (id, bank_id, original_text, content_hash) "
+            f"VALUES ($1, $2, '', '__pending__') "
+            f"ON CONFLICT (id, bank_id) DO UPDATE SET content_hash = {table}.content_hash "
+            f"RETURNING content_hash",
+            doc_id,
+            bank_id,
+        )
+
    async def insert_facts_batch(
        self,
        conn: DatabaseConnection,
@@ -14,6 +14,7 @@ Supports multi-tenant schema isolation via ALTER SESSION SET CURRENT_SCHEMA.
 """

 import datetime
+import inspect
 import json
 import logging
 import re
@@ -155,6 +156,23 @@ _JSON_COL_NAMES = {
    "task_payload",
    "history",
 }
+# NOTE: the history tables' JSON payload column is named ``content`` — deliberately
+# NOT added here, because ``mental_models.content`` is plain text (adding "content"
+# would corrupt those reads). The history read paths json.loads ``content`` directly.
+
+# Columns backed by CLOB in Oracle (large text or JSON). When such a column is
+# returned via a ``RETURNING`` clause it must be bound as DB_TYPE_CLOB; binding
+# it as VARCHAR raises ORA-22835 ("buffer too small for CLOB to CHAR") once the
+# value exceeds 4000 bytes. Union of the JSON-CLOB columns above and the
+# large-text CLOB columns.
+_CLOB_RETURNING_COLS = _JSON_COL_NAMES | {
+    "content",
+    "text",
+    "context",
+    "structured_content",
+    "text_signals",
+    "search_vector",
+}


 def _is_uuid_column(col: str) -> bool:
@@ -685,6 +703,11 @@ class OracleConnection(DatabaseConnection):
                    params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_TIMESTAMP_TZ, arraysize=1)
                elif clean in _NUMERIC_COLS:
                    params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_NUMBER, arraysize=1)
+                elif clean in _CLOB_RETURNING_COLS:
+                    # CLOB-backed column: a VARCHAR out-bind caps at 4000 bytes and
+                    # raises ORA-22835 for larger values. Read back as a LOB in
+                    # _read_returning_values.
+                    params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_CLOB, arraysize=1)
                else:
                    params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_VARCHAR, arraysize=1)

@@ -862,7 +885,7 @@ class OracleConnection(DatabaseConnection):

        return query, params

-    def _read_returning_values(self, returning_cols: list[str], params: dict[str, Any]) -> dict[str, Any] | None:
+    async def _read_returning_values(self, returning_cols: list[str], params: dict[str, Any]) -> dict[str, Any] | None:
        """Read values from RETURNING INTO output variables after execute."""
        row: dict[str, Any] = {}
        for i, col in enumerate(returning_cols):
@@ -872,6 +895,14 @@ class OracleConnection(DatabaseConnection):
                return None
            val = values[0] if isinstance(values, list) else values

+            # CLOB-bound columns return a LOB handle; read it to a string. The
+            # async pool yields AsyncLOB whose read() is a coroutine.
+            if val is not None and not isinstance(val, (str, bytes, int, float)) and hasattr(val, "read"):
+                data = val.read()
+                if inspect.isawaitable(data):
+                    data = await data
+                val = data
+
            # Clean alias: "LOWER(canonical_name) AS name_lower" → "name_lower"
            clean_col = col.strip()
            upper = clean_col.upper()
@@ -1059,7 +1090,7 @@ class OracleConnection(DatabaseConnection):
                raise

            if ret_cols is not None:
-                row_dict = self._read_returning_values(ret_cols, params)
+                row_dict = await self._read_returning_values(ret_cols, params)
                return [ResultRow(row_dict)] if row_dict else []

            columns = [col[0].lower() for col in cursor.description or []]
@@ -1097,7 +1128,7 @@ class OracleConnection(DatabaseConnection):
                raise

            if ret_cols is not None:
-                row_dict = self._read_returning_values(ret_cols, params)
+                row_dict = await self._read_returning_values(ret_cols, params)
                return ResultRow(row_dict) if row_dict else None

            columns = [col[0].lower() for col in cursor.description or []]
@@ -1130,7 +1161,7 @@ class OracleConnection(DatabaseConnection):
            await cursor.execute(query, params)

            if ret_cols is not None:
-                row_dict = self._read_returning_values(ret_cols, params)
+                row_dict = await self._read_returning_values(ret_cols, params)
                if row_dict is None:
                    return None
                vals = list(row_dict.values())
@@ -43,6 +43,10 @@ from ..config import (
    ENV_EMBEDDINGS_LOCAL_FORCE_CPU,
    ENV_EMBEDDINGS_LOCAL_MODEL,
    ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE,
+    ENV_EMBEDDINGS_ONNX_DIMENSIONS,
+    ENV_EMBEDDINGS_ONNX_MODEL_ID,
+    ENV_EMBEDDINGS_ONNX_MODEL_PATH,
+    ENV_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH,
    ENV_EMBEDDINGS_OPENAI_API_KEY,
    ENV_EMBEDDINGS_OPENAI_BASE_URL,
    ENV_EMBEDDINGS_OPENAI_MODEL,
@@ -252,6 +256,172 @@ class LocalSTEmbeddings(Embeddings):
        return [emb.tolist() for emb in embeddings]


+class OnnxEmbeddings(Embeddings):
+    """Local ONNX Runtime embeddings provider.
+
+    This provider runs transformer embedding models in-process with ONNX Runtime,
+    avoiding a sidecar Ollama/TEI server or a remote embeddings API. It supports
+    sentence-transformer style mean pooling and E5-style asymmetric prefixes.
+    """
+
+    def __init__(
+        self,
+        model_id: str,
+        model_path: str | None = None,
+        tokenizer_name_or_path: str | None = None,
+        onnx_file: str = "onnx/model.onnx",
+        dimensions: int | None = None,
+        max_tokens: int = 512,
+        pooling: str = "mean",
+        normalize: bool = True,
+        query_prefix: str = "query: ",
+        passage_prefix: str = "passage: ",
+        output_name: str | None = None,
+    ):
+        self.model_id = model_id
+        self.model_path = model_path
+        if model_path and tokenizer_name_or_path is None:
+            logger.warning(
+                "Embeddings: ONNX model_path is set without tokenizer_name_or_path; "
+                "falling back to tokenizer from model_id %s. Set "
+                "HINDSIGHT_API_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH when using local ONNX artifacts.",
+                model_id,
+            )
+        self.tokenizer_name_or_path = tokenizer_name_or_path or model_id
+        self.onnx_file = onnx_file
+        self.configured_dimensions = dimensions
+        self.max_tokens = max_tokens
+        self.pooling = pooling.lower()
+        if self.pooling not in {"mean", "cls"}:
+            raise ValueError("ONNX embeddings pooling must be 'mean' or 'cls'")
+        self.normalize = normalize
+        self.query_prefix = query_prefix
+        self.passage_prefix = passage_prefix
+        self.output_name = output_name
+        self._session = None
+        self._tokenizer = None
+        self._dimension: int | None = dimensions
+
+    @property
+    def provider_name(self) -> str:
+        return "onnx"
+
+    @property
+    def dimension(self) -> int:
+        if self._dimension is None:
+            raise RuntimeError("Embeddings not initialized. Call initialize() first.")
+        return self._dimension
+
+    async def initialize(self) -> None:
+        if self._session is not None and self._tokenizer is not None:
+            return
+
+        try:
+            import onnxruntime as ort
+            from transformers import AutoTokenizer
+        except ImportError as exc:
+            raise ImportError(
+                "onnxruntime and transformers are required for OnnxEmbeddings. "
+                "Install with: pip install 'hindsight-api-slim[local-onnx]'"
+            ) from exc
+
+        model_path = self.model_path
+        if not model_path:
+            try:
+                from huggingface_hub import snapshot_download
+            except ImportError as exc:
+                raise ImportError(
+                    "huggingface-hub is required to download ONNX embedding models. "
+                    "Set HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH or install local-onnx."
+                ) from exc
+            # Some large ONNX exports, for example BAAI/bge-m3, store weights in
+            # an external sidecar file next to model.onnx. Download both the
+            # requested graph and its conventional *_data sidecar when present.
+            snapshot_dir = snapshot_download(
+                repo_id=self.model_id,
+                allow_patterns=[self.onnx_file, f"{self.onnx_file}_data"],
+            )
+            model_path = os.path.join(snapshot_dir, self.onnx_file)
+
+        logger.info(
+            "Embeddings: initializing ONNX provider with model %s (%s)",
+            self.model_id,
+            model_path,
+        )
+        logger.info(
+            "Embeddings: ONNX query_prefix=%r passage_prefix=%r pooling=%s normalize=%s",
+            self.query_prefix,
+            self.passage_prefix,
+            self.pooling,
+            self.normalize,
+        )
+        self._tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name_or_path)
+        self._session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
+
+        detected = len(self.encode(["test"])[0])
+        if self.configured_dimensions is not None and detected != self.configured_dimensions:
+            raise ValueError(
+                f"Configured ONNX embedding dimension {self.configured_dimensions} does not match model output {detected}"
+            )
+        self._dimension = detected
+        logger.info("Embeddings: ONNX provider initialized (dim: %s)", self._dimension)
+
+    def _encode_prefixed(self, texts: list[str], prefix: str) -> list[list[float]]:
+        if prefix:
+            return self.encode([f"{prefix}{text}" for text in texts])
+        return self.encode(texts)
+
+    def encode_query(self, texts: list[str]) -> list[list[float]]:
+        return self._encode_prefixed(texts, self.query_prefix)
+
+    def encode_documents(self, texts: list[str]) -> list[list[float]]:
+        return self._encode_prefixed(texts, self.passage_prefix)
+
+    def encode(self, texts: list[str]) -> list[list[float]]:
+        if self._session is None or self._tokenizer is None:
+            raise RuntimeError("Embeddings not initialized. Call initialize() first.")
+        if not texts:
+            return []
+
+        import numpy as np
+
+        encoded = self._tokenizer(
+            texts,
+            padding=True,
+            truncation=True,
+            max_length=self.max_tokens,
+            return_tensors="np",
+        )
+        input_names = {inp.name for inp in self._session.get_inputs()}
+        ort_inputs = {name: value for name, value in encoded.items() if name in input_names}
+        if "token_type_ids" in input_names and "token_type_ids" not in ort_inputs:
+            ort_inputs["token_type_ids"] = np.zeros_like(encoded["input_ids"])
+
+        outputs = self._session.run([self.output_name] if self.output_name else None, ort_inputs)
+        token_embeddings = outputs[0]
+
+        # Some exported models expose a pooled 2-D embedding as their first output.
+        if getattr(token_embeddings, "ndim", 0) == 2:
+            embeddings = token_embeddings
+        elif self.pooling == "cls":
+            embeddings = token_embeddings[:, 0]
+        else:
+            attention_mask = encoded.get("attention_mask")
+            if attention_mask is None:
+                attention_mask = np.ones(token_embeddings.shape[:2], dtype=np.float32)
+            mask = attention_mask[..., None].astype(np.float32)
+            summed = (token_embeddings * mask).sum(axis=1)
+            counts = np.clip(mask.sum(axis=1), a_min=1e-9, a_max=None)
+            embeddings = summed / counts
+
+        if self.normalize:
+            norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
+            norms[norms == 0] = 1
+            embeddings = embeddings / norms
+
+        return embeddings.astype(float).tolist()
+
+
 class RemoteTEIEmbeddings(Embeddings):
    """
    Remote embeddings implementation using HuggingFace Text Embeddings Inference (TEI) HTTP API.
@@ -1391,6 +1561,20 @@ def create_embeddings_from_env() -> Embeddings:
            force_cpu=config.embeddings_local_force_cpu,
            trust_remote_code=config.embeddings_local_trust_remote_code,
        )
+    elif provider == "onnx":
+        return OnnxEmbeddings(
+            model_id=config.embeddings_onnx_model_id,
+            model_path=config.embeddings_onnx_model_path,
+            tokenizer_name_or_path=config.embeddings_onnx_tokenizer_name_or_path,
+            onnx_file=config.embeddings_onnx_file,
+            dimensions=config.embeddings_onnx_dimensions,
+            max_tokens=config.embeddings_onnx_max_tokens,
+            pooling=config.embeddings_onnx_pooling,
+            normalize=config.embeddings_onnx_normalize,
+            query_prefix=config.embeddings_onnx_query_prefix,
+            passage_prefix=config.embeddings_onnx_passage_prefix,
+            output_name=config.embeddings_onnx_output_name,
+        )
    elif provider == "openai":
        # Use dedicated embeddings API key, or fall back to LLM API key
        api_key = os.environ.get(ENV_EMBEDDINGS_OPENAI_API_KEY) or os.environ.get(ENV_LLM_API_KEY)
@@ -1492,6 +1676,6 @@ def create_embeddings_from_env() -> Embeddings:
    else:
        raise ValueError(
            f"Unknown embeddings provider: {provider}. "
-            f"Supported: 'local', 'tei', 'openai', 'openai-codex', 'openrouter', 'cohere', 'google', "
+            f"Supported: 'local', 'onnx', 'tei', 'openai', 'openai-codex', 'openrouter', 'cohere', 'google', "
            f"'zeroentropy', 'litellm', 'litellm-sdk'"
        )
@@ -458,8 +458,28 @@ class MemoryEngineInterface(ABC):
            request_context: Request context for authentication.

        Returns:
-            Dict with node_counts, link_counts, link_counts_by_fact_type,
-            link_breakdown, and operations stats.
+            Dict with node_counts, link_counts, link_counts_by_fact_type
+            (deprecated, returns empty), link_breakdown (deprecated, returns
+            empty), and operations stats.
+        """
+        ...
+
+    @abstractmethod
+    async def get_bank_freshness(
+        self,
+        bank_id: str,
+        *,
+        request_context: "RequestContext",
+    ) -> dict[str, Any]:
+        """
+        Get consolidation freshness for a bank.
+
+        Cheap alternative to get_bank_stats when callers only need
+        last_consolidated_at / pending_consolidation / failed_consolidation.
+
+        Returns:
+            Dict with last_consolidated_at (ISO-8601 string or None),
+            pending_consolidation (int), and failed_consolidation (int).
        """
        ...

@@ -69,6 +69,7 @@ class LLMInterface(ABC):
        skip_validation: bool = False,
        strict_schema: bool = False,
        return_usage: bool = False,
+        cached_prefix: str | None = None,
    ) -> Any:
        """
        Make an LLM API call with retry logic.
@@ -83,8 +84,13 @@ class LLMInterface(ABC):
            initial_backoff: Initial backoff time in seconds.
            max_backoff: Maximum backoff time in seconds.
            skip_validation: Return raw JSON without Pydantic validation.
-            strict_schema: Use strict JSON schema enforcement (OpenAI only).
+            strict_schema: Grammar-enforce structured output via json_schema strict
+                (OpenAI-compatible, LiteLLM) instead of the soft json_object path. Gemini
+                enforces its response_schema natively; providers without a strict mode ignore it.
            return_usage: If True, return tuple (result, TokenUsage) instead of just result.
+            cached_prefix: Opaque handle from ``get_or_create_cached_prefix`` for the
+                cacheable system prefix, or None. Providers without explicit prompt
+                caching ignore it (and the wrapper only forwards it when set).

        Returns:
            If return_usage=False: Parsed response if response_format is provided, otherwise text content.
@@ -108,6 +114,7 @@ class LLMInterface(ABC):
        initial_backoff: float = 1.0,
        max_backoff: float = 30.0,
        tool_choice: str | dict[str, Any] = "auto",
+        cached_prefix: str | None = None,
    ) -> LLMToolCallResult:
        """
        Make an LLM API call with tool/function calling support.
@@ -137,6 +144,46 @@ class LLMInterface(ABC):
        """
        return False

+    # ── Prompt prefix caching (optional, per-provider) ─────────────────────────
+
+    def supports_prompt_caching(self) -> bool:
+        """Whether this provider can cache a reusable prompt prefix.
+
+        Default False. Providers that return True must implement
+        ``get_or_create_cached_prefix`` and honour the ``cached_prefix`` argument
+        of ``call`` / ``call_with_tools``.
+        """
+        return False
+
+    async def get_or_create_cached_prefix(
+        self,
+        *,
+        system_instruction: str,
+        response_schema: Any | None = None,
+        tools: list[dict[str, Any]] | None = None,
+    ) -> str | None:
+        """Cache a reusable prompt prefix and return an opaque handle, or None.
+
+        The engine has already decided WHAT is cacheable: it puts the stable,
+        bank-agnostic instructions in ``system_instruction`` (plus ``tools``) and
+        keeps all per-request / per-bank data (documents, facts, the bank mission)
+        in the user message. A provider only chooses HOW to cache that prefix:
+
+        - Explicit-cache providers (e.g. Gemini ``CachedContent``): create the
+          cache, return its handle; the engine passes the handle back via
+          ``call(cached_prefix=...)`` and the provider then drops the prefix from
+          the request, billing it at the cached rate.
+        - Automatic-cache providers (e.g. OpenAI): no handle needed — caching is
+          transparent as long as the prefix is a stable leading block, which it
+          already is. They can keep this default (return None) and still benefit.
+        - Inline-marker providers (e.g. Anthropic ``cache_control``): mark the
+          prefix block inside ``call`` instead; may also keep this default.
+
+        Returns None when caching is disabled/unsupported or the prefix is too
+        small; callers MUST fall back to an uncached call in that case.
+        """
+        return None
+
    async def submit_batch(
        self,
        requests: list[dict[str, Any]],
@@ -0,0 +1,540 @@
+"""Per-bank LLM request tracing.
+
+Opt-in, fire-and-forget recording of every LLM call Hindsight makes (both
+successes and failures) into the ``llm_requests`` table, per bank. Each row
+captures the input messages, the model output, token usage (input / output /
+cached / total), finish reason, and caller metadata. Disabled by default —
+controlled by ``HINDSIGHT_API_LLM_TRACE_ENABLED``.
+
+This plugs into the OpenTelemetry **GenAI** recording pattern: providers already
+call ``tracing.get_span_recorder().record_llm_call(...)`` on success, so the DB
+tracer is registered as one of those recorders (alongside the OTLP span
+exporter) rather than hooking the call path with custom code. Failures, which
+providers don't report to the recorder, are forwarded from the LLM wrapper.
+
+Bank/operation attribution is carried via a ContextVar set by
+``ConfiguredLLMProvider`` (see ``llm_wrapper.py``); outside a traced context
+``bank_id`` is recorded as NULL.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import uuid
+from collections.abc import Callable, Iterable
+from contextvars import ContextVar, Token
+from dataclasses import dataclass, field
+from datetime import datetime, timedelta, timezone
+from typing import Any
+
+from pydantic import BaseModel
+
+from .db_utils import acquire_with_retry
+
+logger = logging.getLogger(__name__)
+
+
+# ── bank/operation attribution (carried across the async call chain) ──────────
+
+
+@dataclass
+class LLMTraceContext:
+    """Attribution for in-flight LLM calls, bound by ``ConfiguredLLMProvider``.
+
+    ``trace_id`` and ``operation_span_id`` are generated once per operation
+    invocation (one ``with_config`` call), so every LLM call of a single
+    reflect/retain/consolidation run shares them — reproducing the OTel
+    parent (operation span) → children (LLM calls) hierarchy in the DB.
+    """
+
+    bank_id: str | None = None
+    operation: str | None = None  # "retain" | "reflect" | "consolidation" | ...
+    metadata: dict[str, Any] = field(default_factory=dict)
+    trace_id: str | None = None
+    operation_span_id: str | None = None
+    # Memory_units this operation produced/consumed, accumulated at the DB-write
+    # sites and flushed onto every row of the trace at operation end (see
+    # LLMTraceRecorder.attach_memory_ids). Lets a retain/consolidation trace map
+    # to the memories it created (outputs) and consumed (source inputs).
+    created_memory_ids: list[str] = field(default_factory=list)
+    source_memory_ids: list[str] = field(default_factory=list)
+
+
+_trace_ctx: ContextVar[LLMTraceContext | None] = ContextVar("hindsight_llm_trace_ctx", default=None)
+
+# Per-call requested parameters (max_completion_tokens, temperature, response
+# schema, tool_choice). Set by ``LLMProvider.call`` around the provider
+# delegation so the recorder can attach them even though success is reported by
+# the provider. Only includes values the caller actually set — never nulls.
+_request_ctx: ContextVar[dict[str, Any] | None] = ContextVar("hindsight_llm_request_ctx", default=None)
+
+# Per-call caller metadata (e.g. document_id for retain extraction). Set by
+# engine code around a specific LLM call; merged into the row's metadata on top
+# of the operation-level LLMTraceContext.metadata.
+_call_metadata_ctx: ContextVar[dict[str, Any] | None] = ContextVar("hindsight_llm_call_metadata_ctx", default=None)
+
+
+def set_trace_context(ctx: LLMTraceContext | None) -> Token:
+    """Bind trace attribution to the current context. Returns a reset token."""
+    return _trace_ctx.set(ctx)
+
+
+def reset_trace_context(token: Token) -> None:
+    """Unwind a binding made by :func:`set_trace_context`."""
+    _trace_ctx.reset(token)
+
+
+def set_request_context(params: dict[str, Any] | None) -> Token:
+    """Bind the current LLM call's requested parameters. Returns a reset token."""
+    return _request_ctx.set(params)
+
+
+def reset_request_context(token: Token) -> None:
+    """Unwind a binding made by :func:`set_request_context`."""
+    _request_ctx.reset(token)
+
+
+def current_request_context() -> dict[str, Any] | None:
+    """Return the active call's requested parameters, or None."""
+    return _request_ctx.get()
+
+
+def set_call_metadata(metadata: dict[str, Any] | None) -> Token:
+    """Bind per-call caller metadata (e.g. ``{"document_id": ...}``)."""
+    return _call_metadata_ctx.set(metadata)
+
+
+def reset_call_metadata(token: Token) -> None:
+    """Unwind a binding made by :func:`set_call_metadata`."""
+    _call_metadata_ctx.reset(token)
+
+
+def current_call_metadata() -> dict[str, Any] | None:
+    """Return the active call's caller metadata, or None."""
+    return _call_metadata_ctx.get()
+
+
+def current_trace_context() -> LLMTraceContext | None:
+    """Return the active trace attribution, or None outside a traced context."""
+    return _trace_ctx.get()
+
+
+def trace_context_of(llm_config: Any) -> LLMTraceContext | None:
+    """Return a configured provider's operation trace context, or None.
+
+    Real providers expose ``trace_context()`` (``ConfiguredLLMProvider``); test
+    or mock substitutes may not, so this degrades gracefully rather than raising
+    — tracing is best-effort and must never break an operation.
+    """
+    getter = getattr(llm_config, "trace_context", None)
+    return getter() if callable(getter) else None
+
+
+def record_created_memory_ids(ids: Iterable[str]) -> None:
+    """Accumulate output memory_units onto the active operation trace.
+
+    No-op outside a traced operation context (e.g. tracing disabled). Child
+    asyncio tasks inherit the same ``LLMTraceContext`` object, so appends from
+    parallel consolidation batches land on one shared list.
+    """
+    ctx = _trace_ctx.get()
+    if ctx is not None:
+        ctx.created_memory_ids.extend(str(i) for i in ids)
+
+
+def record_source_memory_ids(ids: Iterable[str]) -> None:
+    """Accumulate consumed/source memory_units onto the active operation trace.
+
+    No-op outside a traced operation context.
+    """
+    ctx = _trace_ctx.get()
+    if ctx is not None:
+        ctx.source_memory_ids.extend(str(i) for i in ids)
+
+
+# ── serialization helpers ─────────────────────────────────────────────────────
+
+
+def _json_default(obj: Any) -> Any:
+    """JSON serializer for objects not serializable by default."""
+    if isinstance(obj, datetime):
+        return obj.isoformat()
+    if isinstance(obj, uuid.UUID):
+        return str(obj)
+    if isinstance(obj, bytes):
+        return "<bytes>"
+    if isinstance(obj, set):
+        return list(obj)
+    model_dump = getattr(obj, "model_dump", None)
+    if callable(model_dump):
+        try:
+            return model_dump(mode="json")
+        except Exception:
+            return str(obj)
+    return str(obj)
+
+
+def _safe_json(data: Any, max_chars: int) -> str | None:
+    """Serialize ``data`` to a JSON string, truncating beyond ``max_chars``.
+
+    Returns None on total failure. Truncation preserves valid JSON by wrapping
+    the oversized payload in a marker object with a preview.
+    """
+    if data is None:
+        return None
+    try:
+        serialized = json.dumps(data, default=_json_default)
+    except Exception:
+        logger.debug("Failed to serialize llm trace data", exc_info=True)
+        try:
+            serialized = json.dumps(str(data))
+        except Exception:
+            return None
+    if max_chars and max_chars > 0 and len(serialized) > max_chars:
+        return json.dumps({"_truncated": True, "_original_chars": len(serialized), "preview": serialized[:max_chars]})
+    return serialized
+
+
+# ── record ────────────────────────────────────────────────────────────────────
+
+
+@dataclass
+class LLMRequestRecord:
+    """A single LLM request trace row."""
+
+    provider: str
+    model: str | None
+    scope: str
+    status: str  # "success" | "error"
+    started_at: datetime
+    ended_at: datetime
+    bank_id: str | None = None
+    operation: str | None = None
+    trace_id: str | None = None
+    span_id: str | None = None
+    parent_span_id: str | None = None
+    input: Any = None
+    output: Any = None
+    error: str | None = None
+    input_tokens: int | None = None
+    output_tokens: int | None = None
+    cached_tokens: int | None = None
+    total_tokens: int | None = None
+    llm_info: dict[str, Any] = field(default_factory=dict)
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+    @property
+    def duration_ms(self) -> int:
+        return int((self.ended_at - self.started_at).total_seconds() * 1000)
+
+
+# ── read models (returned by MemoryEngine query methods, served by the API) ───
+
+
+class LLMRequestEntry(BaseModel):
+    """A single LLM request trace row, as returned by the read API."""
+
+    id: str
+    bank_id: str | None
+    operation: str | None
+    scope: str | None
+    trace_id: str | None
+    span_id: str | None
+    parent_span_id: str | None
+    provider: str | None
+    model: str | None
+    status: str
+    started_at: str | None
+    ended_at: str | None
+    duration_ms: int | None
+    input_tokens: int | None
+    output_tokens: int | None
+    cached_tokens: int | None
+    total_tokens: int | None
+    # Arbitrary JSON (message list, string, or object) — open `Any` so the
+    # OpenAPI schema stays a plain open type the Go SDK generator can model.
+    input: Any = None
+    output: Any = None
+    error: str | None
+    llm_info: dict[str, Any]
+    metadata: dict[str, Any]
+
+
+class LLMRequestListResponse(BaseModel):
+    """Paginated list of LLM request traces for a bank."""
+
+    bank_id: str
+    total: int
+    limit: int
+    offset: int
+    items: list[LLMRequestEntry]
+
+
+class LLMRequestTokenSums(BaseModel):
+    """Token totals for a time bucket."""
+
+    input: int
+    output: int
+    cached: int
+    total: int
+
+
+class LLMRequestStatsBucket(BaseModel):
+    """A single time bucket in LLM request stats."""
+
+    time: str
+    statuses: dict[str, int]
+    total: int
+    tokens: LLMRequestTokenSums
+
+
+class LLMRequestStatsResponse(BaseModel):
+    """LLM request counts and token sums grouped by time bucket."""
+
+    bank_id: str
+    period: str
+    trunc: str
+    start: str
+    buckets: list[LLMRequestStatsBucket]
+
+
+# ── recorder / writer ─────────────────────────────────────────────────────────
+
+
+class LLMTraceRecorder:
+    """GenAI span recorder that writes per-bank LLM traces to ``llm_requests``.
+
+    Implements ``record_llm_call`` so it can be registered with
+    :func:`hindsight_api.tracing.register_span_recorder`. Writes are
+    fire-and-forget and never surface errors into the calling path. Retention of
+    old rows is handled by the background :class:`MaintenanceLoop`.
+    """
+
+    def __init__(
+        self,
+        pool_getter: Callable[[], Any],
+        schema_getter: Callable[[], str],
+        enabled: bool,
+        allowed_scopes: list[str],
+        max_chars: int = 50000,
+    ) -> None:
+        self._pool_getter = pool_getter
+        self._schema_getter = schema_getter
+        self._enabled = enabled
+        self._allowed_scopes: frozenset[str] | None = frozenset(allowed_scopes) if allowed_scopes else None
+        self._max_chars = max_chars
+        # In-flight fire-and-forget write tasks, bucketed by trace_id so
+        # attach_memory_ids can await only *its own* operation's writes before the
+        # post-operation UPDATE (otherwise the UPDATE could race ahead of the
+        # INSERTs it patches — but it must not block on unrelated operations).
+        self._pending: dict[str | None, set[asyncio.Task]] = {}
+
+    def is_enabled(self, scope: str) -> bool:
+        """Whether tracing is active for the given call scope."""
+        if not self._enabled:
+            return False
+        if self._allowed_scopes is not None:
+            return scope in self._allowed_scopes
+        return True
+
+    # ── GenAI recorder interface ──────────────────────────────────────────────
+
+    def record_llm_call(
+        self,
+        provider: str,
+        model: str,
+        scope: str,
+        messages: list[dict[str, Any]],
+        response_content: Any = None,
+        input_tokens: int = 0,
+        output_tokens: int = 0,
+        duration: float = 0.0,
+        finish_reason: str | None = None,
+        error: BaseException | None = None,
+        tool_calls: list[dict[str, Any]] | None = None,
+        cached_tokens: int = 0,
+        **_extra: Any,
+    ) -> None:
+        """Build a trace record from a GenAI call and schedule a DB write."""
+        if not self.is_enabled(scope):
+            return
+
+        ctx = current_trace_context()
+        ended_at = datetime.now(timezone.utc)
+        started_at = ended_at - timedelta(seconds=max(0.0, duration))
+
+        # Operation-level metadata + any per-call metadata (e.g. document_id).
+        metadata = dict(ctx.metadata) if ctx else {}
+        call_metadata = current_call_metadata()
+        if call_metadata:
+            metadata.update(call_metadata)
+
+        llm_info: dict[str, Any] = {}
+        request_params = current_request_context()
+        if request_params:
+            llm_info["request"] = dict(request_params)
+        if finish_reason:
+            llm_info["finish_reason"] = finish_reason
+        if tool_calls:
+            llm_info["tool_calls"] = [tc.get("name", "") for tc in tool_calls]
+
+        record = LLMRequestRecord(
+            provider=provider,
+            model=model,
+            scope=scope,
+            status="error" if error is not None else "success",
+            started_at=started_at,
+            ended_at=ended_at,
+            bank_id=ctx.bank_id if ctx else None,
+            operation=ctx.operation if ctx else None,
+            # OTel-style hierarchy: all calls of one operation invocation share
+            # the context's trace_id and point at its operation span; this call
+            # gets its own span_id.
+            trace_id=ctx.trace_id if ctx else None,
+            span_id=str(uuid.uuid4()),
+            parent_span_id=ctx.operation_span_id if ctx else None,
+            input=messages,
+            output=None if error is not None else response_content,
+            error=f"{type(error).__name__}: {error}" if error is not None else None,
+            input_tokens=input_tokens or None,
+            output_tokens=output_tokens or None,
+            cached_tokens=cached_tokens or None,
+            total_tokens=(input_tokens + output_tokens) or None,
+            llm_info=llm_info,
+            metadata=metadata,
+        )
+        self._record_fire_and_forget(record)
+
+    def _record_fire_and_forget(self, record: LLMRequestRecord) -> None:
+        """Schedule a trace write as a background task."""
+        try:
+            task = asyncio.create_task(self._safe_write(record))
+        except RuntimeError:
+            # No running event loop (e.g. during shutdown)
+            logger.debug("Cannot schedule llm trace write: no running event loop")
+            return
+        key = record.trace_id
+        self._pending.setdefault(key, set()).add(task)
+        task.add_done_callback(lambda t, k=key: self._discard_pending(k, t))
+
+    def _discard_pending(self, key: str | None, task: asyncio.Task) -> None:
+        bucket = self._pending.get(key)
+        if bucket is not None:
+            bucket.discard(task)
+            if not bucket:
+                self._pending.pop(key, None)
+
+    async def _safe_write(self, record: LLMRequestRecord) -> None:
+        """Write a trace row. Errors are logged, never raised."""
+        pool = self._pool_getter()
+        if pool is None:
+            logger.debug("LLM trace skipped: pool not available")
+            return
+        try:
+            schema = self._schema_getter()
+            table = f"{schema}.llm_requests"
+            async with acquire_with_retry(pool, max_retries=1) as conn:
+                await conn.execute(
+                    f"""
+                    INSERT INTO {table}
+                        (id, bank_id, operation, scope, trace_id, span_id, parent_span_id,
+                         provider, model, status,
+                         started_at, ended_at, duration_ms,
+                         input_tokens, output_tokens, cached_tokens, total_tokens,
+                         input, output, error, llm_info, metadata)
+                    VALUES
+                        ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10,
+                         $11, $12, $13, $14, $15, $16, $17,
+                         $18::jsonb, $19::jsonb, $20, $21::jsonb, $22::jsonb)
+                    """,
+                    uuid.uuid4(),
+                    record.bank_id,
+                    record.operation,
+                    record.scope,
+                    record.trace_id,
+                    record.span_id,
+                    record.parent_span_id,
+                    record.provider,
+                    record.model,
+                    record.status,
+                    record.started_at,
+                    record.ended_at,
+                    record.duration_ms,
+                    record.input_tokens,
+                    record.output_tokens,
+                    record.cached_tokens,
+                    record.total_tokens,
+                    _safe_json(record.input, self._max_chars),
+                    _safe_json(record.output, self._max_chars),
+                    record.error,
+                    _safe_json(record.llm_info, self._max_chars) or "{}",
+                    _safe_json(record.metadata, self._max_chars) or "{}",
+                )
+        except Exception as e:
+            logger.warning(f"LLM trace write failed for scope={record.scope}: {e}")
+
+    async def _flush_pending(self, trace_id: str) -> None:
+        """Await this trace's in-flight writes so its rows exist before an UPDATE."""
+        pending = [t for t in self._pending.get(trace_id, ()) if not t.done()]
+        if pending:
+            await asyncio.gather(*pending, return_exceptions=True)
+
+    def attach_memory_ids(
+        self,
+        trace_ctx: LLMTraceContext | None,
+        *,
+        created: list[str] | None = None,
+        source: list[str] | None = None,
+    ) -> None:
+        """Map a finished operation's memory_units onto every row of its trace.
+
+        Merges the explicitly passed ids with any accumulated on the context
+        (``record_created_memory_ids`` / ``record_source_memory_ids``), de-dupes
+        preserving order, and patches ``metadata.memory_ids`` (outputs created)
+        and ``metadata.source_memory_ids`` (inputs consumed) on all rows sharing
+        the trace_id. No-op when tracing is off or nothing was produced.
+
+        Fire-and-forget: the snapshotted patch is applied on a background task so
+        the retain/consolidation operation never waits on the trace write. The
+        ids are snapshotted synchronously here because the caller may reset the
+        context immediately after.
+        """
+        if not self._enabled or trace_ctx is None or not trace_ctx.trace_id:
+            return
+        created_ids = list(dict.fromkeys([*(created or []), *trace_ctx.created_memory_ids]))
+        source_ids = list(dict.fromkeys([*(source or []), *trace_ctx.source_memory_ids]))
+        patch: dict[str, Any] = {}
+        if created_ids:
+            patch["memory_ids"] = created_ids
+        if source_ids:
+            patch["source_memory_ids"] = source_ids
+        if not patch:
+            return
+        try:
+            asyncio.create_task(self._attach_memory_ids(trace_ctx.bank_id, trace_ctx.trace_id, patch))
+        except RuntimeError:
+            logger.debug("Cannot schedule llm trace memory_id attach: no running event loop")
+
+    async def _attach_memory_ids(self, bank_id: str | None, trace_id: str, patch: dict[str, Any]) -> None:
+        """Background worker: flush this trace's writes, then patch its rows."""
+        # The trace-row INSERTs are fire-and-forget; flush *this trace's* writes
+        # so the UPDATE patches rows that already exist rather than racing ahead
+        # of them (without blocking on unrelated operations' pending writes).
+        await self._flush_pending(trace_id)
+        pool = self._pool_getter()
+        if pool is None:
+            return
+        try:
+            schema = self._schema_getter()
+            table = f"{schema}.llm_requests"
+            async with acquire_with_retry(pool, max_retries=1) as conn:
+                await conn.execute(
+                    f"UPDATE {table} SET metadata = metadata || $3::jsonb WHERE bank_id = $1 AND trace_id = $2",
+                    bank_id,
+                    trace_id,
+                    json.dumps(patch),
+                )
+        except Exception as e:
+            logger.warning(f"LLM trace memory_id attach failed for trace={trace_id}: {e}")
@@ -114,6 +114,32 @@ def _semaphores_for_scope(scope: str) -> list[asyncio.Semaphore]:
    return [per_op, _global_llm_semaphore]


+def _request_params(
+    *,
+    max_completion_tokens: int | None = None,
+    temperature: float | None = None,
+    scope: str | None = None,
+    response_format: Any | None = None,
+    tool_choice: str | dict[str, Any] | None = None,
+) -> dict[str, Any] | None:
+    """Build the requested-params bag for tracing — only values the caller set.
+
+    Omitting unset values avoids the misleading nulls we used to record (e.g.
+    consolidation, which passes no token cap), while surfacing the real cap for
+    callers that do set one (e.g. retain's ``retain_max_completion_tokens``).
+    """
+    params: dict[str, Any] = {}
+    if max_completion_tokens is not None:
+        params["max_completion_tokens"] = max_completion_tokens
+    if temperature is not None:
+        params["temperature"] = temperature
+    if response_format is not None:
+        params["response_schema"] = getattr(response_format, "__name__", None) or "structured"
+    if tool_choice is not None and tool_choice != "auto":
+        params["tool_choice"] = tool_choice if isinstance(tool_choice, str) else "named"
+    return params or None
+
+
 def sanitize_text(text: str | None) -> str | None:
    """
    Sanitize text by removing characters that break downstream systems.
@@ -229,6 +255,7 @@ def create_llm_provider(
    vertexai_region: str | None = None,
    vertexai_credentials: Any = None,
    gemini_safety_settings: list | None = None,
+    prompt_cache_enabled: bool = False,
    litellmrouter_config: dict[str, Any] | None = None,
 ) -> Any:  # Returns LLMInterface
    """
@@ -242,7 +269,11 @@ def create_llm_provider(
        reasoning_effort: Reasoning effort level for supported providers.
        groq_service_tier: Groq service tier (for Groq provider) - "on_demand", "flex", or "auto".
        openai_service_tier: OpenAI service tier (for OpenAI provider) - None (default) or "flex" (50% cheaper).
-        extra_body: Extra body params merged into OpenAI-compatible API calls.
+        extra_body: Extra request-body params merged into the provider's native
+            call. Threaded into OpenAI-compatible, Fireworks, Anthropic, Gemini/
+            VertexAI and LiteLLM providers (each merges them in its own parameter
+            space). Keys must use each provider's native names (e.g. ``max_tokens``
+            for OpenAI/Anthropic vs ``max_output_tokens`` for Gemini).
        default_headers: Custom headers passed as ``default_headers`` to provider SDK clients
            (used by operators routing through proxies / request-tracing middleware). Currently
            wired into the Anthropic provider; other providers may opt in as needed.
@@ -317,6 +348,8 @@ def create_llm_provider(
            vertexai_region=vertexai_region,
            vertexai_credentials=vertexai_credentials,
            gemini_safety_settings=gemini_safety_settings,
+            prompt_cache_enabled=prompt_cache_enabled,
+            extra_body=extra_body,
        )

    elif provider_lower == "anthropic":
@@ -327,6 +360,7 @@ def create_llm_provider(
            model=model,
            reasoning_effort=reasoning_effort,
            default_headers=default_headers,
+            extra_body=extra_body,
        )

    elif provider_lower == "litellm":
@@ -336,6 +370,7 @@ def create_llm_provider(
            base_url=base_url,
            model=model,
            reasoning_effort=reasoning_effort,
+            extra_body=extra_body,
        )

    elif provider_lower == "litellmrouter":
@@ -353,6 +388,7 @@ def create_llm_provider(
            model=model,
            config=litellmrouter_config,
            reasoning_effort=reasoning_effort,
+            extra_body=extra_body,
        )

    elif provider_lower == "bedrock":
@@ -364,6 +400,7 @@ def create_llm_provider(
            base_url=base_url,
            model=bedrock_model,
            reasoning_effort=reasoning_effort,
+            extra_body=extra_body,
        )

    elif provider_lower == "llamacpp":
@@ -442,6 +479,7 @@ class LLMProvider:
        groq_service_tier: str | None = None,
        openai_service_tier: str | None = None,
        gemini_safety_settings: list | None = None,
+        prompt_cache_enabled: bool = False,
        extra_body: dict[str, Any] | None = None,
        default_headers: dict[str, str] | None = None,
        litellmrouter_config: dict[str, Any] | None = None,
@@ -458,7 +496,8 @@ class LLMProvider:
            groq_service_tier: Groq service tier ("on_demand", "flex", "auto") - from config.
            openai_service_tier: OpenAI service tier (None or "flex") - from config.
            gemini_safety_settings: Safety settings for Gemini/VertexAI providers.
-            extra_body: Extra body params merged into OpenAI-compatible API calls.
+            extra_body: Extra request-body params merged into the provider's native call
+                (OpenAI-compatible, Fireworks, Anthropic, Gemini/VertexAI, LiteLLM).
            default_headers: Custom headers passed as ``default_headers`` to provider SDK clients.
                Used by operators routing through proxies / request-tracing middleware. Falls
                back to ``HindsightConfig.llm_default_headers`` (env: ``HINDSIGHT_API_LLM_DEFAULT_HEADERS``)
@@ -480,6 +519,11 @@ class LLMProvider:
        self.openai_service_tier = openai_service_tier
        # Gemini safety settings (instance default; can be overridden per-request via context var)
        self.gemini_safety_settings = gemini_safety_settings
+        # Gemini prompt caching: when True, retain extraction (and any future
+        # caller that opts in) will reuse a CachedContent prefix to cut
+        # input-token cost. Off by default so the change is observable behind
+        # a flip rather than a silent behaviour change on upgrade.
+        self.prompt_cache_enabled = prompt_cache_enabled
        # Extra body params for OpenAI-compatible providers (e.g. chat_template_kwargs)
        self.extra_body = extra_body
        # Default headers passed to provider SDK clients (e.g. proxy auth, request tracing).
@@ -598,6 +642,21 @@ class LLMProvider:
            except Exception:
                pass  # Config may not be initialized in test environments

+        # Prompt-prefix caching is a provider-agnostic toggle (default on): resolve
+        # it from the static server config for every provider when the caller didn't
+        # pass an explicit override. Providers that don't support caching ignore the
+        # value; only those that implement get_or_create_cached_prefix act on it.
+        if not self.prompt_cache_enabled:
+            from ..config import DEFAULT_LLM_PROMPT_CACHE_ENABLED, _get_raw_config
+
+            try:
+                raw_config = _get_raw_config()
+                self.prompt_cache_enabled = bool(
+                    getattr(raw_config, "llm_prompt_cache_enabled", DEFAULT_LLM_PROMPT_CACHE_ENABLED)
+                )
+            except Exception:
+                pass  # Config may not be initialized in test environments
+
        # For litellmrouter: prefer an explicit chain from the caller (per-op
        # construction in MemoryEngine threads the right chain through). If the caller
        # didn't supply one, fall back to the global ``llm_litellmrouter_config`` so
@@ -626,6 +685,7 @@ class LLMProvider:
            vertexai_region=vertexai_region,
            vertexai_credentials=vertexai_credentials,
            gemini_safety_settings=self.gemini_safety_settings,
+            prompt_cache_enabled=self.prompt_cache_enabled,
            litellmrouter_config=router_config,
        )

@@ -689,6 +749,7 @@ class LLMProvider:
        skip_validation: bool = False,
        strict_schema: bool = False,
        return_usage: bool = False,
+        cached_prefix: str | None = None,
    ) -> Any:
        """
        Make an LLM API call with retry logic.
@@ -703,7 +764,10 @@ class LLMProvider:
            initial_backoff: Initial backoff time in seconds.
            max_backoff: Maximum backoff time in seconds.
            skip_validation: Return raw JSON without Pydantic validation.
-            strict_schema: Use strict JSON schema enforcement (OpenAI only). Guarantees all required fields.
+            strict_schema: Per-call override requesting grammar-enforced (json_schema strict)
+                structured output instead of the soft json_object path. The server-level
+                HINDSIGHT_API_LLM_STRICT_SCHEMA flag is OR-ed in here so it applies to every call;
+                providers without a strict mode ignore it.
            return_usage: If True, return tuple (result, TokenUsage) instead of just result.

        Returns:
@@ -723,33 +787,83 @@ class LLMProvider:
        structured = "+structured" if response_format is not None else ""
        set_stage(f"llm.{self.provider}.{scope}{structured}")

-        async with AsyncExitStack() as stack:
-            for sem in _semaphores_for_scope(scope):
-                await stack.enter_async_context(sem)
+        # Resolve strict-schema once, here, rather than in each provider: the
+        # per-call argument OR the server-level HINDSIGHT_API_LLM_STRICT_SCHEMA
+        # flag. Providers with a json_schema response_format (OpenAI-compatible,
+        # LiteLLM) then grammar-enforce structured output instead of the fragile
+        # soft json_object path; Gemini already enforces its native response_schema,
+        # and providers without a strict mode simply ignore the flag.
+        from ..config import get_config

-            # Delegate to provider implementation
-            result = await self._provider_impl.call(
-                messages=messages,
-                response_format=response_format,
+        strict_schema = strict_schema or get_config().llm_strict_schema
+
+        # LLM call observability flows through the OTel GenAI recorder
+        # (tracing.get_span_recorder().record_llm_call). Provider implementations
+        # record successful calls; we forward failures here since they don't.
+        # The requested params are stashed in a contextvar (only what the caller
+        # actually set) so the recorder can attach them to either path.
+        from ..tracing import get_span_recorder
+        from .llm_trace import reset_request_context, set_request_context
+
+        call_start = time.monotonic()
+        request_token = set_request_context(
+            _request_params(
                max_completion_tokens=max_completion_tokens,
                temperature=temperature,
                scope=scope,
-                max_retries=max_retries,
-                initial_backoff=initial_backoff,
-                max_backoff=max_backoff,
-                skip_validation=skip_validation,
-                strict_schema=strict_schema,
-                return_usage=return_usage,
+                response_format=response_format,
            )
+        )
+        try:
+            async with AsyncExitStack() as stack:
+                for sem in _semaphores_for_scope(scope):
+                    await stack.enter_async_context(sem)

-            # Backward compatibility: Update mock call tracking for mock provider
-            # This allows existing tests using LLMProvider._mock_calls to continue working
-            if self.provider == "mock":
-                from .providers.mock_llm import MockLLM
+                # cached_prefix is only set for providers that returned a handle
+                # from get_or_create_cached_prefix() (e.g. Gemini); it's None for
+                # the rest. Forward it only when present so providers that don't
+                # implement caching keep their call() signature untouched.
+                cache_kwarg = {"cached_prefix": cached_prefix} if cached_prefix is not None else {}
+                try:
+                    # Delegate to provider implementation
+                    result = await self._provider_impl.call(
+                        messages=messages,
+                        response_format=response_format,
+                        max_completion_tokens=max_completion_tokens,
+                        temperature=temperature,
+                        scope=scope,
+                        max_retries=max_retries,
+                        initial_backoff=initial_backoff,
+                        max_backoff=max_backoff,
+                        skip_validation=skip_validation,
+                        strict_schema=strict_schema,
+                        return_usage=return_usage,
+                        **cache_kwarg,
+                    )
+                except Exception as e:
+                    get_span_recorder().record_llm_call(
+                        provider=self.provider,
+                        model=self.model,
+                        scope=scope,
+                        messages=messages,
+                        response_content=None,
+                        input_tokens=0,
+                        output_tokens=0,
+                        duration=time.monotonic() - call_start,
+                        error=e,
+                    )
+                    raise

-                if isinstance(self._provider_impl, MockLLM):
-                    # Sync the mock calls from provider implementation to wrapper
-                    self._mock_calls = self._provider_impl.get_mock_calls()
+                # Backward compatibility: Update mock call tracking for mock provider
+                # This allows existing tests using LLMProvider._mock_calls to continue working
+                if self.provider == "mock":
+                    from .providers.mock_llm import MockLLM
+
+                    if isinstance(self._provider_impl, MockLLM):
+                        # Sync the mock calls from provider implementation to wrapper
+                        self._mock_calls = self._provider_impl.get_mock_calls()
+        finally:
+            reset_request_context(request_token)

        return result

@@ -764,6 +878,7 @@ class LLMProvider:
        initial_backoff: float = 1.0,
        max_backoff: float = 30.0,
        tool_choice: str | dict[str, Any] = "auto",
+        cached_prefix: str | None = None,
    ) -> "LLMToolCallResult":
        """
        Make an LLM API call with tool/function calling support.
@@ -786,31 +901,66 @@ class LLMProvider:

        set_stage(f"llm.{self.provider}.{scope}+tools")

-        async with AsyncExitStack() as stack:
-            for sem in _semaphores_for_scope(scope):
-                await stack.enter_async_context(sem)
+        # Failures forwarded to the GenAI recorder; successes recorded by providers.
+        from ..tracing import get_span_recorder
+        from .llm_trace import reset_request_context, set_request_context

-            # Delegate to provider implementation
-            result = await self._provider_impl.call_with_tools(
-                messages=messages,
-                tools=tools,
+        call_start = time.monotonic()
+        request_token = set_request_context(
+            _request_params(
                max_completion_tokens=max_completion_tokens,
                temperature=temperature,
                scope=scope,
-                max_retries=max_retries,
-                initial_backoff=initial_backoff,
-                max_backoff=max_backoff,
                tool_choice=tool_choice,
            )
+        )
+        try:
+            async with AsyncExitStack() as stack:
+                for sem in _semaphores_for_scope(scope):
+                    await stack.enter_async_context(sem)

-            # Backward compatibility: Update mock call tracking for mock provider
-            # This allows existing tests using LLMProvider._mock_calls to continue working
-            if self.provider == "mock":
-                from .providers.mock_llm import MockLLM
+                # cached_prefix is only set for providers that returned a handle
+                # from get_or_create_cached_prefix(); forward it only when present
+                # so non-caching providers keep their signature (same as call()).
+                cache_kwarg = {"cached_prefix": cached_prefix} if cached_prefix is not None else {}
+                try:
+                    # Delegate to provider implementation
+                    result = await self._provider_impl.call_with_tools(
+                        messages=messages,
+                        tools=tools,
+                        max_completion_tokens=max_completion_tokens,
+                        temperature=temperature,
+                        scope=scope,
+                        max_retries=max_retries,
+                        initial_backoff=initial_backoff,
+                        max_backoff=max_backoff,
+                        tool_choice=tool_choice,
+                        **cache_kwarg,
+                    )
+                except Exception as e:
+                    get_span_recorder().record_llm_call(
+                        provider=self.provider,
+                        model=self.model,
+                        scope=scope,
+                        messages=messages,
+                        response_content=None,
+                        input_tokens=0,
+                        output_tokens=0,
+                        duration=time.monotonic() - call_start,
+                        error=e,
+                    )
+                    raise

-                if isinstance(self._provider_impl, MockLLM):
-                    # Sync the mock calls from provider implementation to wrapper
-                    self._mock_calls = self._provider_impl.get_mock_calls()
+                # Backward compatibility: Update mock call tracking for mock provider
+                # This allows existing tests using LLMProvider._mock_calls to continue working
+                if self.provider == "mock":
+                    from .providers.mock_llm import MockLLM
+
+                    if isinstance(self._provider_impl, MockLLM):
+                        # Sync the mock calls from provider implementation to wrapper
+                        self._mock_calls = self._provider_impl.get_mock_calls()
+        finally:
+            reset_request_context(request_token)

        return result

@@ -914,7 +1064,14 @@ class LLMProvider:
        # SDK will automatically check for authentication when first used
        # No need to verify here - let it fail gracefully on first call with helpful error

-    def with_config(self, config: Any) -> "ConfiguredLLMProvider":
+    def with_config(
+        self,
+        config: Any,
+        *,
+        bank_id: str | None = None,
+        operation: str | None = None,
+        metadata: dict[str, Any] | None = None,
+    ) -> "ConfiguredLLMProvider":
        """
        Return a configured wrapper for a specific bank operation.

@@ -924,12 +1081,31 @@ class LLMProvider:

        Args:
            config: Resolved ``HindsightConfig`` for the current bank/request.
+            bank_id: Bank the operation runs for; attributed to LLM trace rows.
+            operation: Logical operation label ("retain", "reflect", ...) for
+                LLM trace rows.
+            metadata: Optional extra caller metadata stored on trace rows.

        Returns:
            A ``ConfiguredLLMProvider`` that delegates to this provider with
            the supplied config applied.
        """
-        return ConfiguredLLMProvider(self, config.llm_gemini_safety_settings)
+        trace_ctx = None
+        if bank_id is not None or operation is not None or metadata:
+            from .llm_trace import LLMTraceContext
+
+            # One trace + operation span per with_config() call — i.e. per
+            # operation invocation. Every LLM call made through this wrapper
+            # shares them, so a reflect/retain/consolidation run groups its
+            # calls as parent (operation) → children (LLM calls).
+            trace_ctx = LLMTraceContext(
+                bank_id=bank_id,
+                operation=operation,
+                metadata=dict(metadata or {}),
+                trace_id=str(uuid.uuid4()),
+                operation_span_id=str(uuid.uuid4()),
+            )
+        return ConfiguredLLMProvider(self, config.llm_gemini_safety_settings, trace_ctx)

    async def cleanup(self) -> None:
        """Clean up resources (e.g. stop llamacpp subprocess)."""
@@ -993,10 +1169,16 @@ class ConfiguredLLMProvider:
    any changes.
    """

-    def __init__(self, provider: "LLMProvider", gemini_safety_settings: list | None) -> None:
+    def __init__(
+        self,
+        provider: "LLMProvider",
+        gemini_safety_settings: list | None,
+        trace_ctx: Any | None = None,
+    ) -> None:
        # Use object.__setattr__ to avoid triggering __getattr__
        object.__setattr__(self, "_provider", provider)
        object.__setattr__(self, "_gemini_safety_settings", gemini_safety_settings)
+        object.__setattr__(self, "_trace_ctx", trace_ctx)

    # ── attribute passthrough ──────────────────────────────────────────────────

@@ -1009,10 +1191,12 @@ class ConfiguredLLMProvider:
        from .providers.gemini_llm import _safety_settings_ctx

        token = _safety_settings_ctx.set(object.__getattribute__(self, "_gemini_safety_settings"))
+        trace_token = self._bind_trace_context()
        try:
            return await object.__getattribute__(self, "_provider").call(messages=messages, **kwargs)
        finally:
            _safety_settings_ctx.reset(token)
+            self._reset_trace_context(trace_token)

    async def call_with_tools(
        self,
@@ -1023,12 +1207,38 @@ class ConfiguredLLMProvider:
        from .providers.gemini_llm import _safety_settings_ctx

        token = _safety_settings_ctx.set(object.__getattribute__(self, "_gemini_safety_settings"))
+        trace_token = self._bind_trace_context()
        try:
            return await object.__getattribute__(self, "_provider").call_with_tools(
                messages=messages, tools=tools, **kwargs
            )
        finally:
            _safety_settings_ctx.reset(token)
+            self._reset_trace_context(trace_token)
+
+    def trace_context(self) -> Any | None:
+        """The operation-level LLM trace context (or None when untraced).
+
+        Lets the engine attach the operation's produced/consumed memory_ids to
+        this run's trace rows once they're known (after the LLM calls).
+        """
+        return object.__getattribute__(self, "_trace_ctx")
+
+    def _bind_trace_context(self) -> Any | None:
+        """Bind bank/operation attribution for the duration of one call."""
+        trace_ctx = object.__getattribute__(self, "_trace_ctx")
+        if trace_ctx is None:
+            return None
+        from .llm_trace import set_trace_context
+
+        return set_trace_context(trace_ctx)
+
+    def _reset_trace_context(self, trace_token: Any | None) -> None:
+        if trace_token is None:
+            return
+        from .llm_trace import reset_trace_context
+
+        reset_trace_context(trace_token)


 # Backwards compatibility alias
@@ -0,0 +1,214 @@
+"""Background maintenance loop.
+
+A single periodic loop that drives all of Hindsight's recurring housekeeping
+from one place, so we don't spawn a separate ``asyncio`` task per concern:
+
+- **Retention sweeps** (hourly): delete ``audit_log`` and ``llm_requests`` rows
+  older than their configured retention, across *all* tenant schemas.
+- **Consolidation reconcile** (configurable, default 5 min): re-schedule
+  consolidation for banks that have eligible-but-unscheduled facts and no
+  in-flight consolidation. This recovers facts that were stranded when a
+  consolidation operation failed terminally and left them with
+  ``consolidated_at IS NULL AND consolidation_failed_at IS NULL`` and nothing to
+  re-trigger them.
+
+The loop wakes on a short fixed tick and runs each job when its own
+``last_run + interval`` is due (run-at-start, then on interval), so adding jobs
+with different cadences doesn't burst CPU. Cross-tenant discovery goes through
+server-side PL/pgSQL routines (``public.schemas_with_expired_rows`` and
+``public.banks_needing_consolidation``) — one round-trip each — instead of a
+per-schema query storm, which matters at thousands of tenants.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from typing import TYPE_CHECKING
+
+from ..config import HindsightConfig, get_config
+from ..models import RequestContext
+from .db_utils import acquire_with_retry
+from .schema import _is_oracle
+
+if TYPE_CHECKING:
+    from .memory_engine import MemoryEngine
+
+logger = logging.getLogger(__name__)
+
+# Short tick so jobs with different cadences share one loop without per-job tasks.
+_TICK_SECONDS = 60
+# Retention sweeps are not time-sensitive; hourly matches the previous per-sweep cadence.
+_RETENTION_INTERVAL_SECONDS = 3600
+
+
+class MaintenanceLoop:
+    """Owns the single periodic maintenance task for a :class:`MemoryEngine`."""
+
+    def __init__(self, engine: "MemoryEngine") -> None:
+        self._engine = engine
+        self._task: asyncio.Task | None = None
+        self._stop = asyncio.Event()
+        # Monotonic timestamps of the last run per job, keyed by job name.
+        self._last_run: dict[str, float] = {}
+
+    # ── lifecycle ──────────────────────────────────────────────────────────
+
+    def start(self) -> None:
+        """Start the loop if any maintenance job is enabled. Idempotent."""
+        if self._task and not self._task.done():
+            return
+        # PostgreSQL-only: the retention sweeps target PG-only tables (audit_log,
+        # llm_requests) and the reconcile relies on PG-only PL/pgSQL routines
+        # installed by the maintenance-routines migration. Oracle support is
+        # intentionally absent (mirrors that PG-only migration).
+        if _is_oracle():
+            logger.debug("Maintenance loop not started: PostgreSQL-only")
+            return
+        if not self._any_job_enabled():
+            logger.debug("Maintenance loop not started: no jobs enabled")
+            return
+        self._stop.clear()
+        try:
+            self._task = asyncio.create_task(self._run())
+        except RuntimeError:
+            logger.debug("Cannot start maintenance loop: no running event loop")
+
+    async def stop(self) -> None:
+        """Stop the loop and wait for the current tick to finish."""
+        self._stop.set()
+        if self._task and not self._task.done():
+            try:
+                await self._task
+            except asyncio.CancelledError:
+                pass
+        self._task = None
+
+    @staticmethod
+    def _any_job_enabled() -> bool:
+        cfg = get_config()
+        reconcile_on = cfg.consolidation_reconcile_interval_seconds > 0
+        audit_on = cfg.audit_log_enabled and cfg.audit_log_retention_days > 0
+        llm_on = cfg.llm_trace_enabled and cfg.llm_trace_retention_days > 0
+        return reconcile_on or audit_on or llm_on
+
+    # ── loop ───────────────────────────────────────────────────────────────
+
+    async def _run(self) -> None:
+        while not self._stop.is_set():
+            try:
+                await self._tick()
+            except Exception:
+                logger.exception("Maintenance tick failed")
+            try:
+                await asyncio.wait_for(self._stop.wait(), timeout=_TICK_SECONDS)
+            except asyncio.TimeoutError:
+                pass
+
+    def _is_due(self, job: str, interval_seconds: int) -> bool:
+        """True if ``job`` has never run or its interval has elapsed; marks it run now."""
+        now = time.monotonic()
+        last = self._last_run.get(job)
+        if last is not None and (now - last) < interval_seconds:
+            return False
+        self._last_run[job] = now
+        return True
+
+    async def _tick(self) -> None:
+        cfg = get_config()
+        if self._is_due("retention", _RETENTION_INTERVAL_SECONDS):
+            await self._run_retention(cfg)
+        interval = cfg.consolidation_reconcile_interval_seconds
+        if interval > 0 and self._is_due("reconcile", interval):
+            await self._run_reconcile()
+
+    # ── retention ──────────────────────────────────────────────────────────
+
+    async def _run_retention(self, cfg: HindsightConfig) -> None:
+        # Retention days are static server-level config, so one global cutoff
+        # applies to every tenant schema (the routine sweeps them all).
+        if cfg.audit_log_enabled and cfg.audit_log_retention_days > 0:
+            await self._purge_expired("audit_log", "started_at", cfg.audit_log_retention_days)
+        if cfg.llm_trace_enabled and cfg.llm_trace_retention_days > 0:
+            await self._purge_expired("llm_requests", "started_at", cfg.llm_trace_retention_days)
+
+    async def _purge_expired(self, table: str, ts_col: str, days: int) -> None:
+        """Delete rows older than ``days`` from ``table`` across every tenant schema."""
+        backend = self._engine._backend
+        try:
+            async with acquire_with_retry(backend, max_retries=1) as conn:
+                rows = await conn.fetch(
+                    "SELECT * FROM public.schemas_with_expired_rows($1, $2, $3)", table, ts_col, days
+                )
+                for row in rows:
+                    schema = row[0]
+                    # schema names come from pg_class; quote defensively all the same.
+                    qschema = '"' + schema.replace('"', '""') + '"'
+                    result = await conn.execute(
+                        f"DELETE FROM {qschema}.{table} WHERE {ts_col} < NOW() - make_interval(days => $1)",
+                        days,
+                    )
+                    if result and result != "DELETE 0":
+                        logger.info(f"Retention sweep {schema}.{table}: {result}")
+        except Exception as e:
+            logger.warning(f"Retention sweep failed for {table}: {e}")
+
+    # ── consolidation reconcile ──────────────────────────────────────────────
+
+    async def _run_reconcile(self) -> None:
+        """Re-schedule consolidation for banks with eligible-but-unscheduled facts."""
+        engine = self._engine
+        try:
+            async with acquire_with_retry(engine._backend, max_retries=1) as conn:
+                rows = await conn.fetch("SELECT schema_name, bank_id FROM public.banks_needing_consolidation()")
+        except Exception as e:
+            logger.warning(f"Consolidation reconcile discovery failed: {e}")
+            return
+        if not rows:
+            return
+
+        # Only enqueue into schemas the worker actually polls (tenant discovery),
+        # otherwise the op would never be claimed and would block future reconciles
+        # for that bank. The tenant_id (when the extension provides one) lets
+        # config resolution honor tenant-level overrides.
+        try:
+            tenants = await engine._tenant_extension.list_tenants()
+        except Exception as e:
+            logger.warning(f"Consolidation reconcile tenant discovery failed: {e}")
+            return
+        tenant_by_schema = {t.schema: t for t in tenants}
+        default_schema = get_config().database_schema
+
+        from .memory_engine import _current_schema
+
+        submitted = 0
+        skipped_unknown = 0
+        for row in rows:
+            schema = row["schema_name"]
+            bank_id = row["bank_id"]
+            tenant = tenant_by_schema.get(schema)
+            if tenant is None and schema != default_schema:
+                skipped_unknown += 1
+                continue
+            tenant_id = tenant.tenant_id if tenant else None
+            token = _current_schema.set(schema)
+            try:
+                context = RequestContext(internal=True, tenant_id=tenant_id)
+                resolved = await engine._config_resolver.resolve_full_config(bank_id, context)
+                # Mirror the retain-time auto-consolidation gate (memory_engine): both
+                # observations and auto-consolidation must be enabled for this bank.
+                if not (resolved.enable_observations and resolved.enable_auto_consolidation):
+                    continue
+                await engine.submit_async_consolidation(bank_id=bank_id, request_context=context)
+                submitted += 1
+            except Exception as e:
+                logger.warning(f"Consolidation reconcile failed for bank {bank_id} in {schema}: {e}")
+            finally:
+                _current_schema.reset(token)
+
+        if submitted or skipped_unknown:
+            logger.info(
+                f"Consolidation reconcile: scheduled {submitted} bank(s)"
+                + (f", skipped {skipped_unknown} in unrecognized schema(s)" if skipped_unknown else "")
+            )
@@ -5,8 +5,10 @@ These dataclasses define the structure of result_metadata for different operatio
 The metadata is exposed in the API for debugging purposes and may change without notice.
 """

-from dataclasses import asdict, dataclass
-from typing import Any
+from dataclasses import asdict, dataclass, field
+from typing import Any, Mapping
+
+MAX_EXTRACTION_ERROR_SAMPLES = 5


@dataclass
@@ -48,6 +50,79 @@ class RetainMetadata:
        return asdict(self)


+@dataclass
+class RetainExtractionErrors:
+    """Non-fatal fact extraction failures observed inside one retain operation."""
+
+    count: int = 0
+    sample: list[str] = field(default_factory=list)
+
+    def add(self, message: str) -> None:
+        """Record one extraction error while keeping the stored sample bounded."""
+        self.count += 1
+        if len(self.sample) < MAX_EXTRACTION_ERROR_SAMPLES:
+            self.sample.append(message[:500])
+
+    def merge_metadata(self, metadata: Mapping[str, Any]) -> None:
+        """Merge errors already present on an operation result_metadata object."""
+        self.count += int(metadata.get("extraction_errors_count") or 0)
+
+        sample = metadata.get("extraction_errors_sample") or []
+        if isinstance(sample, str):
+            sample = [sample]
+        if isinstance(sample, list):
+            for entry in sample:
+                if isinstance(entry, str) and len(self.sample) < MAX_EXTRACTION_ERROR_SAMPLES:
+                    self.sample.append(entry[:500])
+
+    def to_dict(self) -> dict[str, Any]:
+        """Convert to the public result_metadata field shape."""
+        data: dict[str, Any] = {"extraction_errors_count": self.count}
+        if self.sample:
+            data["extraction_errors_sample"] = self.sample
+        return data
+
+
+@dataclass
+class RetainOutcomeMetadata:
+    """Machine-readable outcome metadata for a completed retain operation."""
+
+    unit_ids_count: int
+    extraction_errors_count: int = 0
+    extraction_errors_sample: list[str] = field(default_factory=list)
+
+    def to_dict(self) -> dict[str, Any]:
+        """Convert to dict for JSON serialization, omitting empty optional samples."""
+        data: dict[str, Any] = {
+            "unit_ids_count": self.unit_ids_count,
+            "extraction_errors_count": self.extraction_errors_count,
+        }
+        if self.extraction_errors_sample:
+            data["extraction_errors_sample"] = self.extraction_errors_sample[:MAX_EXTRACTION_ERROR_SAMPLES]
+        return data
+
+
+@dataclass
+class RetainOutcomeAggregate:
+    """Aggregate retain outcome metadata from child retain operations."""
+
+    unit_ids_count: int = 0
+    extraction_errors: RetainExtractionErrors = field(default_factory=RetainExtractionErrors)
+
+    def add_metadata(self, metadata: Mapping[str, Any]) -> None:
+        """Fold one child operation's result_metadata into the aggregate."""
+        self.unit_ids_count += int(metadata.get("unit_ids_count") or 0)
+        self.extraction_errors.merge_metadata(metadata)
+
+    def to_outcome_metadata(self) -> RetainOutcomeMetadata:
+        """Return the aggregate in the public result_metadata field shape."""
+        return RetainOutcomeMetadata(
+            unit_ids_count=self.unit_ids_count,
+            extraction_errors_count=self.extraction_errors.count,
+            extraction_errors_sample=self.extraction_errors.sample,
+        )
+
+
@dataclass
 class ConsolidationMetadata:
    """Metadata for consolidation operations."""
@@ -38,6 +38,7 @@ class AnthropicLLM(LLMInterface):
        reasoning_effort: str = "low",
        timeout: float = 300.0,
        default_headers: dict[str, str] | None = None,
+        extra_body: dict[str, Any] | None = None,
        **kwargs: Any,
    ):
        """
@@ -54,6 +55,10 @@ class AnthropicLLM(LLMInterface):
                the Anthropic SDK client. Used by operators routing through proxies
                or request-tracing middleware. Sourced from ``llm_default_headers`` in
                ``HindsightConfig`` (env: ``HINDSIGHT_API_LLM_DEFAULT_HEADERS``).
+            extra_body: Extra request-body params (e.g. ``{"temperature": 0.2,
+                "top_p": 0.9, "top_k": 40}``) passed via the Anthropic SDK's
+                ``extra_body`` so they merge into the JSON sent to the Messages API.
+                Sourced from ``llm_extra_body`` (env: ``HINDSIGHT_API_LLM_EXTRA_BODY``).
            **kwargs: Additional provider-specific parameters.
        """
        super().__init__(provider, api_key, base_url, model, reasoning_effort, **kwargs)
@@ -61,6 +66,9 @@ class AnthropicLLM(LLMInterface):
        if not self.api_key:
            raise ValueError("API key is required for Anthropic provider")

+        # User-configured extra body params (merged into every Messages API call)
+        self._extra_body = extra_body or {}
+
        # Import and initialize Anthropic client
        try:
            from anthropic import AsyncAnthropic
@@ -178,6 +186,9 @@ class AnthropicLLM(LLMInterface):
        if system_prompt:
            call_params["system"] = system_prompt

+        if self._extra_body:
+            call_params["extra_body"] = self._extra_body
+
        last_exception = None

        for attempt in range(max_retries + 1):
@@ -216,6 +227,7 @@ class AnthropicLLM(LLMInterface):
                input_tokens = response.usage.input_tokens or 0 if response.usage else 0
                output_tokens = response.usage.output_tokens or 0 if response.usage else 0
                total_tokens = input_tokens + output_tokens
+                cached_tokens = getattr(response.usage, "cache_read_input_tokens", 0) or 0 if response.usage else 0

                # Record LLM metrics
                metrics = get_metrics_collector()
@@ -245,6 +257,7 @@ class AnthropicLLM(LLMInterface):
                    duration=duration,
                    finish_reason=finish_reason,
                    error=None,
+                    cached_tokens=cached_tokens,
                )

                # Log slow calls
@@ -260,6 +273,7 @@ class AnthropicLLM(LLMInterface):
                        input_tokens=input_tokens,
                        output_tokens=output_tokens,
                        total_tokens=total_tokens,
+                        cached_tokens=cached_tokens,
                    )
                    return result, token_usage
                return result
@@ -394,6 +408,9 @@ class AnthropicLLM(LLMInterface):
        if system_prompt:
            call_params["system"] = system_prompt

+        if self._extra_body:
+            call_params["extra_body"] = self._extra_body
+
        last_exception = None
        for attempt in range(max_retries + 1):
            try:
@@ -0,0 +1,316 @@
+"""Gemini context-cache manager.
+
+Wraps the ``google-genai`` SDK's CachedContent API to let callers reuse a
+stable system_instruction + response_schema prefix across many requests.
+
+Cached input tokens are billed at ~10× lower than fresh input tokens
+(check the current Gemini pricing for the exact ratio per model), so for
+workloads that repeatedly send a large fixed prefix with a small variable
+user message — fact extraction, structured tagging, classification — the
+input-cost savings are substantial.
+
+This module owns only the create/refresh/lookup lifecycle. It is up to
+the caller to (a) decide that the prefix is stable enough to cache, and
+(b) pass the returned cache name to ``GeminiLLM.call()``. When the
+returned name is ``None`` (because Gemini rejected the create — most
+commonly because the prefix is smaller than the model's minimum), the
+caller MUST fall back to a non-cached call.
+
+Cardinality
+-----------
+The intended cache count per process is small (≲100 entries). Each
+entry corresponds to one combination of (model, system_instruction,
+response_schema). If a caller sees the cache grow unboundedly it
+indicates the system_instruction contains per-request data that should
+move into the user message instead.
+
+TTL
+---
+Gemini's CachedContent has a TTL bounded by the model (currently 1h
+for most generally-available models). This manager refreshes proactively
+at ``ttl_safety_margin`` before expiry. If a cached entry has expired
+between refreshes the next call will recreate it transparently.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import hashlib
+import json
+import logging
+import time
+from dataclasses import dataclass
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+# Default TTL: 55 minutes. Gemini's hard max for CachedContent is 1 hour
+# for most models; we refresh 5 minutes early so a request landing right
+# at the boundary doesn't race against expiry.
+_DEFAULT_TTL_SECONDS = 55 * 60
+_DEFAULT_REFRESH_MARGIN_SECONDS = 5 * 60
+# Cap on the cache-create network call. It runs while holding the manager lock, so
+# a hung create would block every concurrent caller (e.g. all chunks of a 10-chunk
+# retain batch waiting on the cold-start create). On timeout the create soft-fails
+# to None and callers proceed uncached, rather than stalling the whole batch.
+_DEFAULT_CREATE_TIMEOUT_SECONDS = 30.0
+
+
+@dataclass
+class _CacheEntry:
+    name: str  # The CachedContent resource name returned by Gemini.
+    created_at: float
+    ttl_seconds: int
+
+
+class GeminiCacheManager:
+    """Per-process map of (prefix fingerprint) → CachedContent name.
+
+    Thread-safe across asyncio tasks via a single ``asyncio.Lock``. The
+    create/refresh calls are serialised; this is fine because cache
+    creation is a one-shot warm-up per fingerprint (subsequent reads are
+    pure dict lookups outside the lock).
+
+    Not shared across pods — each worker / api replica builds its own
+    cache. The cost of cold-starting one extra full-price call per pod
+    per fingerprint per hour is negligible compared to the steady-state
+    savings.
+    """
+
+    def __init__(
+        self,
+        client: Any,
+        *,
+        ttl_seconds: int = _DEFAULT_TTL_SECONDS,
+        refresh_margin_seconds: int = _DEFAULT_REFRESH_MARGIN_SECONDS,
+        create_timeout_seconds: float = _DEFAULT_CREATE_TIMEOUT_SECONDS,
+    ) -> None:
+        self._client = client
+        self._ttl_seconds = ttl_seconds
+        self._refresh_margin_seconds = refresh_margin_seconds
+        self._create_timeout_seconds = create_timeout_seconds
+        self._entries: dict[str, _CacheEntry] = {}
+        self._lock = asyncio.Lock()
+
+    @staticmethod
+    def fingerprint(
+        model: str,
+        system_instruction: str,
+        response_schema: Any | None = None,
+        tools: list[dict[str, Any]] | None = None,
+    ) -> str:
+        """Stable hash of the cacheable surface.
+
+        ``response_schema`` may be a Pydantic class, a dict, or ``None``.
+        Pydantic schemas are normalised by serialising via
+        ``model_json_schema()`` and stripping the auto-generated
+        ``"title"`` fields so two dynamically-built models with the same
+        shape but different class names hash identically. This matters
+        for callers (e.g. fact extraction) that rebuild the schema
+        class on every request via a builder helper — without the
+        normalisation the cache would never hit.
+
+        ``tools`` is the OpenAI-style tools list (each entry has a
+        ``"function"`` dict with name/description/parameters). When
+        supplied, the tool definitions become part of the cache key so a
+        loop that adds or renames a tool gets a fresh cache and doesn't
+        silently use a stale schema. Tools are serialised with
+        ``sort_keys=True`` to neutralise dict-ordering drift.
+        """
+        hasher = hashlib.sha256()
+        hasher.update(model.encode("utf-8"))
+        hasher.update(b"\x00")
+        hasher.update(system_instruction.encode("utf-8"))
+        hasher.update(b"\x00")
+        if response_schema is None:
+            hasher.update(b"none")
+        elif hasattr(response_schema, "model_json_schema"):
+            try:
+                schema = response_schema.model_json_schema()
+                _strip_titles(schema)
+                hasher.update(json.dumps(schema, sort_keys=True).encode("utf-8"))
+            except Exception:
+                # Fall back to class identity if the schema can't be serialised.
+                hasher.update(repr(response_schema).encode("utf-8"))
+        else:
+            try:
+                hasher.update(json.dumps(response_schema, sort_keys=True).encode("utf-8"))
+            except (TypeError, ValueError):
+                hasher.update(repr(response_schema).encode("utf-8"))
+        hasher.update(b"\x00")
+        if tools:
+            try:
+                hasher.update(json.dumps(tools, sort_keys=True).encode("utf-8"))
+            except (TypeError, ValueError):
+                hasher.update(repr(tools).encode("utf-8"))
+        else:
+            hasher.update(b"no-tools")
+        return hasher.hexdigest()
+
+    async def get_or_create(
+        self,
+        *,
+        model: str,
+        system_instruction: str,
+        response_schema: Any | None = None,
+        tools: list[dict[str, Any]] | None = None,
+    ) -> str | None:
+        """Return a CachedContent resource name for the given prefix, or
+        ``None`` if Gemini rejects the create (prefix too small, model
+        does not support caching, etc.).
+
+        ``tools`` is the OpenAI-style tools list. When supplied, the tool
+        definitions are baked into the CachedContent so the caller's
+        ``call_with_tools`` doesn't need to resend them on every
+        iteration. Pass ``None`` for non-tool calls.
+
+        ``None`` return is a normal, expected value — the caller falls
+        back to an uncached call and the system continues to work.
+        """
+        key = self.fingerprint(model, system_instruction, response_schema, tools)
+
+        async with self._lock:
+            entry = self._entries.get(key)
+            if entry is not None and self._is_fresh(entry):
+                return entry.name
+
+            # Need to (re)create. Pop the stale entry first so a failed
+            # create doesn't leave a name we'd return on the next call.
+            self._entries.pop(key, None)
+
+            try:
+                cache_name = await self._create_cache(
+                    model=model,
+                    system_instruction=system_instruction,
+                    tools=tools,
+                )
+            except _CacheNotEligible as e:
+                logger.debug(
+                    "GeminiCacheManager: prefix not eligible for caching (model=%s, reason=%s) — caller will fall back",
+                    model,
+                    e,
+                )
+                return None
+            except Exception:
+                logger.exception(
+                    "GeminiCacheManager: failed to create cached content "
+                    "(model=%s); caller will fall back to uncached call",
+                    model,
+                )
+                return None
+
+            if cache_name is None:
+                return None
+
+            self._entries[key] = _CacheEntry(
+                name=cache_name,
+                created_at=time.monotonic(),
+                ttl_seconds=self._ttl_seconds,
+            )
+            return cache_name
+
+    def _is_fresh(self, entry: _CacheEntry) -> bool:
+        """An entry is fresh if it's young enough that the next request
+        won't race against the TTL expiry."""
+        age = time.monotonic() - entry.created_at
+        return age < (entry.ttl_seconds - self._refresh_margin_seconds)
+
+    def invalidate(self, name: str) -> None:
+        """Forget a cache name that the server rejected (expired/deleted/invalid).
+
+        Called by the provider when a generate request using this CachedContent
+        fails, so the next ``get_or_create`` recreates it instead of handing back
+        the dead name again. Best-effort and sync — drops the matching entry from
+        the in-process map; the orphaned server-side cache (if any) ages out on
+        its own TTL.
+        """
+        for key, entry in list(self._entries.items()):
+            if entry.name == name:
+                self._entries.pop(key, None)
+
+    async def _create_cache(
+        self,
+        *,
+        model: str,
+        system_instruction: str,
+        tools: list[dict[str, Any]] | None = None,
+    ) -> str | None:
+        """Wrap ``client.aio.caches.create`` with the config we want.
+
+        The SDK surface differs slightly across google-genai versions;
+        this implementation targets the >=1.0.0 line where caches live
+        under ``client.aio.caches``.
+        """
+        # Lazy import so this module doesn't require the SDK at import time.
+        from google.genai import types as genai_types
+
+        # A CachedContent only holds reusable *input* — system_instruction,
+        # contents, tools, ttl. ``response_schema``/``response_mime_type`` are
+        # generation-time output constraints and the SDK rejects them here
+        # (``CreateCachedContentConfig`` forbids those fields). They are applied
+        # per-request on the GenerateContentConfig instead — see the call sites,
+        # which set them alongside ``cached_content``. ``response_schema`` is
+        # still part of the fingerprint so a schema change keys a fresh cache.
+        config_kwargs: dict[str, Any] = {
+            "system_instruction": system_instruction,
+            "ttl": f"{self._ttl_seconds}s",
+        }
+        if tools:
+            # OpenAI-style {"function": {...}} entries must be converted to
+            # Gemini's Tool/FunctionDeclaration shape before caching.
+            gemini_tools = []
+            for tool in tools:
+                func = tool.get("function", {})
+                gemini_tools.append(
+                    genai_types.Tool(
+                        function_declarations=[
+                            genai_types.FunctionDeclaration(
+                                name=func.get("name", ""),
+                                description=func.get("description", ""),
+                                parameters=func.get("parameters"),
+                            )
+                        ]
+                    )
+                )
+            config_kwargs["tools"] = gemini_tools
+
+        try:
+            cached = await asyncio.wait_for(
+                self._client.aio.caches.create(
+                    model=model,
+                    config=genai_types.CreateCachedContentConfig(**config_kwargs),
+                ),
+                timeout=self._create_timeout_seconds,
+            )
+        except Exception as e:
+            # Gemini returns a 400 with a "minimum token count" message
+            # when the prefix is too small. We treat this as a soft
+            # "not eligible" signal rather than a real error so callers
+            # silently fall back to non-cached.
+            msg = str(e).lower()
+            if "minimum" in msg or "too small" in msg or "too short" in msg:
+                raise _CacheNotEligible(str(e)) from e
+            raise
+
+        return getattr(cached, "name", None)
+
+
+class _CacheNotEligible(Exception):
+    """Raised when Gemini rejects the cache create because the prefix
+    is below the model's minimum cacheable size. Treated as a soft
+    fallback by the caller, not an error."""
+
+
+def _strip_titles(node: Any) -> None:
+    """Recursively remove auto-generated ``"title"`` keys from a JSON
+    Schema-like dict tree, in place. Pydantic seeds these from the
+    Python class name, which means structurally-identical schemas built
+    from differently-named classes look distinct to a naive hash."""
+    if isinstance(node, dict):
+        node.pop("title", None)
+        for v in node.values():
+            _strip_titles(v)
+    elif isinstance(node, list):
+        for item in node:
+            _strip_titles(item)
@@ -70,6 +70,22 @@ class GeminiLLM(LLMInterface):
        # Safety settings: None means use Gemini's defaults
        self._safety_settings: list | None = kwargs.get("gemini_safety_settings")

+        # User-configured extra params merged into the GenerateContentConfig of
+        # every call. Gemini's request body nests generation params, so we expose
+        # them in the SDK's native config space rather than as a raw body merge:
+        # keys must be GenerateContentConfig fields (e.g. temperature, top_p,
+        # top_k, max_output_tokens, seed). Sourced from llm_extra_body
+        # (env: HINDSIGHT_API_LLM_EXTRA_BODY).
+        self._extra_body: dict[str, Any] = kwargs.get("extra_body") or {}
+
+        # Context-cache manager. Lazy-initialized on first cache lookup so
+        # nothing happens for models/workloads that never reach it. The instance
+        # default here is off (a directly-constructed GeminiLLM doesn't cache); the
+        # server-level default is on and flows in via the prompt_cache_enabled kwarg
+        # resolved from config in LLMProvider.
+        self._cache_manager: Any | None = None
+        self._prompt_cache_enabled: bool = bool(kwargs.get("prompt_cache_enabled", False))
+
        if self._is_vertexai:
            self._init_vertexai(**kwargs)
        else:
@@ -168,6 +184,7 @@ class GeminiLLM(LLMInterface):
        skip_validation: bool = False,
        strict_schema: bool = False,
        return_usage: bool = False,
+        cached_prefix: str | None = None,
    ) -> Any:
        """
        Make a Gemini/VertexAI API call with retry logic.
@@ -182,8 +199,17 @@ class GeminiLLM(LLMInterface):
            initial_backoff: Initial backoff time in seconds.
            max_backoff: Maximum backoff time in seconds.
            skip_validation: Return raw JSON without Pydantic validation.
-            strict_schema: Use strict JSON schema enforcement (not supported by Gemini).
+            strict_schema: Ignored — Gemini always grammar-enforces structured output via its
+                native response_schema, so it is strict regardless of this flag.
            return_usage: If True, return tuple (result, TokenUsage).
+            cached_prefix: Optional CachedContent resource name (from
+                ``GeminiCacheManager.get_or_create``). When set, the
+                system_instruction is assumed to live in the cache; this call
+                skips resending it and the cached prefix is billed at the
+                cached-input rate instead of the standard input rate. The
+                response_schema is still sent per-request (it is not cacheable).
+                Pass ``None`` to use the
+                normal uncached path.

        Returns:
            If return_usage=False: Parsed response if response_format provided, else text.
@@ -191,9 +217,14 @@ class GeminiLLM(LLMInterface):
        """
        start_time = time.time()

-        # Convert OpenAI-style messages to Gemini format
+        # Convert OpenAI-style messages to Gemini format. We ALWAYS build
+        # system_instruction (even when a cache is in use): the config builder
+        # below omits it from the request while the cache carries the prefix, but
+        # it must be available so the cached-call-failed safety net can re-send it
+        # inline. Whether it's actually sent is decided in _build_generation_config.
        system_instruction = None
        gemini_contents = []
+        using_cache = cached_prefix is not None

        for msg in messages:
            role = msg.get("role", "user")
@@ -209,7 +240,9 @@ class GeminiLLM(LLMInterface):
            else:
                gemini_contents.append(genai_types.Content(role="user", parts=[genai_types.Part(text=content)]))

-        # Add JSON schema instruction if response_format is provided
+        # Add the JSON schema as a textual hint in the system_instruction (matching
+        # the normal uncached path). Structured output is still enforced via
+        # response_schema regardless; this is just guidance text.
        if response_format is not None and hasattr(response_format, "model_json_schema"):
            schema = response_format.model_json_schema()
            schema_msg = f"\n\nYou must respond with valid JSON matching this schema:\n{json.dumps(schema, indent=2, ensure_ascii=False)}"
@@ -218,32 +251,44 @@ class GeminiLLM(LLMInterface):
            else:
                system_instruction = schema_msg

-        # Build generation config
-        config_kwargs: dict[str, Any] = {}
-        if system_instruction:
-            config_kwargs["system_instruction"] = system_instruction
-        if response_format is not None:
-            config_kwargs["response_mime_type"] = "application/json"
-            config_kwargs["response_schema"] = response_format
-        if temperature is not None:
-            config_kwargs["temperature"] = temperature
-        # Gemini's equivalent of OpenAI-style max_completion_tokens is max_output_tokens.
-        # Without it the model can produce arbitrarily long responses, ignoring the
-        # caller's intended cap (e.g. mental_models max_tokens during refresh).
-        if max_completion_tokens is not None:
-            config_kwargs["max_output_tokens"] = max_completion_tokens
-
        # Apply safety settings: context var (per-request bank override) takes precedence over instance default
        effective_safety_settings = _safety_settings_ctx.get()
        if effective_safety_settings is None:
            effective_safety_settings = self._safety_settings
-        if effective_safety_settings is not None:
-            config_kwargs["safety_settings"] = [
-                genai_types.SafetySetting(category=s["category"], threshold=s["threshold"])
-                for s in effective_safety_settings
-            ]

-        generation_config = genai_types.GenerateContentConfig(**config_kwargs) if config_kwargs else None
+        # Build generation config. ``cached_content`` and ``system_instruction``
+        # are mutually exclusive (the cache IS the prefix; the SDK rejects
+        # re-sending it). ``response_schema``/``response_mime_type`` are
+        # request-level output constraints — NOT cacheable — so they're set on
+        # every structured call, including cached ones where they ride alongside
+        # ``cached_content``. Built as a closure so we can rebuild it WITHOUT the
+        # cache and retry inline if a stale/invalid CachedContent makes the call fail.
+        def _build_generation_config(use_cache: bool) -> "genai_types.GenerateContentConfig | None":
+            # Seed with user-configured extra params; explicit settings below win.
+            config_kwargs: dict[str, Any] = dict(self._extra_body)
+            if use_cache:
+                config_kwargs["cached_content"] = cached_prefix
+            elif system_instruction:
+                config_kwargs["system_instruction"] = system_instruction
+            if response_format is not None:
+                config_kwargs["response_mime_type"] = "application/json"
+                config_kwargs["response_schema"] = response_format
+            if temperature is not None:
+                config_kwargs["temperature"] = temperature
+            # Gemini's equivalent of OpenAI-style max_completion_tokens is max_output_tokens.
+            # Without it the model can produce arbitrarily long responses, ignoring the
+            # caller's intended cap (e.g. mental_models max_tokens during refresh).
+            if max_completion_tokens is not None:
+                config_kwargs["max_output_tokens"] = max_completion_tokens
+            if effective_safety_settings is not None:
+                config_kwargs["safety_settings"] = [
+                    genai_types.SafetySetting(category=s["category"], threshold=s["threshold"])
+                    for s in effective_safety_settings
+                ]
+            return genai_types.GenerateContentConfig(**config_kwargs) if config_kwargs else None
+
+        cache_active = using_cache
+        generation_config = _build_generation_config(cache_active)

        last_exception = None

@@ -288,13 +333,24 @@ class GeminiLLM(LLMInterface):
                else:
                    result = content

-                # Extract token usage
+                # Extract token usage. ``cached_content_token_count`` and
+                # ``thoughts_token_count`` are populated on the Gemini 2.5+
+                # family; treat missing fields as 0 so older models still
+                # record sensible metrics.
                input_tokens = 0
                output_tokens = 0
+                cached_input_tokens = 0
+                thoughts_tokens = 0
+                cached_tokens = 0
                if hasattr(response, "usage_metadata") and response.usage_metadata:
                    usage = response.usage_metadata
                    input_tokens = usage.prompt_token_count or 0
                    output_tokens = usage.candidates_token_count or 0
+                    cached_input_tokens = getattr(usage, "cached_content_token_count", 0) or 0
+                    thoughts_tokens = getattr(usage, "thoughts_token_count", 0) or 0
+                    # Tracing/TokenUsage consume ``cached_tokens``; metrics consume
+                    # ``cached_input_tokens`` — same value, two downstream names.
+                    cached_tokens = cached_input_tokens

                # Record metrics
                duration = time.time() - start_time
@@ -307,6 +363,8 @@ class GeminiLLM(LLMInterface):
                    input_tokens=input_tokens,
                    output_tokens=output_tokens,
                    success=True,
+                    cached_input_tokens=cached_input_tokens,
+                    thoughts_tokens=thoughts_tokens,
                )

                # Record trace span
@@ -330,6 +388,7 @@ class GeminiLLM(LLMInterface):
                    duration=duration,
                    finish_reason=finish_reason,
                    error=None,
+                    cached_tokens=cached_tokens,
                )

                # Log slow calls
@@ -345,6 +404,7 @@ class GeminiLLM(LLMInterface):
                        input_tokens=input_tokens,
                        output_tokens=output_tokens,
                        total_tokens=input_tokens + output_tokens,
+                        cached_tokens=cached_tokens,
                    )
                    return result, token_usage
                return result
@@ -366,6 +426,20 @@ class GeminiLLM(LLMInterface):
                    logger.error(f"Gemini auth error (HTTP {e.code}), not retrying: {str(e)}")
                    raise

+                # Cached-request safety net: a stale/invalid/expired CachedContent
+                # (or an incompatibility like cache + tool_config) surfaces as a 400.
+                # Retrying the same cached request can't recover, so on the first
+                # such failure drop the cache, invalidate it so later operations
+                # recreate it, and retry THIS call inline with the prefix inlined.
+                # Caching must never break a request.
+                if cache_active and e.code == 400:
+                    logger.warning(f"Gemini cached call failed (400); retrying uncached. Reason: {str(e)}")
+                    if self._cache_manager is not None and cached_prefix is not None:
+                        self._cache_manager.invalidate(cached_prefix)
+                    cache_active = False
+                    generation_config = _build_generation_config(cache_active)
+                    continue
+
                # Retry on retryable errors (rate limits, server errors, client errors)
                if e.code in (400, 429, 500, 502, 503, 504) or (e.code and e.code >= 500):
                    last_exception = e
@@ -399,6 +473,7 @@ class GeminiLLM(LLMInterface):
        initial_backoff: float = 1.0,
        max_backoff: float = 30.0,
        tool_choice: str | dict[str, Any] = "auto",
+        cached_prefix: str | None = None,
    ) -> LLMToolCallResult:
        """
        Make a Gemini/VertexAI API call with tool/function calling support.
@@ -413,27 +488,39 @@ class GeminiLLM(LLMInterface):
            initial_backoff: Initial backoff time in seconds.
            max_backoff: Maximum backoff time in seconds.
            tool_choice: How to choose tools (Gemini uses "auto" only).
+            cached_prefix: Optional CachedContent resource name (from
+                ``GeminiCacheManager.get_or_create`` with ``tools=...``). When
+                set, the system_instruction and tool definitions are assumed
+                to live in the cache; this call will skip resending them and
+                the cached prefix is billed at the cached-input rate. The
+                ``tools`` argument is still required (the caller may pass
+                an empty list when the cache holds them) so existing call
+                sites don't break.

        Returns:
            LLMToolCallResult with content and/or tool_calls.
        """
        start_time = time.time()
+        using_cache = cached_prefix is not None

-        # Convert tools to Gemini format
+        # Convert tools to Gemini format. When the cache is in use, the
+        # tool definitions are baked into the CachedContent at create time
+        # and the SDK rejects re-sending them alongside ``cached_content``.
        gemini_tools = []
-        for tool in tools:
-            func = tool.get("function", {})
-            gemini_tools.append(
-                genai_types.Tool(
-                    function_declarations=[
-                        genai_types.FunctionDeclaration(
-                            name=func.get("name", ""),
-                            description=func.get("description", ""),
-                            parameters=func.get("parameters"),
-                        )
-                    ]
+        if not using_cache:
+            for tool in tools:
+                func = tool.get("function", {})
+                gemini_tools.append(
+                    genai_types.Tool(
+                        function_declarations=[
+                            genai_types.FunctionDeclaration(
+                                name=func.get("name", ""),
+                                description=func.get("description", ""),
+                                parameters=func.get("parameters"),
+                            )
+                        ]
+                    )
                )
-            )

        # Convert messages
        system_instruction = None
@@ -446,6 +533,10 @@ class GeminiLLM(LLMInterface):
            content = msg.get("content", "")

            if role == "system":
+                # Always capture system_instruction. _build_tools_config omits it
+                # (and tools) from the request while the cache carries the prefix,
+                # but it must be available so the cached-call-failed safety net can
+                # re-send the prefix + tools inline.
                system_instruction = (system_instruction + "\n\n" + content) if system_instruction else content
                i += 1
            elif role == "tool":
@@ -493,49 +584,63 @@ class GeminiLLM(LLMInterface):
                gemini_contents.append(genai_types.Content(role="user", parts=[genai_types.Part(text=content)]))
                i += 1

-        config_kwargs: dict[str, Any] = {"tools": gemini_tools}
-        if system_instruction:
-            config_kwargs["system_instruction"] = system_instruction
-        if temperature is not None:
-            config_kwargs["temperature"] = temperature
-        # See note in `call`: Gemini's max_output_tokens is the equivalent of
-        # OpenAI-style max_completion_tokens.
-        if max_completion_tokens is not None:
-            config_kwargs["max_output_tokens"] = max_completion_tokens
-
-        # Map OpenAI-style tool_choice to Gemini FunctionCallingConfig
-        if tool_choice == "required":
-            config_kwargs["tool_config"] = genai_types.ToolConfig(
-                function_calling_config=genai_types.FunctionCallingConfig(
-                    mode="ANY",
-                )
-            )
-        elif isinstance(tool_choice, dict) and tool_choice.get("type") == "function":
-            fn_name = tool_choice.get("function", {}).get("name")
-            if fn_name:
-                config_kwargs["tool_config"] = genai_types.ToolConfig(
-                    function_calling_config=genai_types.FunctionCallingConfig(
-                        mode="ANY",
-                        allowed_function_names=[fn_name],
-                    )
-                )
-        elif tool_choice == "none":
-            config_kwargs["tool_config"] = genai_types.ToolConfig(
-                function_calling_config=genai_types.FunctionCallingConfig(mode="NONE")
-            )
-        # "auto" is the default (no tool_config needed)
-
        # Apply safety settings: context var (per-request bank override) takes precedence over instance default
        effective_safety_settings = _safety_settings_ctx.get()
        if effective_safety_settings is None:
            effective_safety_settings = self._safety_settings
-        if effective_safety_settings is not None:
-            config_kwargs["safety_settings"] = [
-                genai_types.SafetySetting(category=s["category"], threshold=s["threshold"])
-                for s in effective_safety_settings
-            ]

-        config = genai_types.GenerateContentConfig(**config_kwargs)
+        # When using a cached prefix, the SDK rejects re-sending system_instruction
+        # or tools alongside ``cached_content`` — the cache IS the prefix.
+        # tool_config (mode / allowed_function_names) is a per-request decision and
+        # stays out of the cache. Built as a closure so we can rebuild it WITHOUT
+        # the cache and retry inline if a stale/invalid cache makes the call fail.
+        def _build_tools_config(use_cache: bool) -> "genai_types.GenerateContentConfig":
+            # Seed with user-configured extra params; explicit settings below win.
+            config_kwargs: dict[str, Any] = dict(self._extra_body)
+            if use_cache:
+                config_kwargs["cached_content"] = cached_prefix
+            else:
+                config_kwargs["tools"] = gemini_tools
+                if system_instruction:
+                    config_kwargs["system_instruction"] = system_instruction
+            if temperature is not None:
+                config_kwargs["temperature"] = temperature
+            # See note in `call`: Gemini's max_output_tokens is the equivalent of
+            # OpenAI-style max_completion_tokens.
+            if max_completion_tokens is not None:
+                config_kwargs["max_output_tokens"] = max_completion_tokens
+
+            # Map OpenAI-style tool_choice to Gemini FunctionCallingConfig
+            if tool_choice == "required":
+                config_kwargs["tool_config"] = genai_types.ToolConfig(
+                    function_calling_config=genai_types.FunctionCallingConfig(
+                        mode="ANY",
+                    )
+                )
+            elif isinstance(tool_choice, dict) and tool_choice.get("type") == "function":
+                fn_name = tool_choice.get("function", {}).get("name")
+                if fn_name:
+                    config_kwargs["tool_config"] = genai_types.ToolConfig(
+                        function_calling_config=genai_types.FunctionCallingConfig(
+                            mode="ANY",
+                            allowed_function_names=[fn_name],
+                        )
+                    )
+            elif tool_choice == "none":
+                config_kwargs["tool_config"] = genai_types.ToolConfig(
+                    function_calling_config=genai_types.FunctionCallingConfig(mode="NONE")
+                )
+            # "auto" is the default (no tool_config needed)
+
+            if effective_safety_settings is not None:
+                config_kwargs["safety_settings"] = [
+                    genai_types.SafetySetting(category=s["category"], threshold=s["threshold"])
+                    for s in effective_safety_settings
+                ]
+            return genai_types.GenerateContentConfig(**config_kwargs)
+
+        cache_active = using_cache
+        config = _build_tools_config(cache_active)

        last_exception = None
        for attempt in range(max_retries + 1):
@@ -578,12 +683,18 @@ class GeminiLLM(LLMInterface):

                finish_reason = "tool_calls" if tool_calls else "stop"

-                # Extract token usage
+                # Extract token usage. ``cached_content_token_count`` and
+                # ``thoughts_token_count`` are populated on the Gemini 2.5+
+                # family; absent fields are treated as 0.
                input_tokens = 0
                output_tokens = 0
+                cached_input_tokens = 0
+                thoughts_tokens = 0
                if response.usage_metadata:
                    input_tokens = response.usage_metadata.prompt_token_count or 0
                    output_tokens = response.usage_metadata.candidates_token_count or 0
+                    cached_input_tokens = getattr(response.usage_metadata, "cached_content_token_count", 0) or 0
+                    thoughts_tokens = getattr(response.usage_metadata, "thoughts_token_count", 0) or 0

                # Record metrics
                duration = time.time() - start_time
@@ -596,6 +707,8 @@ class GeminiLLM(LLMInterface):
                    input_tokens=input_tokens,
                    output_tokens=output_tokens,
                    success=True,
+                    cached_input_tokens=cached_input_tokens,
+                    thoughts_tokens=thoughts_tokens,
                )

                # Record OpenTelemetry span
@@ -620,6 +733,7 @@ class GeminiLLM(LLMInterface):
                    finish_reason=finish_reason,
                    error=None,
                    tool_calls=tool_calls_dict,
+                    cached_tokens=cached_input_tokens,
                )

                return LLMToolCallResult(
@@ -636,6 +750,18 @@ class GeminiLLM(LLMInterface):
                    logger.error(f"Gemini auth error (HTTP {e.code}), not retrying: {str(e)}")
                    raise

+                # Cached-request safety net (see ``call``): a stale/invalid cache or
+                # a cache+tool_config conflict surfaces as a 400. Drop the cache,
+                # invalidate it for later operations, and retry THIS call inline
+                # with the prefix + tools re-sent. Caching must never break a call.
+                if cache_active and e.code == 400:
+                    logger.warning(f"Gemini cached tool call failed (400); retrying uncached. Reason: {str(e)}")
+                    if self._cache_manager is not None and cached_prefix is not None:
+                        self._cache_manager.invalidate(cached_prefix)
+                    cache_active = False
+                    config = _build_tools_config(cache_active)
+                    continue
+
                # Retry on retryable errors
                last_exception = e
                if attempt < max_retries:
@@ -652,6 +778,54 @@ class GeminiLLM(LLMInterface):
            raise last_exception
        raise RuntimeError("Gemini tool call failed")

+    def supports_prompt_caching(self) -> bool:
+        """True when explicit Gemini context caching is enabled for this instance.
+
+        Reflects the opt-in flag so callers skip the cache lookup entirely when
+        it's off; ``get_or_create_cached_prefix`` also returns None in that case.
+        """
+        return self._prompt_cache_enabled
+
+    async def get_or_create_cached_prefix(
+        self,
+        *,
+        system_instruction: str,
+        response_schema: Any | None = None,
+        tools: list[dict[str, Any]] | None = None,
+    ) -> str | None:
+        """Return a CachedContent resource name for the given prefix, or
+        ``None`` if context caching is disabled, the provider doesn't
+        support it, or Gemini rejects the create (prefix too small, etc.).
+
+        ``tools`` is the OpenAI-style tools list; pass it when caching a
+        prefix that will be used by ``call_with_tools()``. The fingerprint
+        includes the tool definitions so a loop that swaps a tool gets a
+        fresh cache automatically.
+
+        Callers pass the returned name to ``call(cached_prefix=...)``
+        or ``call_with_tools(cached_prefix=...)`` and treat ``None``
+        as "cache unavailable — use the normal path". That fallback is
+        essential: the system must continue to work if caching is disabled,
+        if Gemini's caching API has an outage, or if the prefix is below
+        the model's minimum cacheable size.
+        """
+        if not self._prompt_cache_enabled:
+            return None
+        if self._client is None:
+            return None
+        if self._cache_manager is None:
+            # Lazy import so the cache module is only loaded when caching
+            # is actually used.
+            from hindsight_api.engine.providers.gemini_cache import GeminiCacheManager
+
+            self._cache_manager = GeminiCacheManager(self._client)
+        return await self._cache_manager.get_or_create(
+            model=self.model,
+            system_instruction=system_instruction,
+            response_schema=response_schema,
+            tools=tools,
+        )
+
    async def cleanup(self) -> None:
        """Clean up resources (close connections, etc.)."""
        # Gemini client doesn't require explicit cleanup
@@ -48,11 +48,18 @@ class LiteLLMLLM(LLMInterface):
        model: str,
        reasoning_effort: str = "low",
        timeout: float = 300.0,
+        extra_body: dict[str, Any] | None = None,
        **kwargs: Any,
    ):
        super().__init__(provider, api_key, base_url, model, reasoning_effort, **kwargs)
        self.timeout = timeout
        self._litellm: Any = None
+        # User-configured extra params merged as top-level kwargs into every
+        # completion call so LiteLLM normalizes them per-provider (e.g. maps
+        # temperature/top_p/max_tokens across OpenAI, Anthropic, Bedrock, …) and
+        # drops any the target model rejects (litellm.drop_params=True below).
+        # Sourced from llm_extra_body (env: HINDSIGHT_API_LLM_EXTRA_BODY).
+        self._extra_body: dict[str, Any] = extra_body or {}

        try:
            import litellm
@@ -107,6 +114,11 @@ class LiteLLMLLM(LLMInterface):
        if temperature is not None:
            kwargs["temperature"] = temperature

+        # User-configured extras fill in only where the caller didn't set a value,
+        # so explicit per-call params (model, messages, temperature, …) always win.
+        for key, value in self._extra_body.items():
+            kwargs.setdefault(key, value)
+
        return kwargs

    # ── per-model output-tokens cap (shared with Router subclass) ────────────
@@ -162,6 +162,12 @@ class MockLLM(LLMInterface):
            # Consolidation: produce a single observation from the input facts
            # so the full pipeline (retain → consolidation → observation → recall) works.
            result = self._build_mock_consolidation(messages, response_format)
+        elif scope == "consolidation_dedup" and response_format is not None:
+            # Observation dedup adjudication. Default to "keep" so mock-LLM consolidation never
+            # spuriously merges observations — this preserves the pre-dedup behaviour that
+            # deterministic consolidation tests assert (the generic branch below can't construct
+            # the model because its "action" field is required and has no default).
+            result = response_format(action="keep", reason="mock")
        elif scope == "memory_think":
            # Reflect: return a plausible text answer
            result = "Based on the available information, the answer is related to the context provided."
@@ -7,7 +7,7 @@ This provider handles all OpenAI API-compatible models including:
 - Groq: Fast inference with seed control and service tiers
 - Ollama: Local models with native streaming API support
 - LMStudio: Local models with OpenAI-compatible API
- MiniMax: MiniMax-M2.7 models with 1M context window
+- MiniMax: MiniMax-M3 / MiniMax-M2.7 models with 1M context window
 - DeepSeek: deepseek-v4-flash / deepseek-v4-pro / deepseek-chat / deepseek-reasoner via api.deepseek.com
 - Opencode Go: deepseek-v4-flash via https://opencode.ai/zen/go/v1

@@ -44,6 +44,16 @@ logger = logging.getLogger(__name__)
 DEFAULT_LLM_SEED = 4242
 JSON_MODE_USER_HINT = "Return valid json only."

+# Self-hosted OpenAI-compatible servers that advertise tool_choice="required"
+# but silently ignore it: instead of forcing a tool call they return
+# finish_reason "stop"/"tool_calls" with an EMPTY tool_calls array and no error.
+# Reflect's agent loop then sees no tool call, runs synthesis with no retrieval,
+# and answers "I don't have information" even when the bank holds the answer.
+# See issues #1563 (LM Studio), #1179 (LM Studio + Qwen), #1877 (vLLM with
+# --enable-auto-tool-choice). llama-server (the "llamacpp" provider) honors
+# "required" correctly and is intentionally excluded (#1179).
+_TOOL_CHOICE_REQUIRED_UNSUPPORTED_PROVIDERS = frozenset({"lmstudio", "ollama"})
+

 class ProviderResponseError(RuntimeError):
    """Raised when a provider returns a success response without usable content."""
@@ -232,7 +242,7 @@ class OpenAICompatibleLLM(LLMInterface):
    - Groq: Fast inference with seed control and service tiers
    - Ollama: Local models with native streaming API for better structured output
    - LMStudio: Local models with OpenAI-compatible API
-    - MiniMax: MiniMax-M2.7 models via OpenAI-compatible API (https://api.minimax.io/v1)
+    - MiniMax: MiniMax-M3 / MiniMax-M2.7 models via OpenAI-compatible API (https://api.minimax.io/v1)
    - DeepSeek: deepseek-v4-flash / deepseek-v4-pro / deepseek-chat / deepseek-reasoner via https://api.deepseek.com
    - opencode-go: deepseek-v4-flash via https://opencode.ai/zen/go/v1
    """
@@ -360,6 +370,21 @@ class OpenAICompatibleLLM(LLMInterface):
            f"base_url={self.base_url or 'default'}"
        )

+    def _drops_tool_choice_required(self) -> bool:
+        """Whether this endpoint silently ignores ``tool_choice="required"``.
+
+        True for self-hosted OpenAI-compatible servers known to return an empty
+        tool_calls array for "required" instead of forcing a call (#1563/#1179/
+        #1877). Covers LM Studio / Ollama directly, plus any server reached via
+        the generic "openai" provider with a custom ``base_url`` (e.g. a local
+        vLLM endpoint). The real OpenAI API (no base_url override) honors
+        "required", and cloud providers keep their own default base_urls, so both
+        are left untouched.
+        """
+        if self.provider in _TOOL_CHOICE_REQUIRED_UNSUPPORTED_PROVIDERS:
+            return True
+        return self.provider == "openai" and bool(self.base_url)
+
    async def verify_connection(self) -> None:
        """
        Verify that the provider is configured correctly by making a simple test call.
@@ -460,7 +485,9 @@ class OpenAICompatibleLLM(LLMInterface):
            initial_backoff: Initial backoff time in seconds.
            max_backoff: Maximum backoff time in seconds.
            skip_validation: Return raw JSON without Pydantic validation.
-            strict_schema: Use strict JSON schema enforcement (OpenAI only).
+            strict_schema: Use strict json_schema (grammar-enforced) response_format instead of
+                the soft json_object path. Supported by OpenAI and schema-capable self-hosted
+                backends (llama.cpp, vLLM). Server-wide via HINDSIGHT_API_LLM_STRICT_SCHEMA.
            return_usage: If True, return tuple (result, TokenUsage) instead of just result.

        Returns:
@@ -650,6 +677,9 @@ class OpenAICompatibleLLM(LLMInterface):
                input_tokens = usage.prompt_tokens or 0 if usage else 0
                output_tokens = usage.completion_tokens or 0 if usage else 0
                total_tokens = usage.total_tokens or 0 if usage else 0
+                cached_tokens = 0
+                if usage and getattr(usage, "prompt_tokens_details", None):
+                    cached_tokens = getattr(usage.prompt_tokens_details, "cached_tokens", 0) or 0

                # Record LLM metrics
                metrics = get_metrics_collector()
@@ -679,14 +709,12 @@ class OpenAICompatibleLLM(LLMInterface):
                    duration=duration,
                    finish_reason=finish_reason,
                    error=None,
+                    cached_tokens=cached_tokens,
                )

                # Log slow calls
                if duration > 10.0 and usage:
                    ratio = max(1, output_tokens) / max(1, input_tokens)
-                    cached_tokens = 0
-                    if hasattr(usage, "prompt_tokens_details") and usage.prompt_tokens_details:
-                        cached_tokens = getattr(usage.prompt_tokens_details, "cached_tokens", 0) or 0
                    cache_info = f", cached_tokens={cached_tokens}" if cached_tokens > 0 else ""
                    logger.info(
                        f"slow llm call: scope={scope}, model={self.provider}/{self.model}, "
@@ -699,6 +727,7 @@ class OpenAICompatibleLLM(LLMInterface):
                        input_tokens=input_tokens,
                        output_tokens=output_tokens,
                        total_tokens=total_tokens,
+                        cached_tokens=cached_tokens,
                    )
                    return result, token_usage
                return result
@@ -867,6 +896,16 @@ class OpenAICompatibleLLM(LLMInterface):
        if request_tool_choice == "auto":
            request_tool_choice = None

+        # vLLM (--enable-auto-tool-choice), LM Studio, Ollama and similar
+        # self-hosted servers silently drop tool_choice="required", returning an
+        # empty tool_calls array instead of forcing a call (#1563/#1179/#1877).
+        # Downgrade to auto (None) so the model still gets to call a tool. Named
+        # tool_choice dicts were already normalized to "required" + a single
+        # filtered tool above, so the call stays practically forced even under
+        # auto. The real OpenAI API honors "required" and is left untouched.
+        if request_tool_choice == "required" and self._drops_tool_choice_required():
+            request_tool_choice = None
+
        # DeepSeek tool-call replies can carry provider-specific reasoning_content.
        # The normalized tool result does not retain it, but replaying assistant
        # tool_calls without the field can trigger a 400. DeepSeek accepts an
@@ -302,6 +302,24 @@ def _is_context_overflow_error(exc: Exception) -> bool:
    )


+def _all_mental_models_are_usable_and_fresh(tool_output: dict[str, Any]) -> bool:
+    """Return whether every retrieved mental model is explicitly fresh and has answerable content.
+
+    Used to decide — without an extra LLM call — whether a forced
+    ``search_mental_models`` result is trustworthy enough to hand control back
+    to the agent. A model is usable only when it is explicitly ``is_stale ==
+    False`` (an unknown/missing staleness flag is treated as unsafe) and has
+    non-empty content.
+    """
+    models = tool_output.get("mental_models") or []
+    for model in models:
+        if model.get("is_stale") is not False:
+            return False
+        if not str(model.get("content") or "").strip():
+            return False
+    return True
+
+
 async def run_reflect_agent(
    llm_config: "LLMProvider",
    bank_id: str,
@@ -382,6 +400,28 @@ async def run_reflect_agent(
        {"role": "user", "content": query},
    ]

+    # Opt into context caching for the agentic tool loop. The system
+    # prompt and tool definitions are stable for the duration of this
+    # reflect call (and across reflects against the same bank), so
+    # caching them once and reusing across every iteration of the loop
+    # collapses the dominant input cost — the prefix repeated on every
+    # turn. ``get_or_create_cached_prefix`` returns None when caching is
+    # disabled, unsupported, or the prefix is too small; the
+    # ``call_with_tools`` invocation below transparently falls back to
+    # the uncached path in that case.
+    cached_prefix_name: str | None = None
+    provider_impl = getattr(llm_config, "_provider_impl", None)
+    if provider_impl is not None and provider_impl.supports_prompt_caching():
+        try:
+            cached_prefix_name = await provider_impl.get_or_create_cached_prefix(
+                system_instruction=system_prompt,
+                tools=tools,
+            )
+        except Exception:
+            # Caching is a soft optimisation; never let a cache-side
+            # error block a reflect.
+            cached_prefix_name = None
+
    # Tracking
    total_tools_called = 0
    tool_trace: list[ToolCall] = []
@@ -442,6 +482,11 @@ async def run_reflect_agent(
        )

    consecutive_errors = 0
+    # When a forced ``search_mental_models`` returns fresh, usable models on a
+    # low/mid-budget call, we stop forcing the lower retrieval layers from this
+    # iteration onward and let the agent answer (or retrieve deeper itself)
+    # under ``auto`` tool choice. None means the full forced path still applies.
+    stop_forcing_from_iteration: int | None = None
    for iteration in range(max_iterations):
        is_last = iteration == max_iterations - 1

@@ -570,18 +615,31 @@ async def run_reflect_agent(
        if include_recall:
            forced_sequence.append("recall")

-        if iteration < len(forced_sequence):
-            iter_tool_choice: str | dict = {"type": "function", "function": {"name": forced_sequence[iteration]}}
+        if stop_forcing_from_iteration is not None and iteration >= stop_forcing_from_iteration:
+            # A fresh mental model already short-circuited the forced path.
+            iter_tool_choice: str | dict = "auto"
+        elif iteration < len(forced_sequence):
+            iter_tool_choice = {"type": "function", "function": {"name": forced_sequence[iteration]}}
        else:
            iter_tool_choice = "auto"

        try:
-            result = await llm_config.call_with_tools(
+            ct_kwargs: dict[str, Any] = dict(
                messages=messages,
                tools=tools,
                scope="reflect_tool_call",
                tool_choice=iter_tool_choice,
            )
+            # Gemini rejects ``cached_content`` alongside a per-request
+            # ``tool_config`` (forced tool choice): "CachedContent can not be used
+            # with GenerateContent request setting system_instruction, tools or
+            # tool_config." The forced-sequence iterations set tool_config, so only
+            # the ``auto`` iterations can reference the cache; forced iterations send
+            # the prefix inline. The cache (tools + system prompt) is identical
+            # either way, so this just limits *which* iterations are billed cached.
+            if cached_prefix_name is not None and iter_tool_choice == "auto":
+                ct_kwargs["cached_prefix"] = cached_prefix_name
+            result = await llm_config.call_with_tools(**ct_kwargs)
            llm_duration = int((time.time() - llm_start) * 1000)
            consecutive_errors = 0
            total_input_tokens += result.input_tokens
@@ -924,6 +982,25 @@ async def run_reflect_agent(
                    for mm in output["mental_models"]:
                        if "id" in mm:
                            available_mental_model_ids.add(mm["id"])
+                    # Deterministic short-circuit (no extra LLM call): on a
+                    # low/mid-budget call, if every retrieved mental model is
+                    # fresh and has usable content, stop forcing the lower
+                    # retrieval layers. The next iteration runs under ``auto``
+                    # tool choice, so the agent can answer directly when the
+                    # mental model suffices, or — having just read it — issue a
+                    # targeted ``search_observations``/``recall`` itself. Stale,
+                    # empty, or missing mental models keep the full forced path.
+                    if (
+                        stop_forcing_from_iteration is None
+                        and (budget or "low").lower() != "high"
+                        and output.get("mental_models")
+                        and _all_mental_models_are_usable_and_fresh(output)
+                    ):
+                        stop_forcing_from_iteration = iteration + 1
+                        logger.info(
+                            f"[REFLECT {reflect_id}] Fresh mental models sufficient on iteration {iteration + 1}; "
+                            "releasing forced lower-level retrieval to auto."
+                        )

                if (
                    normalized_tool_name == "search_observations"
@@ -93,6 +93,7 @@ class TokenUsage(BaseModel):
    input_tokens: int = Field(default=0, description="Number of input/prompt tokens consumed")
    output_tokens: int = Field(default=0, description="Number of output/completion tokens generated")
    total_tokens: int = Field(default=0, description="Total tokens (input + output)")
+    cached_tokens: int = Field(default=0, description="Cached/cache-read prompt tokens, when reported by the provider")

    def __add__(self, other: "TokenUsage") -> "TokenUsage":
        """Allow aggregating token usage from multiple calls."""
@@ -100,6 +101,7 @@ class TokenUsage(BaseModel):
            input_tokens=self.input_tokens + other.input_tokens,
            output_tokens=self.output_tokens + other.output_tokens,
            total_tokens=self.total_tokens + other.total_tokens,
+            cached_tokens=self.cached_tokens + other.cached_tokens,
        )


@@ -6,6 +6,7 @@ import json
 import logging
 import re
 import uuid
+from dataclasses import dataclass
 from typing import TypedDict

 from pydantic import BaseModel, Field
@@ -105,6 +106,18 @@ class BankProfile(TypedDict):
    mission: str


+@dataclass
+class BankProfileResult:
+    """Result of a get-or-create bank lookup.
+
+    ``created`` is True when the bank row was freshly inserted on this call,
+    which callers use to drive the one-time HINDSIGHT_API_DEFAULT_BANK_TEMPLATE hook.
+    """
+
+    profile: BankProfile
+    created: bool
+
+
 class MissionMergeResponse(BaseModel):
    """LLM response for mission merge."""

@@ -123,8 +136,8 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
    Returns:
        BankProfile with name, typed DispositionTraits, and mission
    """
-    profile, _ = await get_or_create_bank_profile(pool, bank_id)
-    return profile
+    result = await get_or_create_bank_profile(pool, bank_id)
+    return result.profile


 async def get_bank_profile_if_exists(pool, bank_id: str) -> BankProfile | None:
@@ -162,70 +175,89 @@ async def get_bank_profile_if_exists(pool, bank_id: str) -> BankProfile | None:
        )


-async def get_or_create_bank_profile(pool, bank_id: str) -> tuple[BankProfile, bool]:
+async def get_or_create_bank_profile(pool, bank_id: str) -> BankProfileResult:
    """
    Get bank profile, auto-creating with defaults if it doesn't exist.

-    Same as get_bank_profile, but also returns a flag indicating whether the
-    bank was freshly created on this call. Used by the memory engine to apply
-    the HINDSIGHT_API_DEFAULT_BANK_TEMPLATE hook on first bank creation.
+    Same as get_bank_profile, but also reports whether the bank was freshly
+    created on this call (``BankProfileResult.created``). Used by the memory
+    engine to apply the HINDSIGHT_API_DEFAULT_BANK_TEMPLATE hook on first bank
+    creation.

-    Returns:
-        Tuple of (BankProfile, created) where created is True if the bank
-        did not exist before this call.
+    Acquires its own connection. When the caller already holds a connection and
+    wants the bank row to share its transaction (so the lazy bank-create commits
+    or rolls back atomically with the caller's write), use
+    ``get_or_create_bank_profile_on_conn`` instead.
    """
    async with acquire_with_retry(pool) as conn:
-        # Try to get existing bank
-        row = await conn.fetchrow(
-            f"""
-            SELECT name, disposition, mission
-            FROM {fq_table("banks")} WHERE bank_id = $1
-            """,
-            bank_id,
+        return await get_or_create_bank_profile_on_conn(conn, bank_id, ops=pool.ops)
+
+
+async def get_or_create_bank_profile_on_conn(conn, bank_id: str, *, ops) -> BankProfileResult:
+    """
+    Connection-bound variant of ``get_or_create_bank_profile``.
+
+    Runs the SELECT, the ``INSERT ... ON CONFLICT DO NOTHING`` and the per-bank
+    vector index creation on the caller-supplied ``conn``. When ``conn`` is
+    inside an open transaction, the lazy bank-create therefore commits (or rolls
+    back) atomically with whatever bank-scoped write the caller performs on the
+    same connection — closing the window where a freshly-created bank could
+    outlive a write that ultimately failed.
+
+    ``ops`` is the backend's dialect ops object (``backend.ops``), needed for
+    per-bank vector index DDL.
+    """
+    # Try to get existing bank
+    row = await conn.fetchrow(
+        f"""
+        SELECT name, disposition, mission
+        FROM {fq_table("banks")} WHERE bank_id = $1
+        """,
+        bank_id,
+    )
+
+    if row:
+        # asyncpg returns JSONB as a string, so parse it
+        disposition_data = row["disposition"]
+        if isinstance(disposition_data, str):
+            disposition_data = json.loads(disposition_data)
+
+        return BankProfileResult(
+            profile=BankProfile(
+                name=row["name"],
+                disposition=DispositionTraits(**disposition_data),
+                mission=row["mission"] or "",
+            ),
+            created=False,
        )

-        if row:
-            # asyncpg returns JSONB as a string, so parse it
-            disposition_data = row["disposition"]
-            if isinstance(disposition_data, str):
-                disposition_data = json.loads(disposition_data)
+    # Bank doesn't exist, create with defaults.
+    # Generate internal_id here so we control the value and can use it
+    # immediately for vector index creation without a RETURNING round-trip.
+    internal_id = uuid.uuid4()
+    inserted = await conn.fetchval(
+        f"""
+        INSERT INTO {fq_table("banks")} (bank_id, name, disposition, mission, internal_id)
+        VALUES ($1, $2, $3::jsonb, $4, $5)
+        ON CONFLICT (bank_id) DO NOTHING
+        RETURNING bank_id
+        """,
+        bank_id,
+        bank_id,  # Default name is the bank_id
+        json.dumps(DEFAULT_DISPOSITION),
+        "",
+        internal_id,
+    )

-            return (
-                BankProfile(
-                    name=row["name"],
-                    disposition=DispositionTraits(**disposition_data),
-                    mission=row["mission"] or "",
-                ),
-                False,
-            )
+    created = inserted is not None
+    if created:
+        # Fresh insert — create per-bank vector indexes (instant on empty bank)
+        await create_bank_vector_indexes(conn, bank_id, str(internal_id), ops=ops)

-        # Bank doesn't exist, create with defaults.
-        # Generate internal_id here so we control the value and can use it
-        # immediately for vector index creation without a RETURNING round-trip.
-        internal_id = uuid.uuid4()
-        inserted = await conn.fetchval(
-            f"""
-            INSERT INTO {fq_table("banks")} (bank_id, name, disposition, mission, internal_id)
-            VALUES ($1, $2, $3::jsonb, $4, $5)
-            ON CONFLICT (bank_id) DO NOTHING
-            RETURNING bank_id
-            """,
-            bank_id,
-            bank_id,  # Default name is the bank_id
-            json.dumps(DEFAULT_DISPOSITION),
-            "",
-            internal_id,
-        )
-
-        created = inserted is not None
-        if created:
-            # Fresh insert — create per-bank vector indexes (instant on empty bank)
-            await create_bank_vector_indexes(conn, bank_id, str(internal_id), ops=pool.ops)
-
-        return (
-            BankProfile(name=bank_id, disposition=DispositionTraits(**DEFAULT_DISPOSITION), mission=""),
-            created,
-        )
+    return BankProfileResult(
+        profile=BankProfile(name=bank_id, disposition=DispositionTraits(**DEFAULT_DISPOSITION), mission=""),
+        created=created,
+    )


 async def update_bank_disposition(pool, bank_id: str, disposition: dict[str, int]) -> None:
@@ -15,11 +15,23 @@ class EmbeddingsBackend(Protocol):
    """Minimal duck-typed surface used by retain/recall — the concrete `Embeddings`
    ABC supplies default implementations that delegate to `encode()`."""

+    @property
+    def dimension(self) -> int: ...
+
    def encode_query(self, texts: list[str]) -> list[list[float]]: ...

    def encode_documents(self, texts: list[str]) -> list[list[float]]: ...


+def _validate_embedding_vector(vector: list[float], *, index: int, expected_dimension: int) -> list[float]:
+    actual_dimension = len(vector)
+    if actual_dimension == 0:
+        raise RuntimeError(f"embedding {index} has dimension 0; expected {expected_dimension}")
+    if actual_dimension != expected_dimension:
+        raise RuntimeError(f"embedding {index} has dimension {actual_dimension}; expected {expected_dimension}")
+    return vector
+
+
 def generate_embedding(
    embeddings_backend: EmbeddingsBackend, text: str, input_type: EmbeddingInputType = "document"
 ) -> list[float]:
@@ -36,10 +48,19 @@ def generate_embedding(
    """
    try:
        embeddings = _encode_with_input_type(embeddings_backend, [text], input_type)
-        return embeddings[0]
    except Exception as e:
        raise Exception(f"Failed to generate embedding: {str(e)}")

+    if len(embeddings) != 1:
+        raise RuntimeError(
+            f"Embeddings backend returned {len(embeddings)} vectors for 1 input text; expected exact 1:1 alignment"
+        )
+    return _validate_embedding_vector(
+        embeddings[0],
+        index=0,
+        expected_dimension=embeddings_backend.dimension,
+    )
+

 def _encode_with_input_type(
    embeddings_backend: EmbeddingsBackend, texts: list[str], input_type: EmbeddingInputType
@@ -81,4 +102,7 @@ async def generate_embeddings_batch(
            "expected exact 1:1 alignment"
        )

-    return embeddings
+    return [
+        _validate_embedding_vector(embedding, index=index, expected_dimension=embeddings_backend.dimension)
+        for index, embedding in enumerate(embeddings)
+    ]
@@ -10,12 +10,13 @@ import json
 import logging
 import re
 from datetime import datetime, timedelta
-from typing import Literal, cast
+from typing import Any, Literal, cast

 from pydantic import BaseModel, ConfigDict, Field, create_model, field_validator

 from ...config import get_config
 from ..llm_wrapper import LLMConfig, OutputTooLongError, sanitize_llm_output
+from ..operation_metadata import RetainExtractionErrors
 from ..response_models import TokenUsage
 from .entity_labels import (
    EntityLabelsConfig,
@@ -510,11 +511,11 @@ LANGUAGE: MANDATORY — Detect the language of the input text and produce ALL ou
 FACT FORMAT - BE CONCISE
 ══════════════════════════════════════════════════════════════════════════

-1. **what**: Core fact - concise but complete (1-2 sentences max)
-2. **when**: Temporal info if mentioned. "N/A" if none. Use day name when known.
-3. **where**: Location if relevant. "N/A" if none.
-4. **who**: People involved with relationships. "N/A" if just general info.
-5. **why**: Context/significance ONLY if important. "N/A" if obvious.
+1. "what": Core fact - concise but complete (1-2 sentences max)
+2. "when": Temporal info if mentioned. "N/A" if none. Use day name when known.
+3. "where": Location if relevant. "N/A" if none.
+4. "who": People involved with relationships. "N/A" if just general info.
+5. "why": Context/significance ONLY if important. "N/A" if obvious.

 CONCISENESS: Capture the essence, not every word. One good sentence beats three mediocre ones.

@@ -887,20 +888,15 @@ def _build_extraction_prompt_and_schema(config) -> tuple[str, type]:
    extraction_mode = config.retain_extraction_mode
    extract_causal_links = config.retain_extract_causal_links

-    # Build retain_mission section if set - injected before the mode-specific guidelines
-    # Escape braces so user-supplied text survives str.format() on the prompt template.
+    # The per-bank retain mission is NOT baked into this system prompt: it would
+    # make the prompt bank-specific and force a separate Gemini context cache per
+    # mission (one per bank). Instead the prompt is bank-agnostic so a single
+    # CachedContent serves every bank, and the mission rides in the per-request
+    # user message via _retain_mission_preamble(). The {retain_mission_section}
+    # placeholder is kept (templates still reference it) but always empty here.
    from hindsight_api.engine.prompt_utils import escape_for_prompt

-    retain_mission = getattr(config, "retain_mission", None)
-    if retain_mission:
-        retain_mission_section = (
-            f"══════════════════════════════════════════════════════════════════════════\n"
-            f"FOCUS — What to retain for this bank\n"
-            f"══════════════════════════════════════════════════════════════════════════\n\n"
-            f"{escape_for_prompt(retain_mission)}\n\n"
-        )
-    else:
-        retain_mission_section = ""
+    retain_mission_section = ""

    # Select base prompt based on extraction mode
    if extraction_mode == "custom":
@@ -997,6 +993,26 @@ def _build_extraction_prompt_and_schema(config) -> tuple[str, type]:
    return prompt, response_schema


+def _retain_mission_preamble(config) -> str:
+    """The bank's retain mission, formatted for the per-request user message.
+
+    Kept OUT of the cached system prompt (which must stay bank-agnostic so one
+    CachedContent serves every bank — otherwise each distinct mission spawns its
+    own cache) and prepended to the user message instead. Returns "" when unset.
+    No brace-escaping needed: unlike the system template, the user message is
+    used verbatim, not passed through str.format().
+    """
+    retain_mission = getattr(config, "retain_mission", None)
+    if not retain_mission:
+        return ""
+    return (
+        "══════════════════════════════════════════════════════════════════════════\n"
+        "FOCUS — What to retain for this bank (takes priority over the general guidelines)\n"
+        "══════════════════════════════════════════════════════════════════════════\n\n"
+        f"{retain_mission}\n\n"
+    )
+
+
 def _build_user_message(
    chunk: str,
    chunk_index: int,
@@ -1005,8 +1021,14 @@ def _build_user_message(
    context: str,
    metadata: dict[str, str] | None = None,
    agent_name: str | None = None,
+    mission_preamble: str = "",
 ) -> str:
-    """Build user message for fact extraction."""
+    """Build user message for fact extraction.
+
+    ``mission_preamble`` (the bank's retain mission, possibly empty) is prepended
+    so the bank-specific focus lives in the variable user turn rather than the
+    cached, bank-agnostic system prompt.
+    """
    from .orchestrator import parse_datetime_flexible

    sanitized_chunk = _sanitize_text(chunk)
@@ -1025,9 +1047,21 @@ def _build_user_message(

    narrator_section = ""
    if agent_name:
-        narrator_section = f'\nNarrator: {agent_name} (AI agent — first-person statements like "I did X" are the agent\'s own actions; classify as "assistant")'
+        narrator_section = (
+            f"\nNarrator: {agent_name} (the AI agent whose memory this is). By default, "
+            f'first-person statements like "I did X" are {agent_name}\'s own actions → classify as '
+            f'"assistant".'
+        )
+        # Only defer to the Context when one was actually provided — otherwise this
+        # clause points at a "Context: none" line and just adds noise.
+        if context:
+            narrator_section += (
+                " BUT the Context above takes precedence: if it identifies a different "
+                "first-person speaker (e.g. a user or customer in a transcript), attribute those "
+                'statements to that speaker and classify them as "world", not "assistant".'
+            )

-    return f"""Extract facts from the following text chunk.
+    return f"""{mission_preamble}Extract facts from the following text chunk.

 Chunk: {chunk_index + 1}/{total_chunks}
 Event Date: {event_date_str}
@@ -1053,12 +1087,15 @@ def _build_request_body(llm_config, config, prompt: str, user_message: str, resp
    if llm_config.provider == "openai" and llm_config._provider_impl.openai_service_tier:
        request_body["service_tier"] = llm_config._provider_impl.openai_service_tier

-    # Add response_format (JSON schema)
+    # Add response_format (JSON schema). The batch path builds the request body
+    # directly instead of going through LLMProvider.call(), so honour
+    # HINDSIGHT_API_LLM_STRICT_SCHEMA here too: strict=True grammar-enforces the
+    # output on capable backends rather than relying on the model to emit clean JSON.
    if hasattr(response_schema, "model_json_schema"):
        schema = response_schema.model_json_schema()
        request_body["response_format"] = {
            "type": "json_schema",
-            "json_schema": {"name": "facts", "schema": schema},
+            "json_schema": {"name": "facts", "schema": schema, "strict": config.llm_strict_schema},
        }

    return request_body
@@ -1094,8 +1131,38 @@ async def _extract_facts_from_chunk(
    extraction_mode = config.retain_extraction_mode
    extract_causal_links = config.retain_extract_causal_links

-    # Build user message using helper function
-    user_message = _build_user_message(chunk, chunk_index, total_chunks, event_date, context, metadata, agent_name)
+    # Build user message — the bank mission rides here (not in the cached prefix).
+    user_message = _build_user_message(
+        chunk,
+        chunk_index,
+        total_chunks,
+        event_date,
+        context,
+        metadata,
+        agent_name,
+        mission_preamble=_retain_mission_preamble(config),
+    )
+
+    # Opt into context caching when the provider supports it. The prompt and
+    # response_schema are bank-agnostic (the mission lives in the user message),
+    # so one cached prefix serves every bank; reusing it across many small-payload
+    # retain calls dramatically lowers per-call input
+    # cost. ``get_or_create_cached_prefix`` returns None when caching is
+    # disabled, unsupported, or the prefix is too small; the LLM call
+    # transparently falls back to the uncached path in that case.
+    cached_prefix_name: str | None = None
+    provider_impl = getattr(llm_config, "_provider_impl", None)
+    if provider_impl is not None and provider_impl.supports_prompt_caching():
+        try:
+            cached_prefix_name = await provider_impl.get_or_create_cached_prefix(
+                system_instruction=prompt,
+                response_schema=response_schema,
+            )
+        except Exception:
+            # Caching is a soft optimisation — never let a cache-side
+            # error block a retain operation.
+            logger.exception("Cache prefix lookup failed; falling back to uncached call")
+            cached_prefix_name = None

    # Retry logic for JSON validation errors
    # Use retain-specific overrides if set, otherwise fall back to global LLM config
@@ -1116,7 +1183,7 @@ async def _extract_facts_from_chunk(
                config.retain_llm_max_backoff if config.retain_llm_max_backoff is not None else config.llm_max_backoff
            )

-            extraction_response_json, call_usage = await llm_config.call(
+            call_kwargs: dict[str, Any] = dict(
                messages=[{"role": "system", "content": prompt}, {"role": "user", "content": user_message}],
                response_format=response_schema,
                scope="retain_extract_facts",
@@ -1128,6 +1195,10 @@ async def _extract_facts_from_chunk(
                skip_validation=True,  # Get raw JSON, we'll validate leniently
                return_usage=True,
            )
+            if cached_prefix_name is not None:
+                call_kwargs["cached_prefix"] = cached_prefix_name
+
+            extraction_response_json, call_usage = await llm_config.call(**call_kwargs)
            usage = usage + call_usage  # Aggregate usage across retries

            # Lenient parsing of facts from raw JSON
@@ -1640,6 +1711,39 @@ logger = logging.getLogger(__name__)
 SECONDS_PER_FACT = 0.01


+async def _write_batch_extraction_errors(
+    pool: Any,
+    operation_id: str | None,
+    schema: str | None,
+    errors: RetainExtractionErrors,
+) -> None:
+    """Persist non-fatal Batch API extraction errors into operation result_metadata."""
+    if not pool or not operation_id or errors.count == 0:
+        return
+
+    from ..db_utils import acquire_with_retry
+    from ..task_backend import fq_table
+
+    # `errors` is the complete set for this extraction run, so overwrite the
+    # extraction_errors_* keys rather than folding in what's already stored. On
+    # batch crash recovery the resumed batch reprocesses every result and
+    # recomputes `errors` from scratch; reading + merging the prior run's
+    # counters here would double-count them. The SQL `||` merge still preserves
+    # unrelated keys (e.g. batch_id) already on result_metadata.
+    table = fq_table("async_operations", schema)
+    async with acquire_with_retry(pool) as conn:
+        await conn.execute(
+            f"""
+            UPDATE {table}
+            SET result_metadata = COALESCE(result_metadata, '{{}}'::jsonb) || $2::jsonb,
+                updated_at = now()
+            WHERE operation_id = $1
+            """,
+            operation_id,
+            json.dumps(errors.to_dict()),
+        )
+
+
 async def extract_facts_from_contents_batch_api(
    contents: list[RetainContent],
    llm_config,
@@ -1731,6 +1835,7 @@ async def extract_facts_from_contents_batch_api(
                item.context,
                item.metadata or None,
                agent_name,
+                mission_preamble=_retain_mission_preamble(config),
            )

            # Build request body using helper function
@@ -1816,6 +1921,7 @@ async def extract_facts_from_contents_batch_api(
    all_facts_from_llm = []
    chunks_metadata = []
    total_usage = TokenUsage()
+    extraction_errors = RetainExtractionErrors()

    for chunk_idx, (chunk_content, content_index, chunk_index_in_content, event_date, context) in enumerate(
        all_chunks_info
@@ -1824,7 +1930,9 @@ async def extract_facts_from_contents_batch_api(
        result = results_by_id.get(custom_id)

        if not result:
-            logger.warning(f"Missing result for {custom_id}, skipping")
+            message = f"{custom_id}: missing batch result"
+            logger.warning(message)
+            extraction_errors.add(message)
            chunks_metadata.append(
                ChunkMetadata(
                    chunk_text=chunk_content, fact_count=0, content_index=content_index, chunk_index=chunk_idx
@@ -1834,7 +1942,9 @@ async def extract_facts_from_contents_batch_api(

        # Check for errors
        if result.get("error"):
-            logger.error(f"Error in {custom_id}: {result['error']}")
+            message = f"{custom_id}: {result['error']}"
+            logger.error(f"Error in {message}")
+            extraction_errors.add(message)
            chunks_metadata.append(
                ChunkMetadata(
                    chunk_text=chunk_content, fact_count=0, content_index=content_index, chunk_index=chunk_idx
@@ -1847,7 +1957,9 @@ async def extract_facts_from_contents_batch_api(
        choices = response_body.get("choices", [])

        if not choices:
-            logger.warning(f"No choices in response for {custom_id}")
+            message = f"{custom_id}: no choices in response"
+            logger.warning(message)
+            extraction_errors.add(message)
            chunks_metadata.append(
                ChunkMetadata(
                    chunk_text=chunk_content, fact_count=0, content_index=content_index, chunk_index=chunk_idx
@@ -1862,7 +1974,9 @@ async def extract_facts_from_contents_batch_api(
        try:
            extraction_response_json = json.loads(content_str)
        except json.JSONDecodeError as e:
-            logger.error(f"Failed to parse JSON for {custom_id}: {e}")
+            message = f"{custom_id}: failed to parse JSON: {e}"
+            logger.error(message)
+            extraction_errors.add(message)
            chunks_metadata.append(
                ChunkMetadata(
                    chunk_text=chunk_content, fact_count=0, content_index=content_index, chunk_index=chunk_idx
@@ -2035,7 +2149,9 @@ async def extract_facts_from_contents_batch_api(
                fact = Fact(fact=combined_text, fact_type=fact_type, **fact_data)
                chunk_facts.append(fact)
            except Exception as e:
-                logger.error(f"Failed to create Fact model for fact {i}: {e}")
+                message = f"{custom_id}: failed to create Fact model for fact {i}: {e}"
+                logger.error(message)
+                extraction_errors.add(message)
                continue

        all_facts_from_llm.extend(chunk_facts)
@@ -2100,6 +2216,8 @@ async def extract_facts_from_contents_batch_api(
    # Step 8: Auto-tag facts from label groups with tag=True
    _inject_label_tags(extracted_facts, config)

+    await _write_batch_extraction_errors(pool, operation_id, schema, extraction_errors)
+
    logger.info(f"Batch API extracted {len(extracted_facts)} facts from {len(all_chunks_info)} chunks")

    return extracted_facts, chunks_metadata, total_usage
@@ -147,12 +147,13 @@ async def ensure_bank_exists(conn, bank_id: str, ops=None) -> None:
    internal_id = uuid.uuid4()
    inserted = await conn.fetchval(
        f"""
-        INSERT INTO {fq_table("banks")} (bank_id, disposition, mission, internal_id)
-        VALUES ($1, $2::jsonb, $3, $4)
+        INSERT INTO {fq_table("banks")} (bank_id, name, disposition, mission, internal_id)
+        VALUES ($1, $2, $3::jsonb, $4, $5)
        ON CONFLICT (bank_id) DO NOTHING
        RETURNING bank_id
        """,
        bank_id,
+        bank_id,  # Default name is the bank_id (matches get_or_create_bank_profile)
        json.dumps(DEFAULT_DISPOSITION),
        "",
        internal_id,
@@ -599,23 +599,35 @@ async def compute_semantic_links_ann(
            t_query = time_mod.time()
            seed_count = sum(1 for ft in fact_types if ft == fact_type)
            logger.debug(f"[ANN] Querying fact_type={fact_type}: {seed_count} seeds")
+            # Cast each seed's text embedding to `vector` exactly once in a
+            # MATERIALIZED CTE. Casting inside the LATERAL (s.emb_text::vector)
+            # re-parses the ~5KB embedding string for every candidate row the
+            # probe touches — seeds × bank_units text-parses per batch, which
+            # dominated the whole job on small banks (see #1919: ~50 seeds over
+            # ~1k units took 1.5-3.7s, ~25-48x slower than casting once). The
+            # stable `vector` column also lets the planner consider an HNSW
+            # index scan, which a cast expression inhibits.
            ft_rows = await conn.fetch(
                f"""
+                WITH seeds AS MATERIALIZED (
+                    SELECT unit_id, emb_text::vector AS emb
+                    FROM _ann_seeds
+                    WHERE fact_type = $2
+                )
                SELECT s.unit_id       AS from_id,
                       n.id::text      AS to_id,
                       n.similarity
-                FROM _ann_seeds s
+                FROM seeds s
                CROSS JOIN LATERAL (
                    SELECT mu.id,
-                           1 - (mu.embedding <=> s.emb_text::vector) AS similarity
+                           1 - (mu.embedding <=> s.emb) AS similarity
                    FROM {fq_table("memory_units")} mu
                    WHERE mu.bank_id = $1
                      AND mu.fact_type = $2
                      AND mu.embedding IS NOT NULL
-                    ORDER BY mu.embedding <=> s.emb_text::vector
+                    ORDER BY mu.embedding <=> s.emb
                    LIMIT $3
                ) n
-                WHERE s.fact_type = $2
                """,
                bank_id,
                fact_type,
@@ -11,6 +11,7 @@ import logging
 import time
 import uuid
 from collections.abc import Awaitable, Callable
+from dataclasses import dataclass
 from datetime import UTC, datetime
 from typing import Any

@@ -111,6 +112,25 @@ RetainOutboxCallback = Callable[[asyncpg.Connection], Awaitable[None]]
 RetainOutboxCallbackFactory = Callable[[list[RetainContentDict]], RetainOutboxCallback | None]


+def _resolve_narrator(profile_name: str, bank_id: str) -> str | None:
+    """Resolve the narrator (memory owner) used to prime fact extraction.
+
+    The narrator is injected as a "Narrator: {name}" line in fact extraction and
+    is stamped into the who-dimension of every first-person fact — and the
+    observations later consolidated from those facts. That is correct for a named
+    agent retaining its own logs, but harmful when ``name`` is just the bank_id:
+    on auto-create the bank ``name`` defaults to ``bank_id``, which is typically a
+    routing key (e.g. ``my-agent::channel-456::user-789``), not a speaker. Priming
+    extraction with a routing key embeds that string into stored fact text and
+    pollutes downstream observations (issue #1680). Suppress it in that case.
+
+    Returns the narrator name, or ``None`` to omit the Narrator line entirely.
+    """
+    if profile_name == bank_id:
+        return None
+    return profile_name
+
+
 def _build_retain_params(contents_dicts, document_tags=None, doc_contents=None):
    """Build retain_params and merged_tags from content dicts."""
    if doc_contents is not None:
@@ -404,6 +424,7 @@ async def retain_batch(
    db_semaphore: "asyncio.Semaphore | None" = None,
    document_body_override: str | None = None,
    chunk_index_offset: int = 0,
+    progress_callback: "Callable[..., Awaitable[None]] | None" = None,
 ) -> tuple[list[list[str]], TokenUsage, int | None]:
    """
    Process a batch of content through the retain pipeline.
@@ -439,7 +460,9 @@ async def retain_batch(

    # Get bank profile
    profile = await bank_utils.get_bank_profile(pool, bank_id)
-    agent_name = profile["name"]
+    # Suppress the narrator when name == bank_id (auto-create default) — see
+    # _resolve_narrator for why a routing-key narrator pollutes extraction (#1680).
+    agent_name = _resolve_narrator(profile["name"], bank_id)

    # Convert dicts to RetainContent objects
    contents = _build_contents(contents_dicts, document_tags)
@@ -692,6 +715,7 @@ async def retain_batch(
        db_semaphore=db_semaphore,
        document_body_override=document_body_override,
        chunk_index_offset=chunk_index_offset,
+        progress_callback=progress_callback,
    )


@@ -831,6 +855,7 @@ async def _streaming_retain_batch(
    db_semaphore: "asyncio.Semaphore | None" = None,
    document_body_override: str | None = None,
    chunk_index_offset: int = 0,
+    progress_callback: "Callable[..., Awaitable[None]] | None" = None,
 ) -> tuple[list[list[str]], TokenUsage]:
    """
    Process a large document in streaming mini-batches to bound memory usage.
@@ -952,19 +977,29 @@ async def _streaming_retain_batch(
                tags=source.tags,
                observation_scopes=source.observation_scopes,
            )
-            extracted, processed, chunk_meta, usage = await _extract_and_embed(
-                [content],
-                llm_config,
-                agent_name,
-                config,
-                embeddings_model,
-                format_date_fn,
-                fact_type_override,
-                log_buffer,
-                pool,
-                operation_id,
-                schema,
-            )
+            # Attribute this chunk's extraction LLM call to its document, so the
+            # trace row carries document_id (a document accrues one such trace
+            # per retain/re-retain). Per-call: the operation-level trace context
+            # is shared across a batch's documents.
+            from ..llm_trace import reset_call_metadata, set_call_metadata
+
+            meta_token = set_call_metadata({"document_id": effective_doc_id})
+            try:
+                extracted, processed, chunk_meta, usage = await _extract_and_embed(
+                    [content],
+                    llm_config,
+                    agent_name,
+                    config,
+                    embeddings_model,
+                    format_date_fn,
+                    fact_type_override,
+                    log_buffer,
+                    pool,
+                    operation_id,
+                    schema,
+                )
+            finally:
+                reset_call_metadata(meta_token)
            await chunk_queue.put((global_idx, content, extracted, processed, chunk_meta, usage))
            # Memory: release the chunk text from the shared list now that it's
            # been extracted and queued. The queued RetainContent holds its own copy.
@@ -999,6 +1034,25 @@ async def _streaming_retain_batch(
    async def _db_consumer() -> None:
        batch: list[tuple] = []
        consumer_batch_idx = 0
+        chunks_committed = 0
+
+        # Best-effort durable progress: how many chunks of this document have been
+        # extracted+committed so far. Written per consumer batch so an operator polling
+        # the retain operation sees "storing 200/1200 chunks" advancing instead of a
+        # single opaque sub-batch tick. Never lets a heartbeat failure break retain.
+        async def _emit_chunk_progress() -> None:
+            if not (progress_callback and operation_id):
+                return
+            try:
+                await progress_callback(
+                    operation_id,
+                    stage="storing",
+                    processed=chunks_committed,
+                    total=total_chunks,
+                    detail={"facts_committed": len(all_unit_ids)},
+                )
+            except Exception:
+                logger.debug("retain chunk-progress write failed", exc_info=True)

        while True:
            item = await chunk_queue.get()
@@ -1010,6 +1064,8 @@ async def _streaming_retain_batch(
                        consumer_batch_idx,
                        is_last=True,
                    )
+                    chunks_committed += len(batch)
+                    await _emit_chunk_progress()
                break

            batch.append(item)
@@ -1029,6 +1085,8 @@ async def _streaming_retain_batch(
                    is_last=False,
                )
                consumer_batch_idx += 1
+                chunks_committed += len(batch)
+                await _emit_chunk_progress()
                batch = []

    async def _process_db_batch(
@@ -1181,20 +1239,17 @@ async def _streaming_retain_batch(
            async with acquire_with_retry(pool) as conn:
                async with conn.transaction():
                    # --- Document ownership gate ---
-                    # Lock the document row to serialize all concurrent writers.
-                    # SELECT ... FOR UPDATE doesn't lock non-existent rows, so we
-                    # first ensure the row exists with a lightweight upsert, THEN lock it.
-                    # The content_hash='__pending__' placeholder is immediately overwritten
-                    # by handle_document_tracking or upsert_document_metadata below.
-                    await conn.execute(
-                        f"INSERT INTO {fq_table('documents')} (id, bank_id, original_text, content_hash) "
-                        f"VALUES ($1, $2, '', '__pending__') "
-                        f"ON CONFLICT (id, bank_id) DO NOTHING",
-                        effective_doc_id,
-                        bank_id,
-                    )
-                    existing_hash = await conn.fetchval(
-                        f"SELECT content_hash FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2 FOR UPDATE",
+                    # Ensure the document row exists, lock it to serialize all
+                    # concurrent same-document writers, and read its pre-existing
+                    # hash. The lock prevents interleaved retains from corrupting
+                    # each other in handle_document_tracking; the returned hash
+                    # ('__pending__' for a freshly inserted row) drives the
+                    # takeover check for later batches below. The PG/Oracle split
+                    # lives in the ops layer because Oracle can't do this upsert +
+                    # RETURNING in a single statement.
+                    existing_hash = await pool.ops.lock_document_for_write(
+                        conn,
+                        fq_table("documents"),
                        effective_doc_id,
                        bank_id,
                    )
@@ -1501,6 +1556,35 @@ async def _streaming_retain_batch(
 # ---------------------------------------------------------------------------


+@dataclass
+class _ChunkDiff:
+    """Classification of chunk indices when diffing new content vs stored chunks."""
+
+    unchanged: list[int]
+    changed: list[int]
+    new: list[int]
+    removed: list[int]
+
+
+def _classify_chunk_diff(existing_by_index: dict[int, Any], new_hashes: dict[int, str]) -> _ChunkDiff:
+    """Classify chunk indices by comparing freshly computed ``new_hashes``
+    (index -> content hash) against the currently stored chunks
+    (``existing_by_index``: index -> chunk row)."""
+    diff = _ChunkDiff(unchanged=[], changed=[], new=[], removed=[])
+    for idx, new_hash in new_hashes.items():
+        existing = existing_by_index.get(idx)
+        if existing and existing.content_hash == new_hash:
+            diff.unchanged.append(idx)
+        elif existing:
+            diff.changed.append(idx)
+        else:
+            diff.new.append(idx)
+    for idx in existing_by_index:
+        if idx not in new_hashes:
+            diff.removed.append(idx)
+    return diff
+
+
 async def _try_delta_retain(
    pool: Any,
    embeddings_model,
@@ -1570,18 +1654,11 @@ async def _try_delta_retain(
    existing_by_index = {c.chunk_index: c for c in existing_chunks}
    new_hashes = {idx: chunk_storage.compute_chunk_hash(text) for idx, text in new_chunks_with_contents.items()}

-    unchanged_indices, changed_indices, new_indices, removed_indices = [], [], [], []
-    for idx, new_hash in new_hashes.items():
-        existing = existing_by_index.get(idx)
-        if existing and existing.content_hash == new_hash:
-            unchanged_indices.append(idx)
-        elif existing:
-            changed_indices.append(idx)
-        else:
-            new_indices.append(idx)
-    for idx in existing_by_index:
-        if idx not in new_hashes:
-            removed_indices.append(idx)
+    diff = _classify_chunk_diff(existing_by_index, new_hashes)
+    unchanged_indices = diff.unchanged
+    changed_indices = diff.changed
+    new_indices = diff.new
+    removed_indices = diff.removed

    log_buffer.append(
        f"[delta] Chunk diff: {len(unchanged_indices)} unchanged, "
@@ -1628,20 +1705,85 @@ async def _try_delta_retain(
            document_body_override=document_body_override,
        )

-    # Extract facts and generate embeddings (shared pipeline)
-    extracted_facts, processed_facts, new_chunk_metadata, usage = await _extract_and_embed(
-        delta_contents,
-        llm_config,
-        agent_name,
-        config,
-        embeddings_model,
-        format_date_fn,
-        fact_type_override,
-        log_buffer,
-        pool,
-        operation_id,
-        schema,
-    )
+    # Freshness recheck BEFORE the (expensive) LLM extraction.
+    #
+    # We snapshotted the document hash and chunks outside any lock. A concurrent
+    # retain for the same document may have committed a new version while we were
+    # chunking and diffing. Re-read the current hash; if it changed, recompute the
+    # diff against the now-committed chunk state. If the concurrent writer already
+    # produced content identical to ours, there is nothing left to extract — skip
+    # the LLM call entirely (metadata-only). If it still differs, fall back to the
+    # streaming path (which dedups per-chunk and re-locks the document).
+    #
+    # This narrows — but cannot fully close — the race window: a writer can still
+    # commit during our extraction. The post-extraction hash gate inside the write
+    # transaction remains the correctness backstop; this check exists purely to
+    # avoid burning LLM tokens on work a concurrent request already did.
+    async with acquire_with_retry(pool) as conn:
+        recheck_hash = await conn.fetchval(
+            f"SELECT content_hash FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
+            effective_doc_id,
+            bank_id,
+        )
+    if recheck_hash is not None and doc_hash_at_load is not None and recheck_hash != doc_hash_at_load:
+        log_buffer.append(
+            f"[delta] Document {effective_doc_id} changed before extraction "
+            f"(concurrent retain) — rechecking diff against current state"
+        )
+        async with acquire_with_retry(pool) as conn:
+            current_chunks = await chunk_storage.load_existing_chunks(conn, bank_id, effective_doc_id)
+        if not current_chunks or any(c.content_hash is None for c in current_chunks):
+            log_buffer.append("[delta] Recheck: current chunks unavailable — falling back to full retain")
+            logger.info("\n" + "\n".join(log_buffer) + "\n")
+            return None
+        current_by_index = {c.chunk_index: c for c in current_chunks}
+        recheck = _classify_chunk_diff(current_by_index, new_hashes)
+        if not (recheck.changed or recheck.new or recheck.removed):
+            log_buffer.append(
+                "[delta] Recheck: concurrent retain already stored identical content — "
+                "skipping extraction, updating metadata only"
+            )
+            return await _delta_metadata_only(
+                pool,
+                bank_id,
+                contents_dicts,
+                contents,
+                effective_doc_id,
+                document_tags,
+                log_buffer,
+                start_time,
+                outbox_callback,
+                document_body_override=document_body_override,
+            )
+        log_buffer.append(
+            f"[delta] Recheck: {len(recheck.changed) + len(recheck.new) + len(recheck.removed)} chunks still differ — "
+            f"falling back to full retain"
+        )
+        logger.info("\n" + "\n".join(log_buffer) + "\n")
+        return None
+
+    # Extract facts and generate embeddings (shared pipeline). Attribute these
+    # extraction calls to the document so the delta re-retain's trace also binds
+    # to it (a document accrues one trace per full/delta retain).
+    from ..llm_trace import reset_call_metadata, set_call_metadata
+
+    meta_token = set_call_metadata({"document_id": effective_doc_id})
+    try:
+        extracted_facts, processed_facts, new_chunk_metadata, usage = await _extract_and_embed(
+            delta_contents,
+            llm_config,
+            agent_name,
+            config,
+            embeddings_model,
+            format_date_fn,
+            fact_type_override,
+            log_buffer,
+            pool,
+            operation_id,
+            schema,
+        )
+    finally:
+        reset_call_metadata(meta_token)

    # Database transaction
    result_unit_ids: list[list[str]] = []
@@ -7,6 +7,27 @@ from typing import Any
 from .types import MergedCandidate, RetrievalResult


+def cap_per_source(results: list[RetrievalResult], cap: int) -> list[RetrievalResult]:
+    """Truncate a single retrieval arm to its top-``cap`` results.
+
+    Applied per source (semantic, BM25, graph, temporal) before fusion so that
+    one over-expanding backend cannot crowd out the others when the merged pool
+    is later trimmed to the reranker's global candidate budget. The caller is
+    responsible for sorting ``results`` by relevance first; this only slices.
+
+    Args:
+        results: Results for a single source, already sorted best-first.
+        cap: Maximum results to keep. ``0`` (or negative) disables the cap.
+
+    Returns:
+        The original list when the cap is disabled or not exceeded, otherwise a
+        truncated copy of the top ``cap`` results.
+    """
+    if cap <= 0 or len(results) <= cap:
+        return results
+    return results[:cap]
+
+
 def reciprocal_rank_fusion(result_lists: list[list[RetrievalResult]], k: int = 60) -> list[MergedCandidate]:
    """
    Merge multiple ranked result lists using Reciprocal Rank Fusion.
@@ -77,6 +98,66 @@ def reciprocal_rank_fusion(result_lists: list[list[RetrievalResult]], k: int = 6
    return merged_results


+def interleave_fusion(result_lists: list[list[RetrievalResult]]) -> list[MergedCandidate]:
+    """Round-robin (interleaved) fusion — an alternative to RRF for dedup-style recall.
+
+    RRF scores a doc by the *sum* of its reciprocal ranks across arms, so a result
+    that is #1 in one arm but absent/low in the others gets averaged down. That is
+    exactly the consolidation-dedup failure mode: the near-identical existing
+    observation (the "twin" to merge into) is semantic rank #1, yet shares no
+    source-fact graph link and little lexical overlap, so RRF drops it below the
+    recall budget cutoff and the LLM never sees it → creates a duplicate.
+
+    Interleave instead *guarantees every arm's top hits a slot*: take each arm's
+    #1, then each arm's #2, … in arm-priority order, de-duplicating, until all
+    results are placed. The arm priority is the order of ``result_lists``
+    (semantic, bm25, graph, temporal), so semantic #1 is always first.
+
+    ``rrf_score`` is assigned strictly decreasing by final interleave position so
+    downstream order-by-score sorts preserve the interleave order; ``source_ranks``
+    mirrors the RRF bookkeeping (each doc's rank within every arm it appears in).
+    """
+    source_names = ["semantic", "bm25", "graph", "temporal"]
+    source_ranks: dict[str, dict[str, int]] = {}
+    all_retrievals: dict[str, RetrievalResult] = {}
+
+    for source_idx, results in enumerate(result_lists):
+        source_name = source_names[source_idx] if source_idx < len(source_names) else f"source_{source_idx}"
+        for rank, retrieval in enumerate(results, start=1):
+            if not isinstance(retrieval, RetrievalResult):
+                raise TypeError(
+                    f"Expected RetrievalResult but got {type(retrieval).__name__} in {source_name} results at rank {rank}"
+                )
+            doc_id = retrieval.id
+            all_retrievals.setdefault(doc_id, retrieval)
+            source_ranks.setdefault(doc_id, {})[f"{source_name}_rank"] = rank
+
+    # Round-robin pick across arms in priority order: all #1s, then all #2s, ...
+    ordered_ids: list[str] = []
+    seen: set[str] = set()
+    max_len = max((len(r) for r in result_lists), default=0)
+    for r in range(max_len):
+        for results in result_lists:
+            if r < len(results):
+                doc_id = results[r].id
+                if doc_id not in seen:
+                    seen.add(doc_id)
+                    ordered_ids.append(doc_id)
+
+    n = len(ordered_ids)
+    return [
+        MergedCandidate(
+            retrieval=all_retrievals[doc_id],
+            # Strictly decreasing by interleave position → sorting desc by rrf_score
+            # reproduces the interleave order downstream.
+            rrf_score=float(n - pos),
+            rrf_rank=pos + 1,
+            source_ranks=source_ranks[doc_id],
+        )
+        for pos, doc_id in enumerate(ordered_ids)
+    ]
+
+
 def normalize_scores_on_deltas(results: list[dict[str, Any]], score_keys: list[str]) -> list[dict[str, Any]]:
    """
    Normalize scores based on deltas (min-max normalization within result set).
@@ -0,0 +1,117 @@
+"""Per-strategy recall boosting.
+
+A deployment can prioritise one retrieval arm (semantic, bm25, graph, temporal)
+over the others via ``HINDSIGHT_API_RECALL_STRATEGY_BOOSTS``, expressed as a
+human priority *level* rather than an opaque number — e.g. ``graph:high`` to
+strongly favour graph hits.
+
+A level is chosen instead of a raw weight because the boost is applied in two
+structurally different places that live on different score scales, so a single
+number could not mean the same thing in both. The level maps to a tuned
+:class:`BoostWeights` pair:
+
+1. **Before the reranker cap** — :func:`boosted_rrf_score` uses ``BoostWeights.rrf``
+   as a weighted-RRF multiplier on the boosted arm's rank contribution, so its
+   candidates survive the global reranker candidate budget instead of being
+   trimmed by raw RRF score. Rank-aware: a candidate ranked #1 in the boosted
+   arm is protected more than one ranked #200.
+
+2. **After the reranker** — :func:`additive_strategy_boost` uses
+   ``BoostWeights.additive`` as a flat bump to the final ranking weight (which
+   sits in ~[0, 1] after cross-encoder + recency/temporal scoring), nudging the
+   boosted arm's candidates up the final ordering.
+
+Both functions are no-ops when ``boosts`` is empty, preserving current behaviour.
+"""
+
+from dataclasses import dataclass
+
+from .types import MergedCandidate
+
+
+@dataclass(frozen=True)
+class BoostWeights:
+    """Per-stage boost magnitudes for one priority level.
+
+    The two fields live on different scales on purpose (see module docstring):
+    ``rrf`` multiplies an arm's ``1/(k+rank)`` RRF contribution; ``additive`` is
+    added directly to the post-rerank weight in ~[0, 1].
+    """
+
+    rrf: float
+    additive: float
+
+
+# Priority level -> per-stage boost magnitudes. Tuned against real recall traces
+# (LoCoMo bank, 336 merged candidates → 300-cap, local ms-marco cross-encoder):
+#
+# Stage 1 (rrf, weighted-RRF multiplier on the arm's 1/(k+rank) contribution).
+# The observed 300-cap boundary RRF score was ~0.0055; a graph-only candidate
+# falls below it past graph-rank ~120. The multipliers map to that boundary:
+#   low=1.0   doubles the arm's vote — rescues at-risk candidates from the cut
+#             (graph-rank 150: 0.0048 → 0.0095) without reshuffling much.
+#   medium=3.0 promotes them into the middle of the pool (~rank 60).
+#   high=6.0   makes the boosted arm dominate the top of the candidate pool.
+#
+# Stage 2 (additive, flat bump to the post-rerank weight in [0, 1]). The local
+# cross-encoder is sharply bimodal: strong direct matches score 0.5–0.999, while
+# everything else — including graph hits the CE undervalues, which is exactly
+# what we boost — collapses near 0. So the additive lifts a ~0 candidate up the
+# weight scale. Levels are calibrated as relevance thresholds it can outrank:
+#   low=0.05  nudges above the near-0 tail; loses to any real CE match.
+#   medium=0.2 competes with weak/moderate matches.
+#   high=0.5  wins over most semantic matches (honouring "prioritise graph over
+#             semantic"); only a strong direct match (>0.5 normalized) still wins.
+#
+# The keys are the user-facing contract; config.py validates env input against
+# them (kept in sync by a guard test).
+BOOST_LEVELS: dict[str, BoostWeights] = {
+    "low": BoostWeights(rrf=1.0, additive=0.05),
+    "medium": BoostWeights(rrf=3.0, additive=0.2),
+    "high": BoostWeights(rrf=6.0, additive=0.5),
+}
+
+
+def boosted_rrf_score(candidate: MergedCandidate, boosts: dict[str, str], k: int = 60) -> float:
+    """Return ``candidate``'s RRF score plus a weighted-RRF boost delta.
+
+    For each boosted arm the candidate appeared in, adds ``level.rrf * 1/(k+rank)``
+    — i.e. scales that arm's RRF contribution by the level's multiplier. Staying
+    in RRF units keeps the boost comparable to the base score and rank-aware.
+
+    Args:
+        candidate: Merged candidate carrying ``rrf_score`` and ``source_ranks``.
+        boosts: Map of strategy name -> priority level. Empty means no boost.
+        k: RRF constant; must match the value used during fusion.
+
+    Returns:
+        The (possibly) boosted score to sort by. Equal to ``rrf_score`` when no
+        boosted arm surfaced this candidate.
+    """
+    if not boosts:
+        return candidate.rrf_score
+    delta = 0.0
+    for strategy, level in boosts.items():
+        rank = candidate.source_ranks.get(f"{strategy}_rank")
+        if rank is not None:
+            delta += BOOST_LEVELS[level].rrf * (1.0 / (k + rank))
+    return candidate.rrf_score + delta
+
+
+def additive_strategy_boost(source_ranks: dict[str, int], boosts: dict[str, str]) -> float:
+    """Return the flat additive boost for a candidate given its source ranks.
+
+    Sums the ``additive`` magnitude of every boosted arm that surfaced the
+    candidate. Flat by design: the bump does not depend on the candidate's rank
+    within the arm, matching the post-rerank "additive boost" semantics.
+
+    Args:
+        source_ranks: ``{"graph_rank": 3, "semantic_rank": 50, ...}`` from RRF.
+        boosts: Map of strategy name -> priority level. Empty means no boost.
+
+    Returns:
+        The additive boost (0.0 when no boosted arm surfaced this candidate).
+    """
+    if not boosts:
+        return 0.0
+    return sum(BOOST_LEVELS[level].additive for strategy, level in boosts.items() if f"{strategy}_rank" in source_ranks)
@@ -160,13 +160,29 @@ class CrossEncoderReranker:

        import asyncio

+        from hindsight_api.config import ENV_MODEL_INIT_TIMEOUT, get_config
+
        cross_encoder = self.cross_encoder
        # For local providers, run in thread pool to avoid blocking event loop
        if cross_encoder.provider_name == "local":
            loop = asyncio.get_event_loop()
-            await loop.run_in_executor(None, lambda: asyncio.run(cross_encoder.initialize()))
+            init = loop.run_in_executor(None, lambda: asyncio.run(cross_encoder.initialize()))
        else:
-            await cross_encoder.initialize()
+            init = cross_encoder.initialize()
+
+        # Cap lazy init with the same wall-clock timeout used at startup so a
+        # hung model download surfaces as a clear error on the request that
+        # triggered it, rather than hanging the caller forever.
+        init_timeout = get_config().model_init_timeout
+        try:
+            await asyncio.wait_for(init, timeout=init_timeout)
+        except TimeoutError as e:
+            raise RuntimeError(
+                f"Cross-encoder initialization did not complete within {init_timeout:g}s. "
+                f"The reranker model is likely blocked loading — e.g. an offline model "
+                f"download. Increase {ENV_MODEL_INIT_TIMEOUT} if the first-time download "
+                f"legitimately needs more time."
+            ) from e
        self._initialized = True

    async def rerank(self, query: str, candidates: list[MergedCandidate]) -> list[ScoredResult]:
@@ -137,6 +137,7 @@ async def retrieve_semantic_bm25_combined(
    """
    result_dict: dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]] = {ft: ([], []) for ft in fact_types}

+    config = get_config()
    tokens = tokenize_query(query_text)

    # Over-fetch for HNSW approximation; semantic results trimmed to limit in Python.
@@ -148,8 +149,6 @@ async def retrieve_semantic_bm25_combined(
    )
    table = fq_table("memory_units")

-    config = get_config()
-
    # Use the SQL dialect to build backend-specific query arms, avoiding
    # inline if/else branches for each database.
    # Use getattr for backward compat: raw asyncpg connections (used in some
@@ -201,6 +200,7 @@ async def retrieve_semantic_bm25_combined(
            embedding_param="$1",
            bank_id_param="$2",
            fetch_limit=hnsw_fetch,
+            min_similarity=config.semantic_min_similarity,
            tags_clause=tags_clause,
            groups_clause=groups_clause,
            extra_where=created_range_clause,
@@ -226,6 +226,7 @@ async def retrieve_semantic_bm25_combined(
                    arm_index=i,
                    text_search_extension=text_ext,
                    bm25_language=config.text_search_extension_native_language,
+                    bm25_min_score=config.bm25_min_score,
                    extra_where=created_range_clause,
                )
            )
@@ -273,6 +274,7 @@ async def retrieve_semantic_bm25_combined(
                    embedding_param="$1",
                    bank_id_param="$2",
                    fetch_limit=hnsw_fetch,
+                    min_similarity=config.semantic_min_similarity,
                    tags_clause=fb_tags_clause,
                    groups_clause=fb_groups_clause,
                    extra_where=fb_created_clause,
@@ -307,6 +309,66 @@ async def retrieve_semantic_bm25_combined(
    return result_dict


+# Temporal entry-point selection tuning.
+_TEMPORAL_POOL_SIZE = 60  # ANN candidates fetched per fact_type before coverage selection
+_TEMPORAL_ENTRY_POINTS = 10  # entry points kept per fact_type after coverage selection
+_TEMPORAL_COVERAGE_BUCKETS = 8  # time-buckets the window is divided into for coverage
+
+
+def _coalesce_date(row: Any) -> datetime | None:
+    """The unit's effective time — matches COALESCE(occurred_start, mentioned_at, occurred_end)."""
+    return row["occurred_start"] or row["mentioned_at"] or row["occurred_end"]
+
+
+def _select_with_temporal_coverage(
+    pool: list,
+    start_date: datetime,
+    end_date: datetime,
+    limit: int,
+    n_buckets: int,
+) -> list:
+    """Pick `limit` entry points from a similarity-ranked pool, spread across the window.
+
+    The window [start_date, end_date] is split into `n_buckets` equal time-buckets.
+    Candidates are taken round-robin across the buckets that contain them — the
+    best-similarity item from each populated bucket first, then the second-best from each,
+    and so on — so every populated slice of the window is represented before any slice
+    contributes a second item. Within a tier, higher-similarity items lead. When the
+    in-window dates are degenerate (all in one bucket — e.g. a batch stamped with a single
+    date) this collapses to plain similarity order.
+    """
+    if len(pool) <= limit:
+        return list(pool)
+
+    ranked = sorted(pool, key=lambda r: r["similarity"], reverse=True)
+    span = (end_date - start_date).total_seconds()
+
+    def _bucket(row: Any) -> int:
+        d = _coalesce_date(row)
+        if d is None or span <= 0:
+            return 0
+        if d.tzinfo is None:
+            d = d.replace(tzinfo=UTC)
+        frac = (d - start_date).total_seconds() / span
+        return max(0, min(int(frac * n_buckets), n_buckets - 1))
+
+    buckets: dict[int, list] = {}
+    for row in ranked:  # ranked is similarity-desc, so each bucket list inherits that order
+        buckets.setdefault(_bucket(row), []).append(row)
+
+    selected: list = []
+    tier = 0
+    while len(selected) < limit and any(len(b) > tier for b in buckets.values()):
+        # The tier-th best item from every bucket that still has one, strongest first.
+        tier_rows = [b[tier] for b in buckets.values() if len(b) > tier]
+        tier_rows.sort(key=lambda r: r["similarity"], reverse=True)
+        for row in tier_rows:
+            if len(selected) < limit:
+                selected.append(row)
+        tier += 1
+    return selected
+
+
 async def retrieve_temporal_combined(
    conn,
    query_emb_str: str,
@@ -350,9 +412,12 @@ async def retrieve_temporal_combined(
        end_date = end_date.replace(tzinfo=UTC)

    # Build tags clause
-    # Entry point query: fixed params are $1-$6, tags at $7
-    tags_clause = build_tags_where_clause_simple(tags, 7, match=tags_match)
-    tag_groups_param_start = 7 + (1 if tags else 0)
+    # Entry-point query: fixed params are $1-$5 (emb, bank, start, end, threshold), tags at $6.
+    # fact_type is inlined as a literal per UNION ALL arm (not a bind) — this avoids `unnest`,
+    # which has no Oracle equivalent (the `<=>` operator and LIMIT are translated to Oracle by
+    # the backend on execute, but `unnest` is not). Mirrors retrieve_semantic_bm25_combined.
+    tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
+    tag_groups_param_start = 6 + (1 if tags else 0)
    groups_clause, groups_params, _ = build_tag_groups_where_clause(tag_groups, tag_groups_param_start)

    # created_at time range filter (after tags/groups)
@@ -368,69 +433,88 @@ async def retrieve_temporal_combined(
        created_range_clause += f" AND updated_at < ${_next_idx}"
        _next_idx += 1

-    params: list = [query_emb_str, bank_id, fact_types, start_date, end_date, semantic_threshold]
+    params: list = [query_emb_str, bank_id, start_date, end_date, semantic_threshold]
    if tags:
        params.append(tags)
    params.extend(groups_params)
    params.extend(created_range_params)

-    # Two-phase entry point query:
-    # Phase 1 (date_ranked): rank by date only — no embedding computation — for all units in
-    #   the temporal window. This lets the planner use date indexes for filtering.
-    # Phase 2 (sim_ranked): join back to memory_units for only the top-50-per-type candidates
-    #   and compute embedding similarity for that small set (≤ 50 × len(fact_types) rows).
-    # This avoids computing embedding distances for potentially thousands of date-range rows.
-    entry_points = await conn.fetch(
-        f"""
-        WITH date_ranked AS MATERIALIZED (
-            SELECT id, fact_type,
-                   ROW_NUMBER() OVER (
-                       PARTITION BY fact_type
-                       ORDER BY COALESCE(occurred_start, mentioned_at, occurred_end) DESC NULLS LAST
-                   ) AS rn
-            FROM {fq_table("memory_units")}
-            WHERE bank_id = $2
-              AND fact_type = ANY($3)
-              AND embedding IS NOT NULL
-              AND (
-                  (occurred_start IS NOT NULL AND occurred_end IS NOT NULL
-                   AND occurred_start <= $5 AND occurred_end >= $4)
-                  OR
-                  (mentioned_at IS NOT NULL AND mentioned_at BETWEEN $4 AND $5)
-                  OR
-                  (occurred_start IS NOT NULL AND occurred_start BETWEEN $4 AND $5)
-                  OR
-                  (occurred_end IS NOT NULL AND occurred_end BETWEEN $4 AND $5)
-              )
-              {tags_clause}
-              {groups_clause}
-              {created_range_clause}
-        ),
-        sim_ranked AS (
-            SELECT mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start, mu.occurred_end, mu.mentioned_at, mu.fact_type, mu.proof_count, mu.document_id, mu.chunk_id, mu.tags, mu.metadata,
-                   1 - (mu.embedding <=> $1::vector) AS similarity,
-                   ROW_NUMBER() OVER (PARTITION BY mu.fact_type ORDER BY mu.embedding <=> $1::vector) AS sim_rn
-            FROM date_ranked dr
-            JOIN {fq_table("memory_units")} mu ON mu.id = dr.id
-            WHERE dr.rn <= 50
-              AND (1 - (mu.embedding <=> $1::vector)) >= $6
-        )
-        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, fact_type, proof_count, document_id, chunk_id, tags, metadata, similarity
-        FROM sim_ranked
-        WHERE sim_rn <= 10
-        """,
-        *params,
-    )
+    # Entry-point selection: similarity-gated, window-filtered, then narrowed for coverage.
+    #
+    # For each fact_type, ANN-rank the units whose time overlaps the window
+    # (ORDER BY embedding <=> query) and keep a pool of the most relevant
+    # (_TEMPORAL_POOL_SIZE). The planner serves this from the per-(bank, fact_type) vector
+    # index when the window is broad — the dense-metadata case, where the window matches
+    # most rows — and from the partial date indexes plus an exact sort when the window is
+    # narrow. Either way the work is bounded; neither path is a scan-and-sort of the whole
+    # match set.
+    #
+    # Selecting by *similarity* (not recency) is deliberate. The earlier form ranked the
+    # entire match set by COALESCE(occurred_start, mentioned_at, occurred_end) and kept the
+    # 50 most recent: that biased results toward the end of the window and, on banks with
+    # dense/near-uniform dates (e.g. a retain batch stamped with one date), the date key was
+    # degenerate so the "50 most recent" became a near-random sample that could drop the
+    # single most relevant in-window memory — and it degraded to a full scan + disk-spilling
+    # sort (30s+ on a 660k-row bank). The pool is then narrowed to _TEMPORAL_ENTRY_POINTS per
+    # fact_type by _select_with_temporal_coverage so the entry points span the window's range
+    # rather than clustering in one slice.
+    if not fact_types:
+        return {}

-    if not entry_points:
+    # One similarity-ranked, window-filtered arm per fact_type, UNION ALL'd — each arm has its
+    # own ORDER BY ... LIMIT so the per-(bank, fact_type) vector index can serve it. fact_type
+    # is inlined as a literal (controlled internal enum, never user input), matching
+    # retrieve_semantic_bm25_combined; this keeps the query free of `unnest`/LATERAL, which the
+    # Oracle backend cannot translate.
+    pool_cols = (
+        "id, text, context, event_date, occurred_start, occurred_end, mentioned_at, "
+        "fact_type, proof_count, document_id, chunk_id, tags, metadata"
+    )
+    table = fq_table("memory_units")
+    arms = [
+        f"""(
+        SELECT {pool_cols}, 1 - (embedding <=> $1::vector) AS similarity
+        FROM {table}
+        WHERE bank_id = $2
+          AND fact_type = '{ft}'
+          AND embedding IS NOT NULL
+          AND (
+              (occurred_start IS NOT NULL AND occurred_end IS NOT NULL
+               AND occurred_start <= $4 AND occurred_end >= $3)
+              OR
+              (mentioned_at IS NOT NULL AND mentioned_at BETWEEN $3 AND $4)
+              OR
+              (occurred_start IS NOT NULL AND occurred_start BETWEEN $3 AND $4)
+              OR
+              (occurred_end IS NOT NULL AND occurred_end BETWEEN $3 AND $4)
+          )
+          AND (1 - (embedding <=> $1::vector)) >= $5
+          {tags_clause}
+          {groups_clause}
+          {created_range_clause}
+        ORDER BY embedding <=> $1::vector
+        LIMIT {_TEMPORAL_POOL_SIZE}
+        )"""
+        for ft in fact_types
+    ]
+    pool_rows = await conn.fetch("\nUNION ALL\n".join(arms), *params)
+
+    if not pool_rows:
        return {ft: [] for ft in fact_types}

-    # Group entry points by fact type
-    entries_by_ft: dict[str, list] = {ft: [] for ft in fact_types}
-    for ep in entry_points:
-        ft = ep["fact_type"]
-        if ft in entries_by_ft:
-            entries_by_ft[ft].append(ep)
+    # Group the ANN pool by fact type, then narrow each to coverage-spread entry points.
+    pool_by_ft: dict[str, list] = {ft: [] for ft in fact_types}
+    for row in pool_rows:
+        ft = row["fact_type"]
+        if ft in pool_by_ft:
+            pool_by_ft[ft].append(row)
+
+    entries_by_ft: dict[str, list] = {
+        ft: _select_with_temporal_coverage(
+            rows, start_date, end_date, _TEMPORAL_ENTRY_POINTS, _TEMPORAL_COVERAGE_BUCKETS
+        )
+        for ft, rows in pool_by_ft.items()
+    }

    # Calculate shared temporal parameters
    total_days = (end_date - start_date).total_seconds() / 86400
@@ -498,7 +582,13 @@ async def retrieve_temporal_combined(
            tag_groups, spreading_groups_param_start, table_alias="mu."
        )

-        while frontier and budget_remaining > 0 and iteration < max_iterations:
+        # Multi-hop temporal spreading expands a batch of seed ids with
+        # ``FROM unnest($2::uuid[])``, which has no Oracle equivalent. On backends
+        # without unnest, skip the spread: the temporal entry points are still
+        # returned above, and the semantic/keyword/graph retrievers cover the rest.
+        supports_unnest = getattr(conn, "backend_type", "postgresql") != "oracle"
+
+        while frontier and budget_remaining > 0 and iteration < max_iterations and supports_unnest:
            iteration += 1
            batch_ids = frontier[:batch_size]
            frontier = frontier[batch_size:]
@@ -358,12 +358,15 @@ class SearchTracer:
        """
        self.rrf_merged = []
        for rank, (doc_id, data, rrf_meta) in enumerate(merged_results, start=1):
+            source_ranks = rrf_meta.get("source_ranks")
+            if source_ranks is None:
+                source_ranks = {key: value for key, value in rrf_meta.items() if key.endswith("_rank")}
            self.rrf_merged.append(
                RRFMergeResult(
                    node_id=doc_id,
                    text=data.get("text", ""),
                    rrf_score=rrf_meta.get("rrf_score", 0.0),
-                    source_ranks=rrf_meta.get("source_ranks", {}),
+                    source_ranks=source_ranks,
                    final_rrf_rank=rank,
                )
            )
@@ -371,6 +371,7 @@ class SQLDialect(ABC):
        embedding_param: str,
        bank_id_param: str,
        fetch_limit: int,
+        min_similarity: float,
        tags_clause: str = "",
        groups_clause: str = "",
        extra_where: str = "",
@@ -387,6 +388,7 @@ class SQLDialect(ABC):
            embedding_param: Parameter placeholder for query embedding.
            bank_id_param: Parameter placeholder for bank_id.
            fetch_limit: Max rows to fetch (over-fetched for HNSW approximation).
+            min_similarity: Minimum cosine similarity to include.
            tags_clause: Optional WHERE clause fragment for tag filtering.
            groups_clause: Optional WHERE clause fragment for tag group filtering.
            extra_where: Optional additional WHERE clause fragment (e.g. time range filter).
@@ -408,6 +410,7 @@ class SQLDialect(ABC):
        arm_index: int = 0,
        text_search_extension: str = "native",
        bm25_language: str = "english",
+        bm25_min_score: float = 0.0,
        extra_where: str = "",
    ) -> str:
        """Build a BM25/full-text search subquery arm.
@@ -430,6 +433,11 @@ class SQLDialect(ABC):
                                   "pg_textsearch", "pgroonga"). Only relevant for PostgreSQL.
            bm25_language: PostgreSQL text search dictionary used by the native
                           backend (e.g. "english", "french"). Ignored by other backends.
+            bm25_min_score: Minimum BM25 relevance score a row must exceed to be
+                            returned. Gates out non-matching rows on backends whose
+                            operator (e.g. VectorChord) ranks every document instead
+                            of pre-filtering to query-term matches. Backends that
+                            already apply a boolean match gate ignore this.
            extra_where: Optional additional WHERE clause fragment (e.g. time range filter).
        """
        ...
@@ -234,6 +234,7 @@ class OracleDialect(SQLDialect):
        embedding_param: str,
        bank_id_param: str,
        fetch_limit: int,
+        min_similarity: float,
        tags_clause: str = "",
        groups_clause: str = "",
        extra_where: str = "",
@@ -249,7 +250,7 @@ class OracleDialect(SQLDialect):
            f" WHERE bank_id = {bank_id_param}"
            f"   AND fact_type = '{fact_type}'"
            f"   AND embedding IS NOT NULL"
-            f"   AND (1 - VECTOR_DISTANCE(embedding, {embedding_param}, COSINE)) >= 0.3"
+            f"   AND (1 - VECTOR_DISTANCE(embedding, {embedding_param}, COSINE)) >= {min_similarity}"
            f"   {tags_clause}"
            f"   {groups_clause}"
            f"   {extra_where}"
@@ -271,6 +272,7 @@ class OracleDialect(SQLDialect):
        arm_index: int = 0,
        text_search_extension: str = "native",
        bm25_language: str = "english",
+        bm25_min_score: float = 0.0,
        extra_where: str = "",
    ) -> str:
        # Oracle Text: CONTAINS() / SCORE() with the CTXSYS.CONTEXT index.
@@ -285,7 +287,9 @@ class OracleDialect(SQLDialect):
            f" FROM {table}"
            f" WHERE bank_id = {bank_id_param}"
            f"   AND fact_type = '{fact_type}'"
-            f"   AND CONTAINS(text, {text_param}, {label}) > 0"
+            # CONTAINS already gates to genuine matches; the configurable floor
+            # (default 0) keeps the threshold semantics uniform across backends.
+            f"   AND CONTAINS(text, {text_param}, {label}) > {bm25_min_score:g}"
            f"   {tags_clause}"
            f"   {groups_clause}"
            f"   {extra_where}"
@@ -148,6 +148,7 @@ class PostgreSQLDialect(SQLDialect):
        embedding_param: str,
        bank_id_param: str,
        fetch_limit: int,
+        min_similarity: float,
        tags_clause: str = "",
        groups_clause: str = "",
        extra_where: str = "",
@@ -161,7 +162,7 @@ class PostgreSQLDialect(SQLDialect):
            f" WHERE bank_id = {bank_id_param}"
            f"   AND fact_type = '{fact_type}'"
            f"   AND embedding IS NOT NULL"
-            f"   AND (1 - (embedding <=> {embedding_param}::vector)) >= 0.3"
+            f"   AND (1 - (embedding <=> {embedding_param}::vector)) >= {min_similarity}"
            f"   {tags_clause}"
            f"   {groups_clause}"
            f"   {extra_where}"
@@ -183,25 +184,32 @@ class PostgreSQLDialect(SQLDialect):
        arm_index: int = 0,
        text_search_extension: str = "native",
        bm25_language: str = "english",
+        bm25_min_score: float = 0.0,
        extra_where: str = "",
    ) -> str:
        if text_search_extension == "vchord":
-            # <&> returns a distance (lower = more relevant), negate for score
+            # <&> returns the NEGATIVE BM25 score (lower = more relevant), negate
+            # for a positive score where higher = more relevant.
            bm25_score_expr = f"-(search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize({text_param}, 'llmlingua2')))"
            bm25_order_by = f"{bm25_score_expr} DESC"
-            bm25_where_filter = ""
+            # Unlike native tsvector (which has a boolean `@@` match gate), the
+            # VectorChord operator ranks *every* document, so a bare ORDER BY ...
+            # LIMIT pads the result with zero-score, non-matching rows. Gate on the
+            # score so only genuine term matches survive into fusion/reranking.
+            bm25_where_filter = f"AND -(search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize({text_param}, 'llmlingua2'))) > {bm25_min_score:g}"
        elif text_search_extension == "pg_textsearch":
            bm25_score_expr = f"-({text_param} <@> to_bm25query({text_param}, 'idx_memory_units_text_search'))"
            bm25_order_by = f"text <@> to_bm25query({text_param}, 'idx_memory_units_text_search') ASC"
            bm25_where_filter = ""
        elif text_search_extension == "pgroonga":
-            # &@~ accepts pgroonga's query syntax (raw query text). pgroonga_score
-            # returns a non-negative relevance score (higher = better).
+            # &@~ accepts pgroonga's query syntax. Escape the bind parameter so
+            # literal memory text containing operators like ">" or "(" is not
+            # parsed as a malformed query expression.
            bm25_score_expr = "pgroonga_score(tableoid, ctid)"
            bm25_order_by = f"{bm25_score_expr} DESC"
            bm25_where_filter = (
                f"AND (COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, '')) "
-                f"&@~ {text_param}"
+                f"&@~ pgroonga_query_escape({text_param})"
            )
        elif text_search_extension == "pg_search":
            # ParadeDB pg_search: BM25 index over (id, text, context, text_signals)
@@ -0,0 +1,43 @@
+"""Document transfer: export/import documents between banks without re-running the LLM.
+
+An export is a ZIP of already-extracted facts (text, entities by canonical name,
+causal relations, chunks) — never embeddings or DB ids. An import replays the
+deterministic half of the retain pipeline against the target bank: it re-embeds
+locally with the target bank's embedding model, re-resolves entities, and
+recreates temporal/semantic/causal links relative to the target bank's existing
+memories. No LLM fact-extraction is involved.
+
+Consolidated observations (``fact_type='observation'``) are intentionally
+excluded from export — they are derived by consolidation and are regenerated in
+the target bank.
+"""
+
+from .export import export_bank, export_documents
+from .importer import BankImportResult, ImportResult, import_bank, import_documents
+from .schema import (
+    SCHEMA_VERSION,
+    TransferCausalRelation,
+    TransferChunk,
+    TransferDocument,
+    TransferFact,
+    TransferManifest,
+    TransferObservation,
+    TransferObservationSource,
+)
+
+__all__ = [
+    "SCHEMA_VERSION",
+    "BankImportResult",
+    "ImportResult",
+    "TransferCausalRelation",
+    "TransferChunk",
+    "TransferDocument",
+    "TransferFact",
+    "TransferManifest",
+    "TransferObservation",
+    "TransferObservationSource",
+    "export_bank",
+    "export_documents",
+    "import_bank",
+    "import_documents",
+]
@@ -0,0 +1,555 @@
+"""Export documents (with extracted facts, entities, causal links, chunks) to a ZIP archive.
+
+Reads directly from the database via the backend connection. Embeddings and
+database ids are deliberately omitted — they are regenerated/re-resolved on
+import. Consolidated observations are excluded unless ``include_observations``
+is set, in which case they are written to ``observations.json``.
+"""
+
+from __future__ import annotations
+
+import base64
+import io
+import json
+import logging
+import zipfile
+from dataclasses import dataclass, field
+from datetime import UTC, date, datetime
+from decimal import Decimal
+from typing import Any
+from uuid import UUID
+
+from ..db_utils import acquire_with_retry
+from ..schema import fq_table
+from .schema import (
+    SCHEMA_VERSION,
+    TransferCausalRelation,
+    TransferChunk,
+    TransferDocument,
+    TransferFact,
+    TransferManifest,
+    TransferObservation,
+    TransferObservationSource,
+)
+
+logger = logging.getLogger(__name__)
+
+# Whole-bank export classification. Every bank-scoped table (admin.cli.BACKUP_TABLES)
+# must fall into exactly one bucket below; tests/test_document_transfer.py's
+# test_export_bank_covers_schema enforces this so a table added by a future
+# migration can't be silently dropped from a migration archive.
+
+# NOT written to the archive — rebuilt on import by replaying the document/fact/
+# observation payload through the import pipeline:
+#   * documents / chunks / memory_units carry their *text* in the logical document
+#     payload (TransferDocument) and are re-embedded with the target model;
+#   * entities / unit_entities / memory_links / entity_cooccurrences are derived
+#     data — the pipeline re-resolves entities and rebuilds links/cooccurrence
+#     stats against the target bank, so they are never exported.
+# Listed here only so the coverage guard can assert every table is classified.
+_REPLAYED_TABLES = frozenset(
+    {
+        "documents",
+        "chunks",
+        "memory_units",
+        "entities",
+        "unit_entities",
+        "memory_links",
+        "entity_cooccurrences",
+        # observation_history FKs to a memory_units observation, but observations
+        # are derived: they're regenerated with FRESH ids when consolidation is
+        # replayed on import (see _EXPORTED_FACT_TYPES — observations are excluded).
+        # There is no stable observation id to re-attach history to, so it is not
+        # carried; the target rebuilds observation history as it re-consolidates.
+        "observation_history",
+    }
+)
+# Carried verbatim as JSON rows (bank config + synthesized state). Embedding-bearing
+# rows have their vector stripped (see _DERIVED_COLUMNS) and are re-embedded on import.
+_BANK_ROW_TABLES = ("banks", "mental_models", "directives", "webhooks")
+# Bank-scoped child-history carried verbatim. Unlike observations, mental models
+# keep their (id, bank_id) across export/import, so their refresh history can be
+# re-attached. The surrogate ``id`` is dropped on dump so the target reassigns it
+# (see _dump_history_rows); restored after its parent table (mental_models).
+_CARRIED_HISTORY_TABLES = ("mental_model_history",)
+# Operational history — only carried with include_history=True.
+_HISTORY_TABLES = ("audit_log", "llm_requests")
+# Intentionally never exported.
+_SKIP_TABLES = frozenset(
+    {
+        "async_operations",  # in-flight ops; drain on the source before migrating
+        "graph_maintenance_queue",  # transient work queue; regenerated on import
+        "file_storage",  # raw uploads; documents.original_text is already carried
+    }
+)
+# Derived columns dropped from carried rows so the target regenerates them with
+# its own embedding model / text-search backend.
+_DERIVED_COLUMNS = ("embedding", "search_vector")
+
+
+@dataclass
+class _UnitLocation:
+    """Where a memory unit's fact lives in the assembled export (document + ordinal)."""
+
+    document_id: str
+    ordinal: int
+
+
+@dataclass
+class _LoadedFacts:
+    """Facts grouped by document plus an index from unit id to its location.
+
+    ``facts_by_doc`` and ``unit_index`` share the same fixed ordering so that
+    causal ``target_fact_index`` ordinals stay consistent across both.
+    """
+
+    facts_by_doc: dict[str, list[TransferFact]] = field(default_factory=dict)
+    unit_index: dict[Any, _UnitLocation] = field(default_factory=dict)
+
+
+@dataclass
+class _LoadedExport:
+    """Assembled documents plus the unit-id → location index.
+
+    ``unit_index`` is retained so observation source unit ids can be resolved to
+    (document_id, fact_index) references when observations are exported.
+    """
+
+    documents: list[TransferDocument] = field(default_factory=list)
+    unit_index: dict[Any, _UnitLocation] = field(default_factory=dict)
+
+
+# Causal link types that retain persists between facts. Only these travel in the
+# archive; temporal/semantic/entity links are regenerated against the target bank.
+_CAUSAL_LINK_TYPES = ("caused_by", "causes", "enables", "prevents")
+
+# Facts of these types are exported; observations are derived and excluded.
+_EXPORTED_FACT_TYPES = ("world", "experience")
+
+
+def _as_jsonb(value: Any) -> Any:
+    """Coerce an asyncpg JSONB column (str or already-decoded) to a Python object."""
+    if value is None:
+        return None
+    if isinstance(value, str):
+        return json.loads(value)
+    return value
+
+
+def _chunk_index_from_chunk_id(chunk_id: str | None) -> int | None:
+    """Recover the chunk ordinal from a ``{bank_id}_{document_id}_{index}`` chunk_id.
+
+    The index is always the final underscore-delimited segment, so rsplit is
+    correct even when bank/document ids themselves contain underscores.
+    """
+    if not chunk_id:
+        return None
+    try:
+        return int(chunk_id.rsplit("_", 1)[1])
+    except (IndexError, ValueError):
+        return None
+
+
+async def export_documents(
+    backend: Any,
+    bank_id: str,
+    document_ids: list[str] | None = None,
+    *,
+    include_observations: bool = False,
+) -> bytes:
+    """Export documents from ``bank_id`` into an in-memory ZIP archive.
+
+    Args:
+        backend: Database backend (provides ``acquire()``).
+        bank_id: Source bank.
+        document_ids: Specific document ids to export. ``None`` exports every
+            document in the bank.
+        include_observations: Also export consolidated observations (written to
+            ``observations.json``). Only valid for a whole-bank export.
+
+    Returns:
+        The ZIP archive as bytes.
+
+    Raises:
+        ValueError: if ``include_observations`` is combined with ``document_ids``.
+    """
+    # Observations are bank-level and can be derived from facts spanning several
+    # documents, so they're only coherent when the whole bank is exported. For a
+    # document subset we'd have to silently drop every cross-document observation
+    # — reject the combination instead so the caller isn't surprised.
+    if include_observations and document_ids is not None:
+        raise ValueError("include_observations is only supported when exporting the whole bank (omit document_id)")
+
+    async with acquire_with_retry(backend) as conn:
+        loaded = await _load_documents(conn, bank_id, document_ids)
+        documents = loaded.documents
+        observations = await _load_observations(conn, bank_id, loaded.unit_index) if include_observations else []
+
+    archive = io.BytesIO()
+    fact_total = 0
+    with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
+        for index, document in enumerate(documents):
+            fact_total += len(document.facts)
+            zf.writestr(
+                f"documents/{index:06d}.json",
+                document.model_dump_json(indent=2, exclude_none=False),
+            )
+
+        if observations:
+            payload = "[\n" + ",\n".join(o.model_dump_json(indent=2) for o in observations) + "\n]\n"
+            zf.writestr("observations.json", payload)
+
+        manifest = TransferManifest(
+            schema_version=SCHEMA_VERSION,
+            source_bank_id=bank_id,
+            exported_at=datetime.now(UTC),
+            document_count=len(documents),
+            fact_count=fact_total,
+            observation_count=len(observations),
+        )
+        zf.writestr("manifest.json", manifest.model_dump_json(indent=2))
+
+    logger.info(
+        "[transfer] Exported %d document(s), %d fact(s), %d observation(s) from bank %s",
+        len(documents),
+        fact_total,
+        len(observations),
+        bank_id,
+    )
+    return archive.getvalue()
+
+
+def _row_json_default(obj: Any) -> Any:
+    """JSON serializer for the value types asyncpg returns from bank rows."""
+    if isinstance(obj, datetime):
+        return obj.isoformat()
+    if isinstance(obj, date):
+        return obj.isoformat()
+    if isinstance(obj, UUID):
+        return str(obj)
+    if isinstance(obj, Decimal):
+        # str preserves precision; import casts back to numeric.
+        return str(obj)
+    if isinstance(obj, (bytes, bytearray, memoryview)):
+        return base64.b64encode(bytes(obj)).decode("ascii")
+    raise TypeError(f"Object of type {type(obj).__name__} is not JSON serializable")
+
+
+async def _dump_bank_rows(conn: Any, table: str, bank_id: str) -> list[dict]:
+    """Dump all rows of a bank-scoped table as JSON-ready dicts (derived columns stripped).
+
+    Embedding/search-vector columns are omitted so the target instance
+    regenerates them with its own model/backend on import.
+    """
+    rows = await conn.fetch(f"SELECT * FROM {fq_table(table)} WHERE bank_id = $1", bank_id)
+    return [{k: v for k, v in dict(row).items() if k not in _DERIVED_COLUMNS} for row in rows]
+
+
+async def _dump_history_rows(conn: Any, table: str, bank_id: str) -> list[dict]:
+    """Dump a bank-scoped child-history table for carrying across instances.
+
+    Drops the surrogate ``id`` so the target reassigns it from its own IDENTITY
+    sequence (carrying explicit ids would leave the sequence un-advanced and
+    collide with later writes). Ordered oldest-first so the reassigned ids keep
+    the same chronological tie-break order the read path relies on.
+    """
+    rows = await conn.fetch(
+        f"SELECT * FROM {fq_table(table)} WHERE bank_id = $1 ORDER BY changed_at, id",
+        bank_id,
+    )
+    return [{k: v for k, v in dict(row).items() if k not in _DERIVED_COLUMNS and k != "id"} for row in rows]
+
+
+async def export_bank(conn: Any, bank_id: str, *, include_history: bool = False) -> bytes:
+    """Export an entire bank into a portable ZIP archive (no embeddings).
+
+    Produces a superset of the documents archive: the logical
+    document/fact/observation export (replayed and re-embedded on import) plus
+    the bank's config, mental models, directives and webhooks as JSON rows. With
+    ``include_history`` the operational tails (audit_log, llm_requests) are also
+    carried. Intended for migrating a bank to a new instance configured with a
+    different embedding model / vector / text-search backend — every vector is
+    regenerated on the target, so nothing here is encoder-specific.
+
+    ``conn`` is a live connection scoped to the bank's schema (the admin CLI sets
+    ``_current_schema`` and passes its raw connection; the engine acquires one
+    after tenant auth).
+    """
+    loaded = await _load_documents(conn, bank_id, None)
+    documents = loaded.documents
+    # Whole-bank export always carries observations (they're bank-level state).
+    observations = await _load_observations(conn, bank_id, loaded.unit_index)
+
+    bank_rows = {table: await _dump_bank_rows(conn, table, bank_id) for table in _BANK_ROW_TABLES}
+    for table in _CARRIED_HISTORY_TABLES:
+        bank_rows[table] = await _dump_history_rows(conn, table, bank_id)
+    history_rows: dict[str, list[dict]] = {}
+    if include_history:
+        history_rows = {table: await _dump_bank_rows(conn, table, bank_id) for table in _HISTORY_TABLES}
+
+    archive = io.BytesIO()
+    fact_total = 0
+    with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
+        for index, document in enumerate(documents):
+            fact_total += len(document.facts)
+            zf.writestr(f"documents/{index:06d}.json", document.model_dump_json(indent=2, exclude_none=False))
+
+        if observations:
+            payload = "[\n" + ",\n".join(o.model_dump_json(indent=2) for o in observations) + "\n]\n"
+            zf.writestr("observations.json", payload)
+
+        for table, rows in bank_rows.items():
+            zf.writestr(f"{table}.json", json.dumps(rows, indent=2, default=_row_json_default))
+        for table, rows in history_rows.items():
+            zf.writestr(f"history/{table}.json", json.dumps(rows, indent=2, default=_row_json_default))
+
+        manifest = TransferManifest(
+            schema_version=SCHEMA_VERSION,
+            source_bank_id=bank_id,
+            exported_at=datetime.now(UTC),
+            document_count=len(documents),
+            fact_count=fact_total,
+            observation_count=len(observations),
+            archive_type="bank",
+            mental_model_count=len(bank_rows.get("mental_models", [])),
+            directive_count=len(bank_rows.get("directives", [])),
+            webhook_count=len(bank_rows.get("webhooks", [])),
+            includes_history=include_history,
+        )
+        zf.writestr("manifest.json", manifest.model_dump_json(indent=2))
+
+    logger.info(
+        "[transfer] Exported bank %s: %d document(s), %d fact(s), %d observation(s), "
+        "%d mental model(s), %d directive(s), %d webhook(s)%s",
+        bank_id,
+        len(documents),
+        fact_total,
+        len(observations),
+        len(bank_rows.get("mental_models", [])),
+        len(bank_rows.get("directives", [])),
+        len(bank_rows.get("webhooks", [])),
+        " (with history)" if include_history else "",
+    )
+    return archive.getvalue()
+
+
+async def _load_documents(
+    conn: Any,
+    bank_id: str,
+    document_ids: list[str] | None,
+) -> _LoadedExport:
+    """Load and assemble TransferDocument payloads for the requested documents."""
+    doc_filter = "AND id = ANY($2)" if document_ids else ""
+    params: list[Any] = [bank_id]
+    if document_ids:
+        params.append(document_ids)
+    doc_rows = await conn.fetch(
+        f"""
+        SELECT id, original_text, retain_params, tags, created_at
+        FROM {fq_table("documents")}
+        WHERE bank_id = $1 {doc_filter}
+        ORDER BY created_at, id
+        """,
+        *params,
+    )
+    if not doc_rows:
+        return _LoadedExport()
+
+    selected_ids = [row["id"] for row in doc_rows]
+
+    chunks_by_doc = await _load_chunks(conn, bank_id, selected_ids)
+    loaded = await _load_facts(conn, bank_id, selected_ids)
+    await _attach_entities(conn, loaded)
+    await _attach_causal_relations(conn, loaded)
+
+    documents: list[TransferDocument] = []
+    for row in doc_rows:
+        doc_id = row["id"]
+        documents.append(
+            TransferDocument(
+                id=doc_id,
+                original_text=row["original_text"],
+                retain_params=_as_jsonb(row["retain_params"]),
+                tags=list(row["tags"] or []),
+                created_at=row["created_at"],
+                chunks=chunks_by_doc.get(doc_id, []),
+                facts=loaded.facts_by_doc.get(doc_id, []),
+            )
+        )
+    return _LoadedExport(documents=documents, unit_index=loaded.unit_index)
+
+
+async def _load_observations(
+    conn: Any,
+    bank_id: str,
+    unit_index: dict[Any, _UnitLocation],
+) -> list[TransferObservation]:
+    """Load observations whose source facts are all present in the exported set.
+
+    Each source unit id is rewritten to its (document_id, fact_index) reference
+    via ``unit_index``. Only called for a whole-bank export, so every live source
+    fact is present; an observation is skipped only if a source no longer exists
+    (stale reference) — that keeps every exported observation resolvable on import.
+    """
+    rows = await conn.fetch(
+        f"""
+        SELECT id, text, tags, event_date, occurred_start, occurred_end,
+               mentioned_at, observation_scopes, proof_count, source_memory_ids
+        FROM {fq_table("memory_units")}
+        WHERE bank_id = $1 AND fact_type = 'observation'
+        ORDER BY created_at, id
+        """,
+        bank_id,
+    )
+
+    observations: list[TransferObservation] = []
+    skipped = 0
+    for row in rows:
+        source_ids = list(row["source_memory_ids"] or [])
+        locations = [unit_index.get(sid) for sid in source_ids]
+        if not source_ids or any(loc is None for loc in locations):
+            # An observation with sources outside the exported documents would be
+            # incoherent on import — skip it rather than emit dangling refs.
+            skipped += 1
+            continue
+        observations.append(
+            TransferObservation(
+                text=row["text"],
+                tags=list(row["tags"] or []),
+                event_date=row["event_date"],
+                occurred_start=row["occurred_start"],
+                occurred_end=row["occurred_end"],
+                mentioned_at=row["mentioned_at"],
+                observation_scopes=_as_jsonb(row["observation_scopes"]),
+                proof_count=row["proof_count"] or len(source_ids),
+                sources=[
+                    TransferObservationSource(document_id=loc.document_id, fact_index=loc.ordinal)
+                    for loc in locations
+                    if loc is not None
+                ],
+            )
+        )
+    if skipped:
+        logger.info("[transfer] Skipped %d observation(s) with sources outside the exported documents", skipped)
+    return observations
+
+
+async def _load_chunks(conn: Any, bank_id: str, doc_ids: list[str]) -> dict[str, list[TransferChunk]]:
+    rows = await conn.fetch(
+        f"""
+        SELECT document_id, chunk_index, chunk_text
+        FROM {fq_table("chunks")}
+        WHERE bank_id = $1 AND document_id = ANY($2)
+        ORDER BY document_id, chunk_index
+        """,
+        bank_id,
+        doc_ids,
+    )
+    chunks_by_doc: dict[str, list[TransferChunk]] = {}
+    for row in rows:
+        chunks_by_doc.setdefault(row["document_id"], []).append(
+            TransferChunk(chunk_index=row["chunk_index"], chunk_text=row["chunk_text"])
+        )
+    return chunks_by_doc
+
+
+async def _load_facts(conn: Any, bank_id: str, doc_ids: list[str]) -> _LoadedFacts:
+    """Load non-observation facts grouped by document, with a unit-id location index.
+
+    The ordering is fixed (created_at, id) so that
+    ``causal_relations.target_fact_index`` ordinals stay consistent.
+    """
+    rows = await conn.fetch(
+        f"""
+        SELECT id, document_id, text, fact_type, context, event_date,
+               occurred_start, occurred_end, mentioned_at, metadata,
+               chunk_id, tags, observation_scopes
+        FROM {fq_table("memory_units")}
+        WHERE bank_id = $1
+          AND document_id = ANY($2)
+          AND fact_type = ANY($3)
+        ORDER BY document_id, created_at, id
+        """,
+        bank_id,
+        doc_ids,
+        list(_EXPORTED_FACT_TYPES),
+    )
+
+    loaded = _LoadedFacts()
+    for row in rows:
+        doc_id = row["document_id"]
+        bucket = loaded.facts_by_doc.setdefault(doc_id, [])
+        ordinal = len(bucket)
+        fact = TransferFact(
+            text=row["text"],
+            fact_type=row["fact_type"],
+            context=row["context"],
+            event_date=row["event_date"],
+            occurred_start=row["occurred_start"],
+            occurred_end=row["occurred_end"],
+            mentioned_at=row["mentioned_at"],
+            metadata=_as_jsonb(row["metadata"]) or {},
+            tags=list(row["tags"] or []),
+            observation_scopes=_as_jsonb(row["observation_scopes"]),
+            chunk_index=_chunk_index_from_chunk_id(row["chunk_id"]),
+        )
+        bucket.append(fact)
+        loaded.unit_index[row["id"]] = _UnitLocation(document_id=doc_id, ordinal=ordinal)
+    return loaded
+
+
+async def _attach_entities(conn: Any, loaded: _LoadedFacts) -> None:
+    """Populate each fact's ``entities`` list with its entities' canonical names."""
+    if not loaded.unit_index:
+        return
+    rows = await conn.fetch(
+        f"""
+        SELECT ue.unit_id, e.canonical_name
+        FROM {fq_table("unit_entities")} ue
+        JOIN {fq_table("entities")} e ON e.id = ue.entity_id
+        WHERE ue.unit_id = ANY($1)
+        ORDER BY e.canonical_name
+        """,
+        list(loaded.unit_index.keys()),
+    )
+    for row in rows:
+        location = loaded.unit_index.get(row["unit_id"])
+        if location is None:
+            continue
+        loaded.facts_by_doc[location.document_id][location.ordinal].entities.append(row["canonical_name"])
+
+
+async def _attach_causal_relations(conn: Any, loaded: _LoadedFacts) -> None:
+    """Reconstruct causal edges as fact ordinals within each document.
+
+    A memory_link (from_unit -> to_unit, link_type) means ``from_unit`` carries
+    the relation pointing at ``to_unit``, so the edge is attached to the source
+    fact with the target's ordinal. Edges spanning two documents are skipped
+    (causal links are created within a single retain batch in practice).
+    """
+    if not loaded.unit_index:
+        return
+    rows = await conn.fetch(
+        f"""
+        SELECT from_unit_id, to_unit_id, link_type
+        FROM {fq_table("memory_links")}
+        WHERE link_type = ANY($1)
+          AND from_unit_id = ANY($2)
+          AND to_unit_id = ANY($2)
+        """,
+        list(_CAUSAL_LINK_TYPES),
+        list(loaded.unit_index.keys()),
+    )
+    for row in rows:
+        source = loaded.unit_index.get(row["from_unit_id"])
+        target = loaded.unit_index.get(row["to_unit_id"])
+        if source is None or target is None:
+            continue
+        if source.document_id != target.document_id:
+            continue
+        loaded.facts_by_doc[source.document_id][source.ordinal].causal_relations.append(
+            TransferCausalRelation(
+                relation_type=row["link_type"],
+                target_fact_index=target.ordinal,
+            )
+        )
@@ -0,0 +1,716 @@
+"""Import documents from a transfer archive by replaying the deterministic retain pipeline.
+
+For each document the importer rebuilds the extracted facts, re-embeds them with
+the *target* bank's embedding model, then runs entity resolution (Phase 1) and
+the fact/link insert (Phase 2) — exactly the steps retain runs after LLM
+extraction. No LLM is called. Temporal/semantic/causal links and entity merges
+are therefore computed relative to the target bank's existing memories.
+"""
+
+from __future__ import annotations
+
+import io
+import json
+import logging
+import uuid
+import zipfile
+from dataclasses import dataclass, field
+from datetime import UTC, date, datetime
+from typing import Any, Literal
+
+from ..db_utils import acquire_with_retry
+from ..retain import bank_utils, chunk_storage, embedding_processing, fact_storage, orchestrator
+from ..retain.types import (
+    CausalRelation,
+    ChunkMetadata,
+    ExtractedFact,
+    ProcessedFact,
+    RetainContent,
+)
+from ..schema import fq_table
+from .schema import (
+    SCHEMA_VERSION,
+    TransferDocument,
+    TransferFact,
+    TransferManifest,
+    TransferObservation,
+)
+
+logger = logging.getLogger(__name__)
+
+OnConflict = Literal["skip", "replace", "new-id"]
+_VALID_CONFLICT_MODES: tuple[OnConflict, ...] = ("skip", "replace", "new-id")
+
+
+@dataclass
+class ImportedDocument:
+    """A single document successfully imported, with the units it produced.
+
+    Carried back so the engine can fire the post-retain extension hook
+    (usage tracking / metrics / notifications) once per imported document,
+    mirroring how retain reports each completed document.
+    """
+
+    document_id: str
+    unit_ids: list[str]
+    content: str
+    tags: list[str]
+
+
+@dataclass
+class ImportResult:
+    """Outcome of importing a transfer archive into a bank."""
+
+    documents_imported: int = 0
+    documents_skipped: int = 0
+    facts_imported: int = 0
+    observations_imported: int = 0
+    # Observations dropped because some source fact was not imported in this run.
+    observations_skipped: int = 0
+    skipped_document_ids: list[str] = field(default_factory=list)
+    # Original id -> freshly generated id, for documents imported under "new-id".
+    remapped_document_ids: dict[str, str] = field(default_factory=dict)
+    # Per-document outcomes, for the engine's post-retain hook. Not serialized
+    # into operation result_metadata (the worker handler writes counts only).
+    imported_documents: list[ImportedDocument] = field(default_factory=list)
+
+
+@dataclass
+class _ObservationOutcome:
+    """Counts from the observation import pass."""
+
+    imported: int = 0
+    skipped: int = 0
+
+
+@dataclass
+class ParsedArchive:
+    """A transfer archive after parsing/validation."""
+
+    manifest: TransferManifest
+    documents: list[TransferDocument]
+    observations: list[TransferObservation] = field(default_factory=list)
+
+
+def parse_archive(archive_bytes: bytes) -> ParsedArchive:
+    """Parse and validate a transfer ZIP archive produced by ``export_documents``."""
+    with zipfile.ZipFile(io.BytesIO(archive_bytes), "r") as zf:
+        names = set(zf.namelist())
+        if "manifest.json" not in names:
+            raise ValueError("Invalid transfer archive: manifest.json is missing")
+        manifest = TransferManifest.model_validate_json(zf.read("manifest.json"))
+        if manifest.schema_version != SCHEMA_VERSION:
+            raise ValueError(
+                f"Unsupported transfer archive schema version {manifest.schema_version} "
+                f"(this build supports {SCHEMA_VERSION})"
+            )
+        doc_names = sorted(n for n in names if n.startswith("documents/") and n.endswith(".json"))
+        documents = [TransferDocument.model_validate_json(zf.read(name)) for name in doc_names]
+        observations: list[TransferObservation] = []
+        if "observations.json" in names:
+            observations = [TransferObservation.model_validate(o) for o in json.loads(zf.read("observations.json"))]
+    return ParsedArchive(manifest=manifest, documents=documents, observations=observations)
+
+
+async def import_documents(
+    *,
+    backend: Any,
+    embeddings_model: Any,
+    entity_resolver: Any,
+    config: Any,
+    format_date_fn: Any,
+    bank_id: str,
+    archive_bytes: bytes,
+    on_conflict: OnConflict = "skip",
+    ops: Any = None,
+    outbox_callback_factory: Any = None,
+) -> ImportResult:
+    """Import every document in ``archive_bytes`` into ``bank_id``.
+
+    Args:
+        backend: Database backend (provides ``acquire()`` and ``ops``).
+        embeddings_model: Target bank's embedding model (used to re-embed facts).
+        entity_resolver: Shared entity resolver for the target bank.
+        config: Resolved bank config for the target bank.
+        format_date_fn: Date formatter used when augmenting fact text for embedding
+            (must match retain so embeddings are consistent).
+        bank_id: Target bank.
+        archive_bytes: A ZIP archive produced by ``export_documents``.
+        on_conflict: How to handle a document id that already exists in the target
+            bank — ``skip`` (default), ``replace`` (delete old data and re-import),
+            or ``new-id`` (import under a freshly generated id).
+        ops: Backend ``DataAccessOps``. Defaults to ``backend.ops``.
+
+    Returns:
+        An :class:`ImportResult` with per-document counts.
+    """
+    if on_conflict not in _VALID_CONFLICT_MODES:
+        raise ValueError(f"Invalid on_conflict '{on_conflict}'; expected one of {_VALID_CONFLICT_MODES}")
+    if ops is None:
+        ops = backend.ops
+
+    parsed = parse_archive(archive_bytes)
+    result = ImportResult()
+
+    # (original document_id, fact ordinal) -> freshly inserted unit id. Used to
+    # resolve observation source references after all facts exist.
+    ref_map: dict[tuple[str, int], str] = {}
+
+    for document in parsed.documents:
+        target_id = await _resolve_target_id(backend, bank_id, document.id, on_conflict)
+        if target_id is None:
+            result.documents_skipped += 1
+            result.skipped_document_ids.append(document.id)
+            continue
+        if target_id != document.id:
+            result.remapped_document_ids[document.id] = target_id
+
+        unit_ids = await _import_one_document(
+            backend=backend,
+            embeddings_model=embeddings_model,
+            entity_resolver=entity_resolver,
+            config=config,
+            format_date_fn=format_date_fn,
+            bank_id=bank_id,
+            document=document,
+            target_id=target_id,
+            ops=ops,
+            outbox_callback_factory=outbox_callback_factory,
+        )
+        result.documents_imported += 1
+        result.facts_imported += len(unit_ids)
+        result.imported_documents.append(
+            ImportedDocument(
+                document_id=target_id,
+                unit_ids=unit_ids,
+                content=document.original_text or "",
+                tags=list(document.tags),
+            )
+        )
+        for ordinal, unit_id in enumerate(unit_ids):
+            ref_map[(document.id, ordinal)] = unit_id
+
+    if parsed.observations:
+        outcome = await _import_observations(
+            backend=backend,
+            embeddings_model=embeddings_model,
+            bank_id=bank_id,
+            observations=parsed.observations,
+            ref_map=ref_map,
+            ops=ops,
+        )
+        result.observations_imported = outcome.imported
+        result.observations_skipped = outcome.skipped
+
+    logger.info(
+        "[transfer] Imported %d document(s), %d fact(s), %d observation(s) into bank %s "
+        "(%d docs skipped, %d observations skipped)",
+        result.documents_imported,
+        result.facts_imported,
+        result.observations_imported,
+        bank_id,
+        result.documents_skipped,
+        result.observations_skipped,
+    )
+    return result
+
+
+# Bank-level config/state tables restored verbatim from a whole-bank archive.
+# Order matters for foreign keys: banks (parent) is restored before any child.
+_BANK_CHILD_TABLES = ("mental_models", "directives", "webhooks")
+# Child-history carried verbatim; restored after its parent (mental_models) so the
+# foreign key resolves. Surrogate ids were dropped on export (the target reassigns
+# them), so these restore via fresh IDENTITY values.
+_CARRIED_HISTORY_TABLES = ("mental_model_history",)
+_HISTORY_TABLES = ("audit_log", "llm_requests")
+
+
+@dataclass
+class BankImportResult:
+    """Outcome of importing a whole-bank archive."""
+
+    bank_id: str
+    documents_imported: int = 0
+    facts_imported: int = 0
+    observations_imported: int = 0
+    mental_models_imported: int = 0
+    mental_model_history_imported: int = 0
+    directives_imported: int = 0
+    webhooks_imported: int = 0
+    history_rows_imported: int = 0
+
+
+@dataclass
+class ParsedBankArchive:
+    """The bank-level sections of a whole-bank archive (documents read separately)."""
+
+    manifest: TransferManifest
+    # table name -> list of verbatim row dicts (banks, mental_models, directives, webhooks)
+    bank_rows: dict[str, list[dict]] = field(default_factory=dict)
+    # table name -> rows (audit_log, llm_requests), present only with --include-history
+    history_rows: dict[str, list[dict]] = field(default_factory=dict)
+
+
+def parse_bank_archive(archive_bytes: bytes) -> ParsedBankArchive:
+    """Parse the bank-level sections of a whole-bank archive (``archive_type='bank'``)."""
+    with zipfile.ZipFile(io.BytesIO(archive_bytes), "r") as zf:
+        names = set(zf.namelist())
+        if "manifest.json" not in names:
+            raise ValueError("Invalid transfer archive: manifest.json is missing")
+        manifest = TransferManifest.model_validate_json(zf.read("manifest.json"))
+        if manifest.archive_type != "bank":
+            raise ValueError(
+                f"Not a whole-bank archive (archive_type={manifest.archive_type!r}); use import_documents instead"
+            )
+        bank_rows: dict[str, list[dict]] = {}
+        for table in ("banks", *_BANK_CHILD_TABLES, *_CARRIED_HISTORY_TABLES):
+            fname = f"{table}.json"
+            bank_rows[table] = json.loads(zf.read(fname)) if fname in names else []
+        history_rows: dict[str, list[dict]] = {}
+        for table in _HISTORY_TABLES:
+            fname = f"history/{table}.json"
+            if fname in names:
+                history_rows[table] = json.loads(zf.read(fname))
+    return ParsedBankArchive(manifest=manifest, bank_rows=bank_rows, history_rows=history_rows)
+
+
+async def _restore_rows(conn: Any, table: str, rows: list[dict]) -> int:
+    """Insert verbatim rows into a bank-scoped table, coercing JSON-encoded values
+    back to the column's type (timestamps, uuids, jsonb). ``ON CONFLICT DO NOTHING``
+    keeps an import idempotent and safe to re-run against a partially-filled target."""
+    if not rows:
+        return 0
+    from ..memory_engine import get_current_schema
+
+    schema = get_current_schema()
+    col_types = {
+        r["column_name"]: r["data_type"]
+        for r in await conn.fetch(
+            "SELECT column_name, data_type FROM information_schema.columns WHERE table_schema = $1 AND table_name = $2",
+            schema,
+            table,
+        )
+    }
+    inserted = 0
+    for row in rows:
+        cols = [c for c in row if c in col_types]
+        placeholders: list[str] = []
+        values: list[Any] = []
+        for position, col in enumerate(cols, start=1):
+            data_type = col_types[col]
+            value = row[col]
+            if data_type in ("jsonb", "json"):
+                # asyncpg has no JSON codec on these raw connections; pass JSON
+                # text and cast. Values may already be str (no codec on export) or
+                # a Python object (codec on export) — normalize to text either way.
+                values.append(value if isinstance(value, str) or value is None else json.dumps(value))
+                placeholders.append(f"${position}::jsonb")
+                continue
+            if value is not None and isinstance(value, str):
+                if data_type in ("timestamp with time zone", "timestamp without time zone"):
+                    value = datetime.fromisoformat(value)
+                elif data_type == "date":
+                    value = date.fromisoformat(value)
+                elif data_type == "uuid":
+                    value = uuid.UUID(value)
+            placeholders.append(f"${position}")
+            values.append(value)
+        col_list = ", ".join(f'"{c}"' for c in cols)
+        await conn.execute(
+            f"INSERT INTO {fq_table(table)} ({col_list}) VALUES ({', '.join(placeholders)}) ON CONFLICT DO NOTHING",
+            *values,
+        )
+        inserted += 1
+    return inserted
+
+
+async def import_bank(
+    *,
+    backend: Any,
+    embeddings_model: Any,
+    entity_resolver: Any,
+    config: Any,
+    format_date_fn: Any,
+    archive_bytes: bytes,
+    target_bank_id: str | None = None,
+    include_history: bool = False,
+    ops: Any = None,
+) -> BankImportResult:
+    """Restore a whole bank from a ``export_bank`` archive into the target instance.
+
+    Re-embeds facts with the *target* instance's embedding model and rebuilds links,
+    entities and search/vector indexes — the path for migrating a bank to an instance
+    configured with a different embedding model / vector / text-search backend.
+
+    The **target bank must not already exist**: import restores a complete bank
+    (config + facts + mental models + …) and is not a merge. If a bank with the
+    target id is present, this raises — delete it first or pass ``target_bank_id``
+    for a fresh id. A migration restores *exact* state, so unlike the document
+    import it fires no retain webhooks and triggers no consolidation/graph
+    maintenance: observations and mental models are restored as exported.
+    """
+    if ops is None:
+        ops = backend.ops
+    parsed = parse_bank_archive(archive_bytes)
+    source_bank_id = parsed.manifest.source_bank_id
+    bank_id = target_bank_id or source_bank_id
+
+    # Remapping to a different id: rewrite the carried bank_id on every row so FKs
+    # and PKs line up with the (also-remapped) documents/facts.
+    if bank_id != source_bank_id:
+        for rows in (*parsed.bank_rows.values(), *parsed.history_rows.values()):
+            for row in rows:
+                if "bank_id" in row:
+                    row["bank_id"] = bank_id
+
+    async with acquire_with_retry(backend) as conn:
+        # Refuse to import into an existing bank — this restores a whole bank, it
+        # does not merge. Merging would silently mix the archive's config/mental
+        # models/webhooks with whatever is already there (and global-unique ids
+        # like webhooks/directives would collide).
+        if await conn.fetchval(f"SELECT 1 FROM {fq_table('banks')} WHERE bank_id = $1", bank_id):
+            raise ValueError(
+                f"Target bank '{bank_id}' already exists; import-bank restores into a fresh bank "
+                f"(it is not a merge). Delete the bank first, or pass a different target bank id."
+            )
+        # Bank row first — children (documents, mental_models, …) FK to it.
+        await _restore_rows(conn, "banks", parsed.bank_rows.get("banks", []))
+    # Ensure the bank's per-bank vector indexes exist (no-op for global-index
+    # extensions); idempotent and keeps the restored banks row (ON CONFLICT DO NOTHING).
+    await bank_utils.get_or_create_bank_profile(backend, bank_id)
+
+    doc_result = await import_documents(
+        backend=backend,
+        embeddings_model=embeddings_model,
+        entity_resolver=entity_resolver,
+        config=config,
+        format_date_fn=format_date_fn,
+        bank_id=bank_id,
+        archive_bytes=archive_bytes,
+        ops=ops,
+        outbox_callback_factory=None,
+    )
+
+    result = BankImportResult(
+        bank_id=bank_id,
+        documents_imported=doc_result.documents_imported,
+        facts_imported=doc_result.facts_imported,
+        observations_imported=doc_result.observations_imported,
+    )
+    async with acquire_with_retry(backend) as conn:
+        result.mental_models_imported = await _restore_rows(
+            conn, "mental_models", parsed.bank_rows.get("mental_models", [])
+        )
+        # Restored after mental_models so the (mental_model_id, bank_id) FK resolves.
+        result.mental_model_history_imported = await _restore_rows(
+            conn, "mental_model_history", parsed.bank_rows.get("mental_model_history", [])
+        )
+        result.directives_imported = await _restore_rows(conn, "directives", parsed.bank_rows.get("directives", []))
+        result.webhooks_imported = await _restore_rows(conn, "webhooks", parsed.bank_rows.get("webhooks", []))
+        if include_history:
+            for table in _HISTORY_TABLES:
+                result.history_rows_imported += await _restore_rows(conn, table, parsed.history_rows.get(table, []))
+
+    logger.info(
+        "[transfer] Imported bank %s: %d doc(s), %d fact(s), %d observation(s), "
+        "%d mental model(s), %d mm-history row(s), %d directive(s), %d webhook(s), %d history row(s)",
+        bank_id,
+        result.documents_imported,
+        result.facts_imported,
+        result.observations_imported,
+        result.mental_models_imported,
+        result.mental_model_history_imported,
+        result.directives_imported,
+        result.webhooks_imported,
+        result.history_rows_imported,
+    )
+    return result
+
+
+async def _resolve_target_id(backend: Any, bank_id: str, document_id: str, on_conflict: OnConflict) -> str | None:
+    """Decide the document id to write under, or ``None`` to skip.
+
+    Returns the original id when there is no conflict, a fresh id under
+    ``new-id``, the original id under ``replace`` (the insert path cascades the
+    old data away), or ``None`` under ``skip`` when the document already exists.
+    """
+    async with acquire_with_retry(backend) as conn:
+        exists = await conn.fetchval(
+            f"SELECT 1 FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
+            document_id,
+            bank_id,
+        )
+    if not exists:
+        return document_id
+    if on_conflict == "skip":
+        return None
+    if on_conflict == "new-id":
+        return str(uuid.uuid4())
+    return document_id  # replace
+
+
+async def _import_one_document(
+    *,
+    backend: Any,
+    embeddings_model: Any,
+    entity_resolver: Any,
+    config: Any,
+    format_date_fn: Any,
+    bank_id: str,
+    document: TransferDocument,
+    target_id: str,
+    ops: Any,
+    outbox_callback_factory: Any = None,
+) -> list[str]:
+    """Re-embed and insert a single document; returns the new unit ids in fact order."""
+    log_buffer: list[str] = []
+
+    # Fire the same retain.completed webhook retain emits, transactionally inside
+    # this document's insert. Factory returns None when no webhook manager exists.
+    outbox_callback = (
+        outbox_callback_factory([{"document_id": target_id, "tags": list(document.tags)}])
+        if outbox_callback_factory
+        else None
+    )
+
+    extracted_facts = [_to_extracted_fact(fact) for fact in document.facts]
+
+    processed_facts: list[ProcessedFact] = []
+    if extracted_facts:
+        augmented = embedding_processing.augment_texts_with_dates(extracted_facts, format_date_fn)
+        embeddings = await embedding_processing.generate_embeddings_batch(embeddings_model, augmented)
+        processed_facts = [ProcessedFact.from_extracted_fact(ef, emb) for ef, emb in zip(extracted_facts, embeddings)]
+
+    contents = [RetainContent(content=document.original_text or "")]
+    chunk_meta = [
+        ChunkMetadata(chunk_text=chunk.chunk_text, fact_count=0, content_index=0, chunk_index=chunk.chunk_index)
+        for chunk in document.chunks
+    ]
+
+    # Phase 1 (entity resolution + semantic ANN) on its own connection, outside
+    # the write transaction — mirrors the retain pipeline.
+    entity_resolver.discard_pending_stats()
+    phase1 = await orchestrator._pre_resolve_phase1(
+        backend,
+        entity_resolver,
+        bank_id,
+        contents,
+        processed_facts,
+        config,
+        log_buffer,
+        skip_semantic_ann=False,
+    )
+
+    async with acquire_with_retry(backend) as conn:
+        async with conn.transaction():
+            # is_first_batch=True: cascade-delete any existing data for this id
+            # (the "replace" path) and (re)insert the document row.
+            await fact_storage.handle_document_tracking(
+                conn,
+                bank_id,
+                target_id,
+                document.original_text or "",
+                True,
+                document.retain_params,
+                document.tags,
+                ops=ops,
+            )
+
+            chunk_id_map: dict[int, str] = {}
+            if chunk_meta:
+                chunk_id_map = await chunk_storage.store_chunks_batch(conn, bank_id, target_id, chunk_meta, ops=ops)
+
+            for extracted, processed in zip(extracted_facts, processed_facts):
+                processed.document_id = target_id
+                if chunk_id_map and extracted.chunk_index is not None:
+                    chunk_id = chunk_id_map.get(extracted.chunk_index)
+                    if chunk_id:
+                        processed.chunk_id = chunk_id
+
+            result_unit_ids = await orchestrator._insert_facts_and_links(
+                conn,
+                entity_resolver,
+                bank_id,
+                contents,
+                extracted_facts,
+                processed_facts,
+                config,
+                log_buffer,
+                resolved_entity_ids=phase1.entities.resolved_entity_ids,
+                entity_to_unit=phase1.entities.entity_to_unit,
+                unit_to_entity_ids=phase1.entities.unit_to_entity_ids,
+                semantic_ann_links=phase1.semantic_ann_links,
+                skip_semantic_links=False,
+                outbox_callback=outbox_callback,
+                ops=ops,
+            )
+
+        try:
+            await entity_resolver.flush_pending_stats()
+        except Exception:
+            logger.warning("[transfer] Entity stats flush failed for document %s", target_id, exc_info=True)
+
+    logger.debug("[transfer] Imported document %s:\n%s", target_id, "\n".join(log_buffer))
+    # Single content item -> result_unit_ids[0] holds the new unit ids in fact order.
+    return list(result_unit_ids[0]) if result_unit_ids else []
+
+
+async def _import_observations(
+    *,
+    backend: Any,
+    embeddings_model: Any,
+    bank_id: str,
+    observations: list[TransferObservation],
+    ref_map: dict[tuple[str, int], str],
+    ops: Any,
+) -> _ObservationOutcome:
+    """Insert observations whose source facts were all imported in this run.
+
+    Observations carry no embedding, links, or entity rows — only the unit row
+    plus ``source_memory_ids`` (remapped to the freshly inserted source units)
+    and ``proof_count``. Their source facts are marked ``consolidated_at`` so the
+    target bank's consolidator won't re-process them. Mirrors what consolidation
+    writes, but driven from the archive instead of the LLM.
+
+    Inserted as-is: imported observations are NOT merged or deduplicated against
+    observations that already exist in the target bank (unlike consolidation,
+    which merges related observations). Importing into a bank that already has
+    observations — or importing the same archive twice — can therefore produce
+    overlapping observations over the same facts.
+    """
+    outcome = _ObservationOutcome()
+
+    # Resolve each observation's sources to new unit ids; drop any whose sources
+    # weren't all imported (e.g. a subset/skip import).
+    resolved: list[tuple[TransferObservation, list[str]]] = []
+    for obs in observations:
+        source_ids = [ref_map.get((s.document_id, s.fact_index)) for s in obs.sources]
+        if not source_ids or any(sid is None for sid in source_ids):
+            outcome.skipped += 1
+            continue
+        resolved.append((obs, [sid for sid in source_ids if sid is not None]))
+
+    if not resolved:
+        return outcome
+
+    # Observations embed the raw text (matching consolidation), not the
+    # date-augmented text used for facts.
+    embeddings = await embedding_processing.generate_embeddings_batch(
+        embeddings_model, [obs.text for obs, _ in resolved]
+    )
+    processed = [
+        ProcessedFact(
+            fact_text=obs.text,
+            fact_type="observation",
+            embedding=embedding,
+            occurred_start=obs.occurred_start,
+            occurred_end=obs.occurred_end,
+            mentioned_at=_observation_mentioned_at(obs),
+            context="",
+            metadata={},
+            tags=list(obs.tags),
+            observation_scopes=obs.observation_scopes,
+            document_id=None,
+            chunk_id=None,
+        )
+        for (obs, _sources), embedding in zip(resolved, embeddings)
+    ]
+
+    async with acquire_with_retry(backend) as conn:
+        async with conn.transaction():
+            obs_unit_ids = await fact_storage.insert_facts_batch(conn, bank_id, processed, ops=ops)
+
+            all_source_ids: set[uuid.UUID] = set()
+            for (obs, sources), obs_unit_id in zip(resolved, obs_unit_ids):
+                source_uuids = [uuid.UUID(s) for s in sources]
+                all_source_ids.update(source_uuids)
+                await _link_observation_sources(
+                    conn, ops, bank_id, uuid.UUID(obs_unit_id), source_uuids, obs.proof_count
+                )
+
+            # Mark source facts consolidated so the target consolidator skips them.
+            if all_source_ids:
+                await conn.execute(
+                    f"UPDATE {fq_table('memory_units')} SET consolidated_at = now() "
+                    f"WHERE bank_id = $1 AND id = ANY($2)",
+                    bank_id,
+                    list(all_source_ids),
+                )
+
+    outcome.imported = len(resolved)
+    return outcome
+
+
+async def _link_observation_sources(
+    conn: Any,
+    ops: Any,
+    bank_id: str,
+    observation_id: uuid.UUID,
+    source_ids: list[uuid.UUID],
+    proof_count: int,
+) -> None:
+    """Attach source ids + proof_count to a freshly inserted observation row.
+
+    PG stores the sources in the ``source_memory_ids`` array column; Oracle uses
+    the ``observation_sources`` junction table (same split as consolidation).
+    """
+    if ops.uses_observation_sources_table:
+        await conn.executemany(
+            f"INSERT INTO {fq_table('observation_sources')} (observation_id, source_id) "
+            f"VALUES ($1, $2) ON CONFLICT (observation_id, source_id) DO NOTHING",
+            [(observation_id, sid) for sid in dict.fromkeys(source_ids)],
+        )
+        await conn.execute(
+            f"UPDATE {fq_table('memory_units')} SET proof_count = $1 WHERE id = $2 AND bank_id = $3",
+            proof_count,
+            observation_id,
+            bank_id,
+        )
+    else:
+        await conn.execute(
+            f"UPDATE {fq_table('memory_units')} SET source_memory_ids = $1, proof_count = $2 "
+            f"WHERE id = $3 AND bank_id = $4",
+            source_ids,
+            proof_count,
+            observation_id,
+            bank_id,
+        )
+
+
+def _observation_mentioned_at(obs: TransferObservation) -> datetime | None:
+    """event_date (NOT NULL) is derived from occurred_start or mentioned_at on
+    insert; fall back so the column stays populated for observations too."""
+    mentioned_at = obs.mentioned_at
+    if obs.occurred_start is None and mentioned_at is None:
+        mentioned_at = obs.event_date or datetime.now(UTC)
+    return mentioned_at
+
+
+def _to_extracted_fact(fact: TransferFact) -> ExtractedFact:
+    """Rebuild the retain pipeline's ExtractedFact from a serialized transfer fact."""
+    # event_date is NOT NULL in the schema and is derived from occurred_start or
+    # mentioned_at on insert. When neither is present, fall back to the carried
+    # event_date (or now) via mentioned_at so the column stays populated.
+    mentioned_at = fact.mentioned_at
+    if fact.occurred_start is None and mentioned_at is None:
+        mentioned_at = fact.event_date or datetime.now(UTC)
+
+    return ExtractedFact(
+        fact_text=fact.text,
+        fact_type=fact.fact_type,
+        entities=list(fact.entities),
+        occurred_start=fact.occurred_start,
+        occurred_end=fact.occurred_end,
+        where=None,
+        causal_relations=[
+            CausalRelation(relation_type=rel.relation_type, target_fact_index=rel.target_fact_index)
+            for rel in fact.causal_relations
+        ],
+        content_index=0,
+        chunk_index=fact.chunk_index,
+        context=fact.context or "",
+        mentioned_at=mentioned_at,
+        metadata=dict(fact.metadata),
+        tags=list(fact.tags),
+        observation_scopes=fact.observation_scopes,
+    )
@@ -0,0 +1,138 @@
+"""Serialization schema for the document transfer archive (manifest + per-document payloads).
+
+The archive is a ZIP:
+
+    manifest.json              -- TransferManifest
+    documents/000000.json      -- TransferDocument (one file per document)
+    documents/000001.json
+    ...
+
+Documents are stored under a zero-padded index rather than their id so that
+arbitrary document ids (which may contain path-unsafe characters) never leak
+into archive entry names. The real id lives inside each payload.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Literal
+
+from pydantic import BaseModel, Field
+
+# Bump when the archive layout changes in a backward-incompatible way.
+SCHEMA_VERSION = 1
+
+ObservationScopes = Literal["per_tag", "combined", "all_combinations"] | list[list[str]]
+
+
+class TransferCausalRelation(BaseModel):
+    """A causal edge from this fact to an earlier fact in the same document.
+
+    ``target_fact_index`` is the ordinal of the target fact within the document's
+    ``facts`` list (not a database id), so it survives transfer to a new bank.
+    """
+
+    relation_type: str
+    target_fact_index: int
+
+
+class TransferFact(BaseModel):
+    """One extracted fact (memory unit) without its embedding or database id.
+
+    Everything here is reused verbatim on import except the embedding, which is
+    regenerated by the target bank's model, and the entity ids, which are
+    re-resolved against the target bank by canonical name.
+    """
+
+    text: str
+    fact_type: str
+    context: str | None = None
+    # event_date is a fallback used only when both occurred_start and
+    # mentioned_at are absent, to satisfy the NOT NULL event_date column.
+    event_date: datetime | None = None
+    occurred_start: datetime | None = None
+    occurred_end: datetime | None = None
+    mentioned_at: datetime | None = None
+    metadata: dict[str, str] = Field(default_factory=dict)
+    tags: list[str] = Field(default_factory=list)
+    observation_scopes: ObservationScopes | None = None
+    # Ordinal of the source chunk within the document (parsed from chunk_id).
+    chunk_index: int | None = None
+    # Entity canonical names; re-resolved against the target bank on import.
+    entities: list[str] = Field(default_factory=list)
+    causal_relations: list[TransferCausalRelation] = Field(default_factory=list)
+
+
+class TransferChunk(BaseModel):
+    """A raw text chunk of the source document, reused verbatim."""
+
+    chunk_index: int
+    chunk_text: str
+
+
+class TransferObservationSource(BaseModel):
+    """A reference to a source fact of an observation, by document + ordinal.
+
+    Observations span documents and reference their source facts by unit id;
+    those ids don't survive transfer, so each source is carried as the
+    (document_id, fact_index) of the fact within the exported document set.
+    """
+
+    document_id: str
+    fact_index: int
+
+
+class TransferObservation(BaseModel):
+    """A consolidated observation (``fact_type='observation'``).
+
+    Observations are bank-level (not tied to one document), carry no embedding
+    (re-generated on import) and no entity/link associations — retrieval reaches
+    entities/links through their source facts. Only exported when explicitly
+    requested, and only when every source resolves within the archive.
+    """
+
+    text: str
+    tags: list[str] = Field(default_factory=list)
+    event_date: datetime | None = None
+    occurred_start: datetime | None = None
+    occurred_end: datetime | None = None
+    mentioned_at: datetime | None = None
+    observation_scopes: ObservationScopes | None = None
+    proof_count: int = 1
+    sources: list[TransferObservationSource] = Field(default_factory=list)
+
+
+class TransferDocument(BaseModel):
+    """A single document plus its chunks and extracted facts."""
+
+    id: str
+    original_text: str | None = None
+    retain_params: dict | None = None
+    tags: list[str] = Field(default_factory=list)
+    created_at: datetime | None = None
+    chunks: list[TransferChunk] = Field(default_factory=list)
+    facts: list[TransferFact] = Field(default_factory=list)
+
+
+class TransferManifest(BaseModel):
+    """Top-level archive descriptor (``manifest.json``).
+
+    The bank-level fields default to a documents-only archive so older
+    document-only archives (and the document import path) keep parsing
+    unchanged; ``export_bank`` populates them for a whole-bank archive.
+    """
+
+    schema_version: int = SCHEMA_VERSION
+    source_bank_id: str
+    exported_at: datetime | None = None
+    document_count: int = 0
+    fact_count: int = 0
+    observation_count: int = 0
+    # "documents" = doc/fact/observation subset; "bank" = whole-bank export
+    # (also carries bank config, mental models, directives, webhooks).
+    archive_type: Literal["documents", "bank"] = "documents"
+    mental_model_count: int = 0
+    directive_count: int = 0
+    webhook_count: int = 0
+    # True when --include-history carried audit_log / llm_requests.
+    includes_history: bool = False
@@ -40,6 +40,11 @@ class Tenant:
    """

    schema: str
+    # Optional tenant identifier. When provided, background maintenance (e.g. the
+    # consolidation reconcile sweep) can build a RequestContext carrying this id so
+    # tenant-level config overrides are honored. Leave as None for single-tenant
+    # setups or extensions that do not key config by tenant id.
+    tenant_id: str | None = None


 class TenantExtension(Extension, ABC):
@@ -184,6 +184,8 @@ class MetricsCollectorBase:
        input_tokens: int = 0,
        output_tokens: int = 0,
        success: bool = True,
+        cached_input_tokens: int = 0,
+        thoughts_tokens: int = 0,
    ):
        """
        Record metrics for an LLM call.
@@ -193,9 +195,11 @@ class MetricsCollectorBase:
            model: Model name
            scope: Scope identifier (e.g., "memory", "reflect", "consolidation")
            duration: Call duration in seconds
-            input_tokens: Number of input/prompt tokens
-            output_tokens: Number of output/completion tokens
+            input_tokens: Number of input/prompt tokens (total)
+            output_tokens: Number of output/completion tokens visible in candidates
            success: Whether the call was successful
+            cached_input_tokens: Subset of input_tokens billed at the cached rate
+            thoughts_tokens: Reasoning tokens (billed as output, hidden from candidates)
        """
        raise NotImplementedError

@@ -233,6 +237,8 @@ class NoOpMetricsCollector(MetricsCollectorBase):
        input_tokens: int = 0,
        output_tokens: int = 0,
        success: bool = True,
+        cached_input_tokens: int = 0,
+        thoughts_tokens: int = 0,
    ):
        """No-op LLM call recording."""
        pass
@@ -287,6 +293,27 @@ class MetricsCollector(MetricsCollectorBase):
            name="hindsight.llm.calls.total", description="Total number of LLM API calls", unit="calls"
        )

+        # Cached input tokens (subset of input_tokens billed at the cached rate).
+        # Useful for tracking prompt-cache hit-rate independently of total
+        # input volume. provider.scope.model labels matche llm_tokens_input.
+        self.llm_tokens_cached_input = self.meter.create_counter(
+            name="hindsight.llm.tokens.cached_input",
+            description="Number of cached input tokens (billed at cached rate) for LLM calls",
+            unit="tokens",
+        )
+
+        # Thinking / reasoning tokens (Gemini 2.5+ family). Billed at the
+        # output rate by the provider but invisible to candidates_token_count.
+        # Surfacing them as a distinct counter is required for honest cost
+        # attribution: a workload that "looks cheap" by output volume can be
+        # silently expensive if the model is doing long reasoning chains.
+        self.llm_tokens_thoughts = self.meter.create_counter(
+            name="hindsight.llm.tokens.thoughts",
+            description="Number of reasoning/thinking tokens emitted by the model "
+            "(billed as output but not surfaced in candidates)",
+            unit="tokens",
+        )
+
        # HTTP request metrics
        self.http_request_duration = self.meter.create_histogram(
            name="hindsight.http.duration", description="Duration of HTTP requests in seconds", unit="s"
@@ -370,6 +397,8 @@ class MetricsCollector(MetricsCollectorBase):
        input_tokens: int = 0,
        output_tokens: int = 0,
        success: bool = True,
+        cached_input_tokens: int = 0,
+        thoughts_tokens: int = 0,
    ):
        """
        Record metrics for an LLM call.
@@ -379,9 +408,15 @@ class MetricsCollector(MetricsCollectorBase):
            model: Model name
            scope: Scope identifier (e.g., "memory", "reflect", "consolidation")
            duration: Call duration in seconds
-            input_tokens: Number of input/prompt tokens
-            output_tokens: Number of output/completion tokens
+            input_tokens: Number of input/prompt tokens (total, including cached portion)
+            output_tokens: Number of output/completion tokens visible in candidates
            success: Whether the call was successful
+            cached_input_tokens: Subset of input_tokens billed at the cached
+                rate (Gemini context caching). Defaults to 0 when caching is
+                disabled or the provider doesn't surface this field.
+            thoughts_tokens: Reasoning/thinking tokens (Gemini 2.5+ family).
+                Billed at the output rate but not counted in candidates.
+                Defaults to 0 for providers that don't emit thoughts.
        """
        # Base attributes for all metrics
        base_attributes = {
@@ -413,6 +448,18 @@ class MetricsCollector(MetricsCollectorBase):
            }
            self.llm_tokens_output.add(output_tokens, output_attributes)

+        if cached_input_tokens > 0:
+            self.llm_tokens_cached_input.add(
+                cached_input_tokens,
+                {**base_attributes, "token_bucket": get_token_bucket(cached_input_tokens)},
+            )
+
+        if thoughts_tokens > 0:
+            self.llm_tokens_thoughts.add(
+                thoughts_tokens,
+                {**base_attributes, "token_bucket": get_token_bucket(thoughts_tokens)},
+            )
+
    @contextmanager
    def record_http_request(self, method: str, endpoint: str, status_code_getter: Callable[[], int]):
        """
@@ -273,6 +273,8 @@ class LLMSpanRecorder:
        finish_reason: Optional[str] = None,
        error: Optional[Exception] = None,
        tool_calls: Optional[list[dict[str, Any]]] = None,
+        cached_tokens: int = 0,
+        **_extra: Any,
    ) -> None:
        """
        Record a completed LLM call as a span with GenAI semantic conventions.
@@ -293,6 +295,8 @@ class LLMSpanRecorder:
            finish_reason: Reason the model stopped (stop, length, tool_calls, etc.)
            error: Exception if call failed
            tool_calls: List of tool calls made (for function calling)
+            cached_tokens: Cached/cache-read prompt tokens, when reported by the provider.
+            _extra: Tolerated forward-compatible kwargs from other recorders.
        """
        try:
            # Map provider name to GenAI semantic convention
@@ -326,6 +330,8 @@ class LLMSpanRecorder:
                span.set_attribute(GenAIAttributes.RESPONSE_MODEL, model)
                span.set_attribute(GenAIAttributes.USAGE_INPUT_TOKENS, input_tokens)
                span.set_attribute(GenAIAttributes.USAGE_OUTPUT_TOKENS, output_tokens)
+                if cached_tokens:
+                    span.set_attribute("gen_ai.usage.cached_tokens", cached_tokens)

                # Add custom attributes for Hindsight context
                span.set_attribute("hindsight.scope", scope)
@@ -460,22 +466,61 @@ class NoOpLLMSpanRecorder:
        pass


-# Global span recorder instance
+class CompositeSpanRecorder:
+    """Fans out ``record_llm_call`` to every registered recorder.
+
+    This lets multiple GenAI consumers observe the same LLM calls — e.g. the
+    OpenTelemetry span exporter and the per-bank DB tracer — through the single
+    ``record_llm_call`` chokepoint each provider already calls. A failure in one
+    recorder never affects the others or the LLM call itself.
+    """
+
+    def __init__(self) -> None:
+        self._recorders: list[Any] = []
+
+    def register(self, recorder: Any) -> None:
+        if recorder not in self._recorders:
+            self._recorders.append(recorder)
+
+    def unregister(self, recorder: Any) -> None:
+        if recorder in self._recorders:
+            self._recorders.remove(recorder)
+
+    def record_llm_call(self, **kwargs: Any) -> None:
+        for recorder in self._recorders:
+            try:
+                recorder.record_llm_call(**kwargs)
+            except Exception as e:  # never let one recorder break others
+                logger.debug(f"Span recorder {type(recorder).__name__} failed: {e}", exc_info=True)
+
+
+# Global composite recorder — always present; fans out to whatever is registered.
+_composite_recorder = CompositeSpanRecorder()
+# Backward-compat reference to the OTel recorder (if created).
 _span_recorder: Optional[LLMSpanRecorder] = None


-def get_span_recorder() -> LLMSpanRecorder | NoOpLLMSpanRecorder:
-    """Get the global span recorder (NoOp if tracing disabled)."""
-    if _span_recorder is None:
-        return NoOpLLMSpanRecorder()
-    return _span_recorder
+def get_span_recorder() -> CompositeSpanRecorder:
+    """Get the global composite span recorder (fans out to all registered recorders)."""
+    return _composite_recorder
+
+
+def register_span_recorder(recorder: Any) -> None:
+    """Register an additional GenAI recorder (e.g. the per-bank DB tracer)."""
+    _composite_recorder.register(recorder)
+
+
+def unregister_span_recorder(recorder: Any) -> None:
+    """Remove a previously registered recorder."""
+    _composite_recorder.unregister(recorder)


 def create_span_recorder() -> LLMSpanRecorder:
-    """Create and set the global span recorder."""
+    """Create and register the OpenTelemetry span recorder."""
    global _span_recorder
    tracer = get_tracer()
    if tracer is None:
        raise RuntimeError("Tracing not initialized. Call initialize_tracing() first.")
    _span_recorder = LLMSpanRecorder(tracer)
+    register_span_recorder(_span_recorder)
    return _span_recorder
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"

 [project]
 name = "hindsight-api-slim"
-version = "0.7.1"
+version = "0.7.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -96,6 +96,13 @@ local-llm = [
    "llama-cpp-python[server]>=0.3.0",
    "huggingface-hub>=0.20.0",
 ]
+local-onnx = [
+    # In-process ONNX Runtime embeddings without an Ollama/TEI sidecar
+    "onnxruntime>=1.17.0",
+    "transformers>=4.53.0",
+    "huggingface-hub>=0.20.0",
+    "numpy>=1.26.0",
+]
 embedded-db = [
    "pg0-embedded>=0.14.2",
 ]
@@ -103,7 +110,7 @@ oracle = [
    "oracledb>=2.5.0",
 ]
 all = [
-    "hindsight-api-slim[local-ml,embedded-db]",
+    "hindsight-api-slim[local-ml,local-onnx,embedded-db]",
 ]
 test = [
    "pytest>=7.0.0",
@@ -20,6 +20,16 @@ from hindsight_api.pg0 import EmbeddedPostgres
 DEFAULT_PG0_INSTANCE_NAME = "hindsight-test"
 DEFAULT_PG0_PORT = int(os.environ.get("HINDSIGHT_TEST_PG_PORT", "5556"))

+# Keep the background MaintenanceLoop from auto-starting during tests. In
+# production it sweeps retention and re-schedules consolidation, but its timers
+# would race shared-pg0 test data (e.g. delete llm_requests/audit_log rows a test
+# just inserted). Disabling the reconcile interval and llm-trace retention — with
+# audit retention already off by default — leaves no job enabled, so the loop
+# never starts. Tests that exercise it call MaintenanceLoop methods
+# (_run_reconcile / _purge_expired) directly.
+os.environ.setdefault("HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS", "0")
+os.environ.setdefault("HINDSIGHT_API_LLM_TRACE_RETENTION_DAYS", "-1")
+

 # Load environment variables from .env at the start of test session
 def pytest_configure(config):
@@ -103,6 +103,11 @@ async def test_small_async_batch_no_splitting(memory, request_context):
    assert status["result_metadata"]["num_sub_batches"] == 1  # Single sub-batch
    assert len(status["child_operations"]) == 1
    assert status["child_operations"][0]["status"] == "completed"
+    child_meta = await _child_metadata(memory, bank_id, operation_id, request_context)
+    assert child_meta["unit_ids_count"] > 0
+    assert child_meta["extraction_errors_count"] == 0
+    assert status["result_metadata"]["unit_ids_count"] == child_meta["unit_ids_count"]
+    assert status["result_metadata"]["extraction_errors_count"] == 0


@pytest.mark.asyncio
@@ -166,6 +171,19 @@ async def test_large_async_batch_auto_splits(memory, request_context):

    # Parent status should be aggregated as "completed"
    assert parent_status["status"] == "completed"
+    child_unit_counts = []
+    for child in child_ops:
+        child_status = await memory.get_operation_status(
+            bank_id=bank_id,
+            operation_id=child["operation_id"],
+            request_context=request_context,
+        )
+        child_meta = child_status["result_metadata"]
+        assert child_meta["unit_ids_count"] > 0
+        assert child_meta["extraction_errors_count"] == 0
+        child_unit_counts.append(child_meta["unit_ids_count"])
+    assert parent_status["result_metadata"]["unit_ids_count"] == sum(child_unit_counts)
+    assert parent_status["result_metadata"]["extraction_errors_count"] == 0


@pytest.mark.asyncio
@@ -461,6 +479,42 @@ async def _child_metadata(memory, bank_id: str, parent_operation_id: str, reques
    return child["result_metadata"]


+@pytest.mark.asyncio
+async def test_retain_outcome_metadata_records_zero_counts(memory, request_context, monkeypatch):
+    """Completed retain operations expose explicit zero outcome counters."""
+    from hindsight_api.engine.response_models import TokenUsage
+    from hindsight_api.engine.retain import fact_extraction
+
+    async def empty_extract_facts_from_contents(
+        *args: object, **kwargs: object
+    ) -> tuple[list[object], list[object], TokenUsage]:
+        return [], [], TokenUsage()
+
+    monkeypatch.setattr(fact_extraction, "extract_facts_from_contents", empty_extract_facts_from_contents)
+
+    bank_id = "test_retain_outcome_zero_counts"
+    result = await memory.submit_async_retain(
+        bank_id=bank_id,
+        contents=[{"content": "No extracted facts for this item."}],
+        request_context=request_context,
+    )
+    await asyncio.sleep(0.2)
+
+    parent = await memory.get_operation_status(
+        bank_id=bank_id,
+        operation_id=result["operation_id"],
+        request_context=request_context,
+    )
+    child_meta = await _child_metadata(memory, bank_id, result["operation_id"], request_context)
+
+    assert child_meta["unit_ids_count"] == 0
+    assert child_meta["extraction_errors_count"] == 0
+    assert "extraction_errors_sample" not in child_meta
+    assert parent["result_metadata"]["unit_ids_count"] == 0
+    assert parent["result_metadata"]["extraction_errors_count"] == 0
+    assert "extraction_errors_sample" not in parent["result_metadata"]
+
+
@pytest.mark.asyncio
 async def test_retain_records_user_provided_document_ids(memory, request_context):
    """User-supplied document_ids land in child op result_metadata.document_ids."""
@@ -1060,3 +1114,67 @@ async def test_submit_async_batch_retain_rolls_back_parent_on_child_failure(
        f"{[(r['operation_type'], r['status'], r['task_payload'] is not None) for r in rows]}. "
        "The parent INSERT must be transactionally coupled to the child INSERTs."
    )
+
+
+@pytest.mark.asyncio
+async def test_submit_async_batch_retain_creates_missing_bank(memory, request_context, monkeypatch):
+    """First async retain to a new bank lazily creates the bank (async_operations
+    has an FK to banks) instead of raising a constraint error."""
+
+    async def noop_submit_task(_task_dict):
+        return None
+
+    monkeypatch.setattr(memory._task_backend, "submit_task", noop_submit_task)
+
+    bank_id = f"test_batch_newbank_{uuid.uuid4().hex[:8]}"
+    pool = await memory._get_pool()
+
+    await memory.submit_async_retain(
+        bank_id=bank_id,
+        contents=[{"content": "Alice works at Google.", "document_id": "doc1"}],
+        request_context=request_context,
+    )
+
+    bank = await pool.fetchrow("SELECT bank_id FROM banks WHERE bank_id = $1", bank_id)
+    assert bank is not None, "submit_async_retain should have lazily created the bank"
+
+
+@pytest.mark.asyncio
+async def test_submit_async_batch_retain_rolls_back_missing_bank_on_child_failure(
+    memory_no_llm_verify, request_context, monkeypatch
+):
+    """The lazy bank-create shares the parent+child transaction. When the child
+    loop fails for a bank that did not previously exist, the freshly-created bank
+    must roll back together with the operation rows — no orphan bank."""
+    import hindsight_api.engine.memory_engine as me
+    from hindsight_api.engine.memory_engine import count_tokens
+
+    bank_id = f"test_batch_bank_rollback_{uuid.uuid4().hex[:8]}"
+    pool = await memory_no_llm_verify._get_pool()
+    # Intentionally do NOT pre-create the bank — it must be created (and then
+    # rolled back) inside submit_async_retain's transaction.
+
+    large_content = "The quick brown fox jumps over the lazy dog. " * 500
+    contents = [{"content": large_content + f" item {i}", "document_id": f"doc{i}"} for i in range(2)]
+    assert sum(count_tokens(item["content"]) for item in contents) > 10_000
+
+    real_class = me.BatchRetainChildMetadata
+    call_count = {"n": 0}
+
+    def failing_child_metadata(*args, **kwargs):
+        call_count["n"] += 1
+        if call_count["n"] == 2:
+            raise RuntimeError("Simulated child-step failure mid-batch")
+        return real_class(*args, **kwargs)
+
+    monkeypatch.setattr(me, "BatchRetainChildMetadata", failing_child_metadata)
+
+    with pytest.raises(RuntimeError, match="Simulated child-step failure"):
+        await memory_no_llm_verify.submit_async_retain(
+            bank_id=bank_id,
+            contents=contents,
+            request_context=request_context,
+        )
+
+    bank = await pool.fetchrow("SELECT bank_id FROM banks WHERE bank_id = $1", bank_id)
+    assert bank is None, "the lazily-created bank must roll back with the failed operation inserts"
@@ -1,6 +1,6 @@
 """Unit tests for async retain tag propagation."""

-from unittest.mock import AsyncMock, MagicMock, patch
+from unittest.mock import AsyncMock, MagicMock

 import pytest

@@ -56,19 +56,18 @@ async def test_submit_async_retain_includes_document_tags_in_task_payload():
    contents = [{"content": "Async retain payload test."}]
    document_tags = ["scope:tools", "user:alice"]

-    # Return (profile, created=False) so the default-template-on-create hook is skipped.
-    with patch(
-        "hindsight_api.engine.memory_engine.bank_utils.get_or_create_bank_profile",
-        new_callable=AsyncMock,
-        return_value=(MagicMock(), False),
-    ):
-        result = await MemoryEngine.submit_async_retain(
-            engine,
-            bank_id="bank-1",
-            contents=contents,
-            document_tags=document_tags,
-            request_context=request_context,
-        )
+    # Stub the lazy bank-create/default-template hook to a no-op (created=False)
+    # so the inline transaction path runs against the mock connection without
+    # real DB work. The hook itself is covered by dedicated tests.
+    engine._ensure_bank_exists = AsyncMock(return_value=False)
+
+    result = await MemoryEngine.submit_async_retain(
+        engine,
+        bank_id="bank-1",
+        contents=contents,
+        document_tags=document_tags,
+        request_context=request_context,
+    )

    # Check result structure
    assert "operation_id" in result
@@ -5,6 +5,7 @@ Covers the new fields exposed by GET /v1/default/banks/{bank_id}/stats
 (operations_by_status) and the new endpoint
 GET /v1/default/banks/{bank_id}/stats/memories-timeseries.
 """
+
 import uuid
 from datetime import datetime

@@ -83,9 +84,7 @@ async def test_bank_stats_exposes_operations_by_status(api_client, test_bank_id)
        ("90d", 90, "day"),
    ],
 )
-async def test_memories_timeseries_periods(
-    api_client, test_bank_id, period, expected_count, expected_trunc
-):
+async def test_memories_timeseries_periods(api_client, test_bank_id, period, expected_count, expected_trunc):
    """Every period must return the full expected bucket count and trunc."""
    try:
        response = await api_client.post(
@@ -139,9 +138,7 @@ async def test_memories_timeseries_invalid_period_falls_back(api_client, test_ba


@pytest.mark.asyncio
-async def test_memories_timeseries_empty_bank_returns_zero_filled_buckets(
-    api_client, test_bank_id
-):
+async def test_memories_timeseries_empty_bank_returns_zero_filled_buckets(api_client, test_bank_id):
    """A bank with no memories must still return the full zero-filled bucket set."""
    try:
        response = await api_client.get(
@@ -242,3 +239,146 @@ async def test_list_memories_filter_by_consolidation_state_rejects_unknown(api_c
        assert response.status_code == 400
    finally:
        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
+
+
+@pytest.mark.asyncio
+async def test_bank_stats_link_counts_have_no_join(api_client, test_bank_id):
+    """link_counts must be populated; the deprecated breakdown fields must be empty.
+
+    Confirms the simplified single-table aggregation still produces the totals
+    the UI reads (`links_by_link_type`) without the historical
+    memory_links⇒memory_units join that powered the 2D `links_breakdown` no
+    consumer reads.
+    """
+    try:
+        response = await api_client.post(
+            f"/v1/default/banks/{test_bank_id}/memories",
+            json={"items": [{"content": "Carol leads platform engineering.", "context": "team"}]},
+        )
+        assert response.status_code == 200
+
+        response = await api_client.get(f"/v1/default/banks/{test_bank_id}/stats")
+        assert response.status_code == 200
+        stats = response.json()
+
+        # link totals must still come back so the UI overview cards render.
+        assert isinstance(stats["links_by_link_type"], dict)
+        assert stats["total_links"] >= 0
+
+        # Deprecated breakdown fields stay in the response shape but are empty.
+        assert stats["links_breakdown"] == {}
+        assert stats["links_by_fact_type"] == {}
+    finally:
+        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
+
+
+@pytest.mark.asyncio
+async def test_get_bank_freshness_returns_only_consolidation_fields(memory, test_bank_id):
+    """get_bank_freshness must return just the freshness keys, no link aggregation."""
+    from hindsight_api.extensions import RequestContext
+
+    try:
+        await _insert_memory(memory, test_bank_id, "Headed for consolidation.", failed=False)
+        await _insert_memory(memory, test_bank_id, "Also pending.", failed=True)
+
+        freshness = await memory.get_bank_freshness(
+            test_bank_id,
+            request_context=RequestContext(internal=True),
+        )
+
+        assert set(freshness.keys()) == {
+            "last_consolidated_at",
+            "pending_consolidation",
+            "failed_consolidation",
+        }
+        assert freshness["pending_consolidation"] >= 2
+        assert freshness["failed_consolidation"] >= 1
+    finally:
+        await memory._bank_stats_cache.clear()
+        async with memory._pool.acquire() as conn:
+            await conn.execute("DELETE FROM memory_units WHERE bank_id = $1", test_bank_id)
+
+
+@pytest.mark.asyncio
+async def test_reflect_uses_freshness_not_bank_stats(memory, test_bank_id):
+    """reflect() must call the cheap freshness query, not get_bank_stats.
+
+    Counts calls to `_compute_bank_stats` (the heavy loader) during a reflect
+    invocation; it must stay at zero — reflect should route through
+    `get_bank_freshness` instead.
+    """
+    from hindsight_api.extensions import RequestContext
+
+    try:
+        # Seed a single memory so reflect has something to inspect.
+        await _insert_memory(memory, test_bank_id, "Reflect seed.", failed=False)
+
+        compute_calls = 0
+        original_compute = memory._compute_bank_stats
+
+        async def counting_compute(bank_id: str):
+            nonlocal compute_calls
+            compute_calls += 1
+            return await original_compute(bank_id)
+
+        memory._compute_bank_stats = counting_compute  # type: ignore[method-assign]
+        try:
+            await memory._bank_stats_cache.clear()
+            try:
+                await memory.reflect(
+                    test_bank_id,
+                    "What do you know about this bank?",
+                    request_context=RequestContext(internal=True),
+                )
+            except Exception:
+                # reflect may fail without a configured LLM in this test env;
+                # we only care that it did not invoke the heavy stats loader
+                # before failing.
+                pass
+            assert compute_calls == 0
+        finally:
+            memory._compute_bank_stats = original_compute  # type: ignore[method-assign]
+    finally:
+        await memory._bank_stats_cache.clear()
+        async with memory._pool.acquire() as conn:
+            await conn.execute("DELETE FROM memory_units WHERE bank_id = $1", test_bank_id)
+
+
+@pytest.mark.asyncio
+async def test_bank_stats_served_from_cache_on_repeat_call(api_client, memory, test_bank_id):
+    """A second /stats call within the TTL must not re-run the aggregations.
+
+    The cache layer wraps the DB-heavy `_compute_bank_stats` body; counting
+    its invocations is the cleanest way to prove the wiring works without
+    relying on timing.
+    """
+    try:
+        response = await api_client.post(
+            f"/v1/default/banks/{test_bank_id}/memories",
+            json={"items": [{"content": "Bob is a project manager.", "context": "team"}]},
+        )
+        assert response.status_code == 200
+
+        original = memory._compute_bank_stats
+        call_count = 0
+
+        async def counting_compute(bank_id: str):
+            nonlocal call_count
+            call_count += 1
+            return await original(bank_id)
+
+        # Make sure no stale entry exists from prior test ordering.
+        await memory._bank_stats_cache.clear()
+        memory._compute_bank_stats = counting_compute  # type: ignore[method-assign]
+        try:
+            first = await api_client.get(f"/v1/default/banks/{test_bank_id}/stats")
+            second = await api_client.get(f"/v1/default/banks/{test_bank_id}/stats")
+            assert first.status_code == 200
+            assert second.status_code == 200
+            assert first.json() == second.json()
+            assert call_count == 1
+        finally:
+            memory._compute_bank_stats = original  # type: ignore[method-assign]
+            await memory._bank_stats_cache.clear()
+    finally:
+        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
@@ -0,0 +1,201 @@
+"""Unit tests for `BankStatsCache` — TTL, eviction, and concurrent coalescing.
+
+These tests don't touch the database; they exercise the cache wrapper
+directly so the semantics are checked in isolation from `MemoryEngine`.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Any
+
+import pytest
+
+from hindsight_api.engine.bank_stats_cache import BankStatsCache
+
+
+def make_loader(return_value: dict[str, Any]) -> tuple[Any, list[int]]:
+    """Returns (loader_fn, call_count_list). `call_count_list[0]` is the count."""
+    calls = [0]
+
+    async def loader() -> dict[str, Any]:
+        calls[0] += 1
+        return return_value
+
+    return loader, calls
+
+
+@pytest.mark.asyncio
+async def test_cache_disabled_passes_through() -> None:
+    cache = BankStatsCache(ttl_seconds=0, max_entries=100)
+    loader, calls = make_loader({"v": 1})
+
+    for _ in range(3):
+        result = await cache.get_or_load("schema", "bank", loader)
+        assert result == {"v": 1}
+    assert calls[0] == 3
+
+
+@pytest.mark.asyncio
+async def test_cache_serves_hits_within_ttl() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    loader, calls = make_loader({"v": 1})
+
+    first = await cache.get_or_load("schema", "bank", loader)
+    second = await cache.get_or_load("schema", "bank", loader)
+    assert first == second == {"v": 1}
+    assert calls[0] == 1
+
+
+@pytest.mark.asyncio
+async def test_cache_reloads_after_ttl_expires(monkeypatch) -> None:
+    cache = BankStatsCache(ttl_seconds=0.05, max_entries=100)
+    loader, calls = make_loader({"v": 1})
+
+    fake_time = [1000.0]
+    monkeypatch.setattr(cache, "_now", lambda: fake_time[0])
+
+    await cache.get_or_load("schema", "bank", loader)
+    fake_time[0] += 0.1  # advance past TTL
+    await cache.get_or_load("schema", "bank", loader)
+    assert calls[0] == 2
+
+
+@pytest.mark.asyncio
+async def test_cache_isolates_by_schema_and_bank() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    loader, calls = make_loader({"v": 1})
+
+    await cache.get_or_load("schema_a", "bank", loader)
+    await cache.get_or_load("schema_b", "bank", loader)
+    await cache.get_or_load("schema_a", "other", loader)
+    # 3 distinct keys → 3 loader calls.
+    assert calls[0] == 3
+
+
+@pytest.mark.asyncio
+async def test_concurrent_misses_are_coalesced() -> None:
+    """6 concurrent callers on the same cold key must trigger exactly one loader."""
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    calls = [0]
+    started = asyncio.Event()
+    release = asyncio.Event()
+
+    async def slow_loader() -> dict[str, Any]:
+        calls[0] += 1
+        started.set()
+        await release.wait()
+        return {"v": calls[0]}
+
+    tasks = [asyncio.create_task(cache.get_or_load("schema", "bank", slow_loader)) for _ in range(6)]
+    await started.wait()
+    # All other tasks should now be queued behind the in-flight loader.
+    release.set()
+    results = await asyncio.gather(*tasks)
+
+    assert calls[0] == 1
+    assert all(r == {"v": 1} for r in results)
+
+
+@pytest.mark.asyncio
+async def test_loader_exception_does_not_poison_cache() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    calls = [0]
+
+    async def flaky_loader() -> dict[str, Any]:
+        calls[0] += 1
+        if calls[0] == 1:
+            raise RuntimeError("boom")
+        return {"v": calls[0]}
+
+    with pytest.raises(RuntimeError, match="boom"):
+        await cache.get_or_load("schema", "bank", flaky_loader)
+
+    # Second call should still attempt the loader (cache wasn't populated).
+    result = await cache.get_or_load("schema", "bank", flaky_loader)
+    assert result == {"v": 2}
+    assert calls[0] == 2
+
+
+@pytest.mark.asyncio
+async def test_concurrent_loader_exception_propagates_to_waiters() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    started = asyncio.Event()
+    release = asyncio.Event()
+
+    async def failing_loader() -> dict[str, Any]:
+        started.set()
+        await release.wait()
+        raise RuntimeError("loader failed")
+
+    tasks = [asyncio.create_task(cache.get_or_load("schema", "bank", failing_loader)) for _ in range(3)]
+    await started.wait()
+    release.set()
+    results = await asyncio.gather(*tasks, return_exceptions=True)
+    assert all(isinstance(r, RuntimeError) for r in results)
+
+
+@pytest.mark.asyncio
+async def test_lru_eviction_respects_max_entries() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=2)
+
+    async def loader_for(value: int):
+        async def _loader() -> dict[str, Any]:
+            return {"v": value}
+
+        return _loader
+
+    await cache.get_or_load("s", "a", await loader_for(1))
+    await cache.get_or_load("s", "b", await loader_for(2))
+    # Touch "a" so it's most-recently-used.
+    await cache.get_or_load("s", "a", await loader_for(99))
+    # Insert "c" — should evict "b" (the LRU), not "a".
+    await cache.get_or_load("s", "c", await loader_for(3))
+
+    # "a" is still cached (loader for "a" with value=99 must NOT be called again).
+    miss_check_calls = [0]
+
+    async def should_not_run() -> dict[str, Any]:
+        miss_check_calls[0] += 1
+        return {"v": -1}
+
+    cached_a = await cache.get_or_load("s", "a", should_not_run)
+    assert cached_a == {"v": 1}
+    assert miss_check_calls[0] == 0
+
+    # "b" was evicted; the loader must run on the next get.
+    new_b_calls = [0]
+
+    async def new_b() -> dict[str, Any]:
+        new_b_calls[0] += 1
+        return {"v": 200}
+
+    fetched_b = await cache.get_or_load("s", "b", new_b)
+    assert fetched_b == {"v": 200}
+    assert new_b_calls[0] == 1
+
+
+@pytest.mark.asyncio
+async def test_invalidate_drops_entry() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    loader, calls = make_loader({"v": 1})
+
+    await cache.get_or_load("schema", "bank", loader)
+    await cache.invalidate("schema", "bank")
+    await cache.get_or_load("schema", "bank", loader)
+    assert calls[0] == 2
+
+
+@pytest.mark.asyncio
+async def test_clear_drops_all_entries() -> None:
+    cache = BankStatsCache(ttl_seconds=60, max_entries=100)
+    loader, calls = make_loader({"v": 1})
+
+    await cache.get_or_load("s", "a", loader)
+    await cache.get_or_load("s", "b", loader)
+    assert calls[0] == 2
+
+    await cache.clear()
+    await cache.get_or_load("s", "a", loader)
+    await cache.get_or_load("s", "b", loader)
+    assert calls[0] == 4
@@ -7,21 +7,23 @@ Tests cover:
 - Hard error when provider doesn't support the batch API (no silent fallback)
 - Worker recovery on restart
 """
-import pytest
 import asyncio
-import logging
 import json
+import logging
 import uuid
 from datetime import datetime, timezone
 from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
 from hindsight_api import RequestContext
-from hindsight_api.engine.retain.fact_extraction import (
-    extract_facts_from_contents_batch_api,
-    extract_facts_from_contents,
-    RetainContent,
-)
 from hindsight_api.config import HindsightConfig
 from hindsight_api.engine.llm_wrapper import create_llm_provider
+from hindsight_api.engine.retain.fact_extraction import (
+    RetainContent,
+    extract_facts_from_contents,
+    extract_facts_from_contents_batch_api,
+)
 from hindsight_api.worker.poller import WorkerPoller

 logger = logging.getLogger(__name__)
@@ -331,6 +333,101 @@ async def test_batch_api_crash_recovery(mock_llm_config, test_contents, hindsigh
            pass


+@pytest.mark.asyncio
+async def test_batch_api_records_non_fatal_extraction_errors(
+    mock_llm_config, test_contents, hindsight_config, memory, request_context
+):
+    """Batch API skipped chunks are surfaced in operation result_metadata."""
+    bank_id = f"test_batch_errors_{datetime.now(timezone.utc).timestamp()}"
+    operation_id = str(uuid.uuid4())
+
+    try:
+        await memory.get_bank_profile(bank_id, request_context=request_context)
+        pool = memory._pool
+        schema = request_context.tenant_id
+
+        from hindsight_api.engine.task_backend import fq_table
+
+        table = fq_table("async_operations", schema)
+        await pool.execute(
+            f"""
+            INSERT INTO {table} (operation_id, operation_type, bank_id, status, result_metadata)
+            VALUES ($1, 'retain', $2, 'processing', $3::jsonb)
+            """,
+            operation_id,
+            bank_id,
+            json.dumps({}),
+        )
+
+        batch_id = "batch_partial_errors"
+        mock_llm_config._provider_impl.supports_batch_api = AsyncMock(return_value=True)
+        mock_llm_config._provider_impl.submit_batch = AsyncMock(return_value={"batch_id": batch_id})
+        mock_llm_config._provider_impl.get_batch_status = AsyncMock(
+            return_value={
+                "status": "completed",
+                "request_counts": {"total": 2, "completed": 2, "failed": 0},
+            }
+        )
+        mock_llm_config._provider_impl.retrieve_batch_results = AsyncMock(
+            return_value=[
+                {
+                    "custom_id": "chunk_0",
+                    "response": {
+                        "body": {
+                            "choices": [
+                                {
+                                    "message": {
+                                        "content": json.dumps({
+                                            "facts": [
+                                                {
+                                                    "what": "Alice is a senior software engineer",
+                                                    "when": "present",
+                                                    "where": "TechCorp",
+                                                    "who": "Alice",
+                                                    "why": "Background",
+                                                    "fact_type": "world",
+                                                    "fact_kind": "conversation",
+                                                }
+                                            ]
+                                        })
+                                    }
+                                }
+                            ],
+                            "usage": {"prompt_tokens": 100, "completion_tokens": 50, "total_tokens": 150},
+                        }
+                    },
+                }
+            ]
+        )
+
+        facts, chunks, usage = await extract_facts_from_contents_batch_api(
+            contents=test_contents,
+            llm_config=mock_llm_config,
+            agent_name="test_agent",
+            config=hindsight_config,
+            pool=pool,
+            operation_id=operation_id,
+            schema=schema,
+        )
+
+        assert len(facts) == 1
+        assert len(chunks) == 2
+        assert chunks[1].fact_count == 0
+        assert usage.total_tokens == 150
+
+        row = await pool.fetchrow(f"SELECT result_metadata FROM {table} WHERE operation_id = $1", operation_id)
+        metadata = json.loads(row["result_metadata"]) if isinstance(row["result_metadata"], str) else row["result_metadata"]
+        assert metadata["batch_id"] == batch_id
+        assert metadata["extraction_errors_count"] == 1
+        assert metadata["extraction_errors_sample"] == ["chunk_1: missing batch result"]
+
+    finally:
+        try:
+            await memory.delete_bank(bank_id, request_context=request_context)
+        except Exception:
+            pass
+
+
@pytest.mark.asyncio
 async def test_batch_api_raises_for_unsupported_provider(mock_llm_config, test_contents, hindsight_config):
    """Batch extraction must surface a hard error (not silently fall back) when
@@ -22,6 +22,7 @@ def setup_test_env():
        "HINDSIGHT_API_LLM_PROVIDER",
        "HINDSIGHT_API_LLM_MODEL",
        "HINDSIGHT_API_LLM_REASONING_EFFORT",
+        "HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY",
        "HINDSIGHT_API_DATABASE_URL",
        "HINDSIGHT_API_MIGRATION_DATABASE_URL",
    ]
@@ -103,6 +104,26 @@ def test_valid_retain_config_succeeds():
    assert config.retain_chunk_size == 3000


+def test_semantic_min_similarity_reads_from_env():
+    """Semantic retrieval min similarity can be configured at the server level."""
+    from hindsight_api.config import HindsightConfig
+
+    os.environ["HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY"] = "0.58"
+
+    config = HindsightConfig.from_env()
+    assert config.semantic_min_similarity == 0.58
+
+
+def test_semantic_min_similarity_must_be_between_zero_and_one():
+    """Invalid semantic min similarity fails fast during configuration loading."""
+    from hindsight_api.config import HindsightConfig
+
+    os.environ["HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY"] = "1.5"
+
+    with pytest.raises(ValueError, match="semantic_min_similarity"):
+        HindsightConfig.from_env()
+
+
 def test_log_config_masks_database_urls(caplog):
    """Config startup logs must not expose database credentials."""
    from hindsight_api.config import HindsightConfig
@@ -376,3 +397,48 @@ def test_llm_reasoning_effort_loaded_from_env(monkeypatch):

    config = HindsightConfig.from_env()
    assert config.llm_reasoning_effort == "xhigh"
+
+
+# ---------------------------------------------------------------------------
+# Recall candidate gating (BM25 score floor + per-source cap) — issue #1707
+# ---------------------------------------------------------------------------
+
+
+def test_bm25_min_score_defaults_to_zero(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.delenv("HINDSIGHT_API_BM25_MIN_SCORE", raising=False)
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.bm25_min_score == 0.0
+
+
+def test_bm25_min_score_loaded_from_env(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_BM25_MIN_SCORE", "1.5")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.bm25_min_score == 1.5
+
+
+def test_recall_max_candidates_per_source_defaults_to_disabled(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.delenv("HINDSIGHT_API_RECALL_MAX_CANDIDATES_PER_SOURCE", raising=False)
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.recall_max_candidates_per_source == 0
+
+
+def test_recall_max_candidates_per_source_loaded_from_env(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_RECALL_MAX_CANDIDATES_PER_SOURCE", "150")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.recall_max_candidates_per_source == 150
@@ -464,7 +464,7 @@ class TestConsolidationIntegration:
        async with memory._pool.acquire() as conn:
            observations = await conn.fetch(
                """
-                SELECT id, text, source_memory_ids, history
+                SELECT id, text, source_memory_ids
                FROM memory_units
                WHERE bank_id = $1 AND fact_type = 'observation'
                """,
@@ -3040,10 +3040,13 @@ def _make_mock_llm_one_obs_per_fact():
    def callback(messages, scope):
        if scope != "consolidation":
            return _ConsolidationBatchResponse()
-        # Parse all fact UUIDs from the prompt — one create per fact
+        # Parse all fact UUIDs from the prompt — one create per fact. Read only
+        # the user message(s): consolidation sends the facts there, while the
+        # stable (cacheable) system message carries example UUIDs in its OUTPUT
+        # FORMAT samples that must not be mistaken for real facts.
        import re

-        prompt = messages[0]["content"] if messages else ""
+        prompt = "\n".join(m.get("content", "") for m in messages if m.get("role") == "user")
        fact_ids = re.findall(r"\[([0-9a-f-]{36})\]", prompt)
        creates = [_CreateAction(text=f"Observation about fact {fid[:8]}", source_fact_ids=[fid]) for fid in fact_ids]
        return _ConsolidationBatchResponse(creates=creates)
@@ -3145,7 +3148,9 @@ async def test_max_observations_per_scope_allows_updates_at_capacity(memory: Mem
            call_count += 1
            import re

-            prompt = messages[0]["content"] if messages else ""
+            # Facts live in the user message; the system message (stable, cached)
+            # carries example UUIDs in its OUTPUT samples — read user only.
+            prompt = "\n".join(m.get("content", "") for m in messages if m.get("role") == "user")
            fact_ids = re.findall(r"\[([0-9a-f-]{36})\]", prompt)
            if call_count == 1 and fact_ids:
                # First call: create an observation
@@ -3512,3 +3517,52 @@ async def test_enable_auto_consolidation_flag(memory: MemoryEngine, request_cont
    finally:
        memory._config_resolver._global_config = original_global_config
        await memory.delete_bank(bank_id, request_context=request_context)
+
+
+def test_consolidation_prompt_split_is_cacheable_and_complete():
+    """The split consolidation prompt: bank-agnostic system prefix + per-batch user.
+
+    The system prefix must be byte-identical across batches AND across banks (the
+    property that lets a single Gemini context cache serve every bank), carry only
+    stable instructions, and the per-batch/per-bank data (mission, facts,
+    observations, capacity note) must live in the user message — never in the
+    cached prefix.
+    """
+    from hindsight_api.engine.consolidation.prompts import (
+        build_consolidation_input,
+        build_consolidation_system_prompt,
+    )
+
+    sys_prompt = build_consolidation_system_prompt()
+    # Byte-stable across calls and independent of any mission → one cache for all banks.
+    assert sys_prompt == build_consolidation_system_prompt()
+    # Instructions only: no per-batch placeholders leaked into the prefix.
+    assert "{facts_text}" not in sys_prompt
+    assert "{observations_text}" not in sys_prompt
+    # JSON examples are unescaped (single braces), i.e. .format() ran.
+    assert '{"creates"' in sys_prompt
+    assert "{{" not in sys_prompt
+    # The stable observation-format boilerplate lives in the cached prefix.
+    assert "proof_count" in sys_prompt
+
+    # Two banks with DIFFERENT missions share the identical cached prefix; the
+    # mission rides in the per-batch user message instead.
+    user_a = build_consolidation_input(
+        facts_text="[id-a] Fact A.", observations_text="[]", observations_mission="Track widgets."
+    )
+    user_b = build_consolidation_input(
+        facts_text="[id-b] Fact B.", observations_text="[]", observations_mission="Track gadgets."
+    )
+    assert "Track widgets." in user_a
+    assert "Track widgets." not in sys_prompt  # mission NOT in the cached prefix
+    assert "Fact A." in user_a
+    assert user_a != user_b
+    # The format boilerplate is NOT re-sent per batch (it's in the cached prefix).
+    assert "proof_count" not in user_a
+
+    # The capacity note is per-batch too — kept out of the cached prefix.
+    capped = build_consolidation_input(
+        facts_text="[id] F.", observations_text="[]", observation_capacity_note="OBSERVATION LIMIT REACHED"
+    )
+    assert "OBSERVATION LIMIT REACHED" in capped
+    assert "OBSERVATION LIMIT REACHED" not in sys_prompt
@@ -0,0 +1,259 @@
+"""Deterministic unit tests for the consolidation duplicate-create guard.
+
+These exercise the dedup decision directly (no LLM, no DB), so they reliably
+guard the fix in CI — unlike the real-LLM integration test, which only triggers
+the path stochastically.
+"""
+
+import types
+import uuid
+from dataclasses import dataclass
+from unittest.mock import AsyncMock, patch
+
+from hindsight_api.engine.consolidation.consolidator import (
+    _dedup_active,
+    _dedup_reconcile_create,
+    _dedup_reconcile_update,
+    _DedupDecision,
+    _duplicate_create_target,
+    _norm_obs_text,
+)
+from hindsight_api.engine.search.types import RetrievalResult
+
+
+@dataclass
+class _FakeObs:
+    id: str
+    text: str
+
+
+def _shown(*observations: _FakeObs) -> dict[str, _FakeObs]:
+    return {_norm_obs_text(o.text): o for o in observations}
+
+
+def test_norm_obs_text_collapses_whitespace_preserves_case() -> None:
+    # Whitespace (incl. newlines) collapses; case is preserved.
+    assert _norm_obs_text("  The  User  likes BASIL.\n") == "The User likes BASIL."
+    assert _norm_obs_text(None) == ""
+
+
+def test_create_matching_shown_observation_is_duplicate() -> None:
+    shown = _shown(_FakeObs(id="11111111-aaaa", text="User waters the herbs early in the morning."))
+    # Same text with only-whitespace differences still matches.
+    target = _duplicate_create_target("User waters the   herbs early in the morning.", shown, set())
+    assert target is not None
+    assert target.startswith("shown observation 11111111")
+
+
+def test_create_differing_only_in_case_is_not_duplicate() -> None:
+    # Case-folding would lose information (e.g. acronyms), so a case-only difference
+    # is treated as novel rather than silently dropped.
+    shown = _shown(_FakeObs(id="22222222-bbbb", text="The user prefers TLS."))
+    assert _duplicate_create_target("The user prefers tls.", shown, set()) is None
+
+
+def test_create_matching_inresponse_update_is_duplicate() -> None:
+    update_texts = {_norm_obs_text("Mint is kept in its own separate bed.")}
+    target = _duplicate_create_target("Mint is kept in its own separate bed.", {}, update_texts)
+    assert target == "an UPDATE in this response"
+
+
+def test_novel_create_is_not_duplicate() -> None:
+    shown = _shown(_FakeObs(id="22222222-bbbb", text="User waters the herbs early in the morning."))
+    assert _duplicate_create_target("Rosemary is drought-tolerant.", shown, set()) is None
+    assert _duplicate_create_target("", {}, set()) is None
+
+
+# ── semantic dedup (_dedup_reconcile_create) ──────────────────────────────────
+#
+# Mocks the embedder, the obs-anchored ANN probe, and the LLM so the decision logic is
+# tested without a DB or a real model.
+
+_TWIN_ID = "33333333-3333-4333-8333-333333333333"
+
+
+def _obs(text: str, sim: float, oid: str = _TWIN_ID) -> RetrievalResult:
+    return RetrievalResult(id=oid, text=text, fact_type="observation", similarity=sim)
+
+
+def _ctx(threshold: float = 0.97):
+    """Return (kwargs, conn_mock, llm_mock) for a _dedup_reconcile_create call."""
+    conn = AsyncMock()
+    llm = types.SimpleNamespace(call=AsyncMock())
+    kwargs = dict(
+        conn=conn,
+        memory_engine=types.SimpleNamespace(embeddings=object()),
+        bank_id="bank1",
+        config=types.SimpleNamespace(consolidation_dedup_threshold=threshold),
+        dedup_llm_config=llm,
+        create_text="YouTube content in Uzbek is very rich.",
+        create_source_ids=[uuid.uuid4()],
+        tags=["t1"],
+    )
+    return kwargs, conn, llm
+
+
+def _patch_probe(results):
+    return patch(
+        "hindsight_api.engine.search.retrieval.retrieve_semantic_bm25_combined",
+        AsyncMock(return_value={"observation": (results, [])}),
+    )
+
+
+def _patch_embed():
+    return patch(
+        "hindsight_api.engine.retain.embedding_utils.generate_embeddings_batch",
+        AsyncMock(return_value=[[0.1, 0.2, 0.3]]),
+    )
+
+
+async def test_dedup_no_twin_above_threshold_returns_none() -> None:
+    kwargs, conn, llm = _ctx(threshold=0.97)
+    with _patch_embed(), _patch_probe([_obs("something loosely related", 0.81)]):
+        result = await _dedup_reconcile_create(**kwargs)
+    assert result is None
+    llm.call.assert_not_called()  # below threshold → no LLM call
+    conn.execute.assert_not_called()  # no merge
+
+
+async def test_dedup_llm_keep_does_not_merge() -> None:
+    kwargs, conn, llm = _ctx()
+    llm.call.return_value = _DedupDecision(action="keep", reason="different language")
+    with _patch_embed(), _patch_probe([_obs("Uzbek content on YouTube is described as very rich.", 0.98)]):
+        result = await _dedup_reconcile_create(**kwargs)
+    assert result is None
+    llm.call.assert_awaited_once()
+    conn.execute.assert_not_called()  # kept distinct → no merge
+
+
+async def test_dedup_llm_merge_folds_into_twin() -> None:
+    kwargs, conn, llm = _ctx()
+    kwargs["create_source_ids"] = [uuid.uuid4(), uuid.uuid4()]
+    llm.call.return_value = _DedupDecision(action="merge", text="Uzbek content on YouTube is very rich.")
+    with _patch_embed(), _patch_probe([_obs("Uzbek content on YouTube is described as very rich.", 0.99)]):
+        result = await _dedup_reconcile_create(**kwargs)
+    assert result == _TWIN_ID  # merged into the twin; caller skips the CREATE
+    conn.execute.assert_awaited_once()
+    args = conn.execute.await_args.args
+    assert args[1] == "Uzbek content on YouTube is very rich."  # merged text persisted
+    assert args[2] == kwargs["create_source_ids"]  # new source facts folded in
+    assert args[3] == uuid.UUID(_TWIN_ID)  # onto the twin row
+
+
+async def test_dedup_picks_highest_above_threshold_skips_below() -> None:
+    # Only the >=threshold candidate is considered; a 0.95 result is ignored at threshold 0.97.
+    kwargs, conn, llm = _ctx(threshold=0.97)
+    llm.call.return_value = _DedupDecision(action="keep")
+    with _patch_embed(), _patch_probe([_obs("near but distinct", 0.95), _obs("the real twin", 0.98)]):
+        await _dedup_reconcile_create(**kwargs)
+    # the twin passed to the LLM is the >=0.97 one, not the 0.95
+    sent = llm.call.await_args.kwargs["messages"][0]["content"]
+    assert "the real twin" in sent
+    assert "near but distinct" not in sent
+
+
+# ── UPDATE-path dedup (_dedup_reconcile_update) ───────────────────────────────
+#
+# An UPDATE rewrites+re-embeds an observation, which can drift it into a near-twin of a
+# DIFFERENT existing observation. These cover the fold-and-delete reconciliation (unlike
+# CREATE, both rows already exist), the self-exclusion, and the keep/no-twin no-ops.
+
+_UPDATED_ID = "44444444-4444-4444-8444-444444444444"
+
+
+def _update_ctx(threshold: float = 0.97):
+    """Return (kwargs, conn_mock, llm_mock) for a _dedup_reconcile_update call."""
+    conn = AsyncMock()
+    llm = types.SimpleNamespace(call=AsyncMock())
+    kwargs = dict(
+        conn=conn,
+        memory_engine=types.SimpleNamespace(embeddings=object()),
+        bank_id="bank1",
+        config=types.SimpleNamespace(consolidation_dedup_threshold=threshold),
+        dedup_llm_config=llm,
+        updated_id=_UPDATED_ID,
+        updated_text="Uzbek content on YouTube is very rich and growing.",
+        updated_emb_str="[0.1, 0.2, 0.3]",  # already embedded by _execute_update_action
+        tags=["t1"],
+    )
+    return kwargs, conn, llm
+
+
+async def test_dedup_update_merge_folds_into_twin_and_deletes_updated() -> None:
+    kwargs, conn, llm = _update_ctx()
+    llm.call.return_value = _DedupDecision(action="merge", text="Uzbek YouTube content is very rich and growing.")
+    with _patch_probe([_obs("Uzbek content on YouTube is described as very rich.", 0.98)]):
+        await _dedup_reconcile_update(**kwargs)
+    llm.call.assert_awaited_once()
+    # Two writes: fold-into-twin UPDATE, then DELETE of the updated row.
+    assert conn.execute.await_count == 2
+    fold_args = conn.execute.await_args_list[0].args
+    assert fold_args[1] == "Uzbek YouTube content is very rich and growing."  # merged text on the twin
+    assert fold_args[2] == uuid.UUID(_TWIN_ID)  # survivor = the twin
+    assert fold_args[3] == uuid.UUID(_UPDATED_ID)  # folded-from = the updated row
+    delete_args = conn.execute.await_args_list[1].args
+    assert delete_args[1] == uuid.UUID(_UPDATED_ID)  # the updated row is deleted
+
+
+async def test_dedup_update_keep_does_not_merge() -> None:
+    kwargs, conn, llm = _update_ctx()
+    llm.call.return_value = _DedupDecision(action="keep", reason="different growth claim")
+    with _patch_probe([_obs("Uzbek content on YouTube is described as very rich.", 0.98)]):
+        await _dedup_reconcile_update(**kwargs)
+    llm.call.assert_awaited_once()
+    conn.execute.assert_not_called()  # kept distinct → neither fold nor delete
+
+
+async def test_dedup_update_excludes_self() -> None:
+    # The probe surfaces the updated observation itself at 1.0; it must be excluded so we don't
+    # "merge" a row into itself. With no other candidate, there is no twin → no LLM, no writes.
+    kwargs, conn, llm = _update_ctx()
+    with _patch_probe([_obs("its own current text", 1.0, oid=_UPDATED_ID)]):
+        await _dedup_reconcile_update(**kwargs)
+    llm.call.assert_not_called()
+    conn.execute.assert_not_called()
+
+
+async def test_dedup_update_no_twin_above_threshold() -> None:
+    kwargs, conn, llm = _update_ctx(threshold=0.97)
+    with _patch_probe([_obs("loosely related", 0.8)]):
+        await _dedup_reconcile_update(**kwargs)
+    llm.call.assert_not_called()
+    conn.execute.assert_not_called()
+
+
+# ── dedup activation gate (_dedup_active) ─────────────────────────────────────
+#
+# Enabled by default (threshold < 1.0), but skipped on Oracle because the merge path is
+# Postgres-only — so the feature can ship on-by-default without breaking Oracle.
+
+
+def _gate_cfg(threshold: float):
+    return types.SimpleNamespace(consolidation_dedup_threshold=threshold)
+
+
+def _patch_backend(name: str):
+    return patch(
+        "hindsight_api.engine.consolidation.consolidator.get_config",
+        return_value=types.SimpleNamespace(database_backend=name),
+    )
+
+
+def test_dedup_active_enabled_on_postgres() -> None:
+    with _patch_backend("postgresql"):
+        assert _dedup_active(_gate_cfg(0.97)) is True
+
+
+def test_dedup_active_disabled_when_threshold_is_one() -> None:
+    with _patch_backend("postgresql"):
+        assert _dedup_active(_gate_cfg(1.0)) is False
+
+
+def test_dedup_active_skipped_on_oracle() -> None:
+    # PG-only merge path → dedup is skipped on Oracle even with a sub-1.0 threshold.
+    with _patch_backend("oracle"):
+        assert _dedup_active(_gate_cfg(0.97)) is False
+
+
+def test_dedup_active_none_config() -> None:
+    assert _dedup_active(None) is False
@@ -0,0 +1,41 @@
+import uuid
+
+import pytest
+
+from hindsight_api.engine.consolidation import consolidator
+
+
+class _ZeroLengthEmbeddings:
+    dimension = 384
+
+    def encode_documents(self, texts):
+        assert texts == ["Consolidated observation text."]
+        return [[]]
+
+
+class _FakeMemoryEngine:
+    embeddings = _ZeroLengthEmbeddings()
+
+
+class _FailingConn:
+    async def fetchrow(self, *args, **kwargs):
+        raise AssertionError("zero-length embedding should be rejected before database insert")
+
+
+@pytest.mark.asyncio
+async def test_create_observation_rejects_zero_length_embedding_before_insert(monkeypatch):
+    source_id = uuid.uuid4()
+
+    async def fake_filter_live_source_memories(conn, bank_id, source_memory_ids):
+        return source_memory_ids
+
+    monkeypatch.setattr(consolidator, "_filter_live_source_memories", fake_filter_live_source_memories)
+
+    with pytest.raises(RuntimeError, match="embedding 0 has dimension 0; expected 384"):
+        await consolidator._create_observation_directly(
+            conn=_FailingConn(),
+            memory_engine=_FakeMemoryEngine(),
+            bank_id="test-bank",
+            source_memory_ids=[source_id],
+            observation_text="Consolidated observation text.",
+        )
@@ -109,7 +109,9 @@ def _mock_llm_one_obs_per_fact():
    def callback(messages, scope):
        if scope != "consolidation":
            return _ConsolidationBatchResponse()
-        prompt = messages[0]["content"] if messages else ""
+        # Facts live in the user message; the system message (stable, cached) carries
+        # example UUIDs in its OUTPUT samples — read user only.
+        prompt = "\n".join(m.get("content", "") for m in messages if m.get("role") == "user")
        fact_ids = re.findall(r"\[([0-9a-f-]{36})\]", prompt)
        creates = [
            _CreateAction(text=f"Observation about fact {fid[:8]}", source_fact_ids=[fid])
@@ -194,9 +194,10 @@ class TestPostgreSQLDialect:
    def test_build_semantic_arm(self, d):
        arm = d.build_semantic_arm(
            table="schema.memory_units", cols="id, text", fact_type="world",
-            embedding_param="$1", bank_id_param="$2", fetch_limit=100,
+            embedding_param="$1", bank_id_param="$2", fetch_limit=100, min_similarity=0.58,
        )
        assert "1 - (embedding <=> $1::vector)" in arm
+        assert ">= 0.58" in arm
        assert "fact_type = 'world'" in arm
        assert "LIMIT 100" in arm
        assert "'semantic' AS source" in arm
@@ -232,16 +233,39 @@ class TestPostgreSQLDialect:
        assert "to_bm25query" in arm
        assert "tokenize" in arm

+    def test_build_bm25_arm_vchord_gates_zero_score_by_default(self, d):
+        """VectorChord ranks every doc, so a score gate must filter non-matches.
+
+        The negated `<&>` score is BM25 (>= 0); the default 0 floor keeps only
+        rows with a genuine query-term match, mirroring native tsvector's `@@`.
+        """
+        arm = d.build_bm25_arm(
+            table="t", cols="id", fact_type="world",
+            bank_id_param="$2", limit_param="$3", text_param="$4",
+            text_search_extension="vchord",
+        )
+        assert "-(search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($4, 'llmlingua2'))) > 0" in arm
+
+    def test_build_bm25_arm_vchord_honors_custom_min_score(self, d):
+        arm = d.build_bm25_arm(
+            table="t", cols="id", fact_type="world",
+            bank_id_param="$2", limit_param="$3", text_param="$4",
+            text_search_extension="vchord",
+            bm25_min_score=2.5,
+        )
+        assert "> 2.5" in arm
+
    def test_build_bm25_arm_pgroonga(self, d):
        arm = d.build_bm25_arm(
            table="schema.memory_units", cols="id, text", fact_type="world",
            bank_id_param="$2", limit_param="$3", text_param="$4",
            text_search_extension="pgroonga",
        )
-        # pgroonga uses the &@~ operator + pgroonga_score for ranking. The
-        # configured bm25_language is intentionally NOT used here — pgroonga's
-        # tokenizer is set at index creation, not query time.
-        assert "&@~ $4" in arm
+        # pgroonga uses the &@~ operator + pgroonga_score for ranking. Escape
+        # the query parameter so literal text containing pgroonga operators is
+        # not parsed as query syntax.
+        assert "&@~ pgroonga_query_escape($4)" in arm
+        assert "&@~ $4" not in arm
        assert "pgroonga_score(tableoid, ctid)" in arm
        assert "to_tsquery" not in arm

@@ -281,8 +305,8 @@ class TestPostgreSQLDialect:
        assert result == "hello world"

    def test_prepare_bm25_text_pgroonga(self, d):
-        # pgroonga accepts raw query text via &@~ and parses it with its own
-        # query syntax; we pass the original query through unchanged.
+        # Keep the user's text unchanged here; the SQL builder escapes the bind
+        # parameter at query time before invoking pgroonga's query parser.
        result = d.prepare_bm25_text(["hello", "world"], "hello world", text_search_extension="pgroonga")
        assert result == "hello world"

@@ -337,9 +361,10 @@ class TestOracleDialect:
    def test_build_semantic_arm(self, d):
        arm = d.build_semantic_arm(
            table="memory_units", cols="id, text", fact_type="world",
-            embedding_param=":1", bank_id_param=":2", fetch_limit=100,
+            embedding_param=":1", bank_id_param=":2", fetch_limit=100, min_similarity=0.58,
        )
        assert "VECTOR_DISTANCE" in arm
+        assert ">= 0.58" in arm
        assert "fact_type = 'world'" in arm
        assert "FETCH FIRST 100 ROWS ONLY" in arm
        assert "'semantic' AS source" in arm
@@ -0,0 +1,915 @@
+"""Tests for document export/import between banks (LLM-free transfer).
+
+These exercise the full export → import round trip on a real (pg0) database with
+the mock LLM fixture, and crucially assert that import does NOT invoke fact
+extraction (the LLM) — it replays the deterministic pipeline and re-embeds.
+"""
+
+import io
+import json
+import uuid
+import zipfile
+from datetime import datetime, timezone
+
+import httpx
+import pytest
+import pytest_asyncio
+
+from hindsight_api.api import create_app
+from hindsight_api.engine.consolidation.consolidator import _create_observation_directly
+from hindsight_api.engine.db_utils import acquire_with_retry
+from hindsight_api.engine.schema import fq_table
+from hindsight_api.engine.transfer import import_documents
+from hindsight_api.engine.transfer.importer import parse_archive
+from hindsight_api.engine.transfer.schema import SCHEMA_VERSION, TransferManifest
+from hindsight_api.extensions import (
+    OperationValidatorExtension,
+    RecallContext,
+    ReflectContext,
+    RetainContext,
+    RetainResult,
+    ValidationResult,
+)
+from hindsight_api.webhooks.manager import WebhookManager
+
+
+class _RetainResultCapture(OperationValidatorExtension):
+    """Records each RetainResult the engine reports via on_retain_complete.
+
+    The pre-operation validators are required by the abstract base; they always
+    accept so they don't interfere with the operations under test.
+    """
+
+    def __init__(self) -> None:
+        self.results: list[RetainResult] = []
+
+    async def validate_retain(self, ctx: RetainContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def validate_recall(self, ctx: RecallContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def validate_reflect(self, ctx: ReflectContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def on_retain_complete(self, result: RetainResult) -> None:
+        self.results.append(result)
+
+
+@pytest_asyncio.fixture
+async def api_client(memory):
+    """Async HTTP client over the FastAPI app backed by the mock-LLM engine."""
+    app = create_app(memory, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        yield client
+
+
+def _unique_bank(prefix: str) -> str:
+    return f"{prefix}_{datetime.now(timezone.utc).timestamp()}"
+
+
+async def _retain(memory, bank_id, content, request_context, document_id):
+    await memory.retain_async(
+        bank_id=bank_id,
+        content=content,
+        context="Test context",
+        document_id=document_id,
+        request_context=request_context,
+    )
+
+
+async def _import(memory, bank_id, archive, request_context, on_conflict="skip"):
+    """Submit an import and return its result_metadata counts.
+
+    Import is async; the test fixture uses SyncTaskBackend so the operation runs
+    inline and is already completed when submit returns.
+    """
+    submission = await memory.import_documents_async(bank_id, archive, request_context, on_conflict)
+    status = await memory.get_operation_status(
+        bank_id, submission["operation_id"], request_context=request_context
+    )
+    assert status["status"] == "completed", status
+    return status["result_metadata"]
+
+
+def test_export_bank_covers_schema():
+    """Every bank-scoped table must be classified by export_bank — logical, carried,
+    history, or explicitly skipped — so a future migration can't silently drop one."""
+    from hindsight_api.admin.cli import BACKUP_TABLES
+    from hindsight_api.engine.transfer.export import (
+        _BANK_ROW_TABLES,
+        _CARRIED_HISTORY_TABLES,
+        _HISTORY_TABLES,
+        _REPLAYED_TABLES,
+        _SKIP_TABLES,
+    )
+
+    buckets = [
+        set(_REPLAYED_TABLES),
+        set(_BANK_ROW_TABLES),
+        set(_CARRIED_HISTORY_TABLES),
+        set(_HISTORY_TABLES),
+        set(_SKIP_TABLES),
+    ]
+    classified = set().union(*buckets)
+    assert classified == set(BACKUP_TABLES), (
+        f"export-bank classification drifted from BACKUP_TABLES: "
+        f"missing={set(BACKUP_TABLES) - classified}, extra={classified - set(BACKUP_TABLES)}"
+    )
+    # No table may appear in two buckets.
+    assert sum(len(b) for b in buckets) == len(classified), "a table is classified in more than one bucket"
+
+
+@pytest.mark.asyncio
+async def test_export_bank_contents(memory, request_context):
+    """export_bank produces a whole-bank archive: docs + bank config + webhooks,
+    no embeddings, with history gated behind include_history."""
+    from hindsight_api.engine.transfer import export_bank
+
+    bank = _unique_bank("export_bank")
+    webhook_id = uuid.uuid4()
+    try:
+        await _retain(memory, bank, "Carol lives in Paris.", request_context, "doc-1")
+        backend = await memory._get_backend()
+        async with acquire_with_retry(backend) as conn:
+            await conn.execute(
+                f"INSERT INTO {fq_table('webhooks')} "
+                f"(id, bank_id, url, secret, event_types, enabled, created_at, updated_at) "
+                f"VALUES ($1, $2, $3, NULL, $4, true, NOW(), NOW())",
+                webhook_id,
+                bank,
+                "https://example.com/hook",
+                ["retain.completed"],
+            )
+
+        # Without history.
+        async with acquire_with_retry(backend) as conn:
+            archive = await export_bank(conn, bank, include_history=False)
+        with zipfile.ZipFile(io.BytesIO(archive)) as zf:
+            names = set(zf.namelist())
+            manifest = TransferManifest.model_validate_json(zf.read("manifest.json"))
+            bank_rows = json.loads(zf.read("banks.json"))
+            webhooks = json.loads(zf.read("webhooks.json"))
+
+        assert manifest.archive_type == "bank"
+        assert manifest.document_count == 1
+        assert manifest.webhook_count == 1
+        assert "mental_models.json" in names and "directives.json" in names
+        assert "mental_model_history.json" in names
+        assert any(d.endswith(".json") and d.startswith("documents/") for d in names)
+        # No history files unless requested.
+        assert not any(n.startswith("history/") for n in names)
+        # The bank row and webhook are carried.
+        assert [r["bank_id"] for r in bank_rows] == [bank]
+        assert webhooks[0]["bank_id"] == bank and webhooks[0]["url"] == "https://example.com/hook"
+        # No embeddings anywhere — the target instance regenerates them.
+        assert "embedding" not in archive.decode("utf-8", errors="ignore")
+
+        # With history.
+        async with acquire_with_retry(backend) as conn:
+            archive_h = await export_bank(conn, bank, include_history=True)
+        with zipfile.ZipFile(io.BytesIO(archive_h)) as zf:
+            names_h = set(zf.namelist())
+            manifest_h = TransferManifest.model_validate_json(zf.read("manifest.json"))
+        assert manifest_h.includes_history is True
+        assert "history/audit_log.json" in names_h and "history/llm_requests.json" in names_h
+    finally:
+        await memory.delete_bank(bank, request_context=request_context)
+
+
+def _as_json(value):
+    """Normalize a jsonb column value (str or already-decoded) to a Python object."""
+    return json.loads(value) if isinstance(value, str) else value
+
+
+async def _bank_content_snapshot(memory, bank_id):
+    """Capture the meaningful (non-embedding, non-volatile) content of a bank for
+    exact round-trip comparison across export → import."""
+    backend = await memory._get_backend()
+    async with acquire_with_retry(backend) as conn:
+        bank = await conn.fetchrow(
+            f"SELECT name, disposition, mission, config FROM {fq_table('banks')} WHERE bank_id = $1", bank_id
+        )
+        docs = await conn.fetch(
+            f"SELECT id, original_text, tags FROM {fq_table('documents')} WHERE bank_id = $1", bank_id
+        )
+        facts = await conn.fetch(
+            f"SELECT text, fact_type, context FROM {fq_table('memory_units')} "
+            f"WHERE bank_id = $1 AND fact_type != 'observation'",
+            bank_id,
+        )
+        obs = await conn.fetch(
+            f"SELECT text, proof_count FROM {fq_table('memory_units')} WHERE bank_id = $1 AND fact_type = 'observation'",
+            bank_id,
+        )
+        ents = await conn.fetch(f"SELECT canonical_name FROM {fq_table('entities')} WHERE bank_id = $1", bank_id)
+        links = await conn.fetch(
+            f"SELECT link_type, count(*) AS c FROM {fq_table('memory_links')} WHERE bank_id = $1 GROUP BY link_type",
+            bank_id,
+        )
+        hooks = await conn.fetch(
+            f"SELECT url, event_types, enabled FROM {fq_table('webhooks')} WHERE bank_id = $1", bank_id
+        )
+        dirs = await conn.fetch(
+            f"SELECT name, content, priority, is_active FROM {fq_table('directives')} WHERE bank_id = $1", bank_id
+        )
+        mms = await conn.fetch(
+            f"SELECT subtype, name, description, tags FROM {fq_table('mental_models')} WHERE bank_id = $1", bank_id
+        )
+        null_emb = await conn.fetchval(
+            f"SELECT count(*) FROM {fq_table('memory_units')} "
+            f"WHERE bank_id = $1 AND fact_type != 'observation' AND embedding IS NULL",
+            bank_id,
+        )
+    return {
+        "bank": (bank["name"], _as_json(bank["disposition"]), bank["mission"], _as_json(bank["config"])),
+        "documents": sorted((d["id"], d["original_text"], tuple(sorted(d["tags"] or []))) for d in docs),
+        "facts": sorted((f["text"], f["fact_type"], f["context"]) for f in facts),
+        "observations": sorted((o["text"], o["proof_count"]) for o in obs),
+        "entities": sorted(e["canonical_name"].lower() for e in ents),
+        "links": {row["link_type"]: row["c"] for row in links},
+        "webhooks": sorted((h["url"], tuple(h["event_types"] or []), h["enabled"]) for h in hooks),
+        "directives": sorted((d["name"], d["content"], d["priority"], d["is_active"]) for d in dirs),
+        "mental_models": sorted(
+            (m["subtype"], m["name"], m["description"], tuple(sorted(m["tags"] or []))) for m in mms
+        ),
+        "null_embeddings": null_emb,
+    }
+
+
+@pytest.mark.asyncio
+async def test_bank_export_import_exact_roundtrip(memory, request_context):
+    """A whole-bank archive restores EXACT bank content (config, docs, facts,
+    observations, entities, links, webhooks, directives, mental models) with facts
+    re-embedded. Uses export → delete → import so ids round-trip without collisions
+    (mirroring a fresh target instance)."""
+    bank = _unique_bank("bank_exact")
+    try:
+        await _retain(memory, bank, "Alice works at Google. Bob works at Microsoft.", request_context, "doc-1")
+        await _retain(memory, bank, "Carol lives in Paris.", request_context, "doc-2")
+
+        backend = await memory._get_backend()
+        async with acquire_with_retry(backend) as conn:
+            await conn.execute(
+                f"UPDATE {fq_table('banks')} SET name = $2, disposition = $3::jsonb, "
+                f"mission = $4, config = $5::jsonb WHERE bank_id = $1",
+                bank,
+                "My Bank",
+                json.dumps({"skepticism": 5, "literalism": 2, "empathy": 4}),
+                "Be terse and precise.",
+                json.dumps({"reflect_mission": "be terse"}),
+            )
+            await conn.execute(
+                f"INSERT INTO {fq_table('webhooks')} "
+                f"(id, bank_id, url, secret, event_types, enabled, created_at, updated_at) "
+                f"VALUES ($1, $2, $3, NULL, $4, true, NOW(), NOW())",
+                uuid.uuid4(),
+                bank,
+                "https://example.com/hook",
+                ["retain.completed", "consolidation.completed"],
+            )
+            await conn.execute(
+                f"INSERT INTO {fq_table('directives')} "
+                f"(id, bank_id, name, content, priority, is_active, tags, created_at, updated_at) "
+                f"VALUES ($1, $2, $3, $4, $5, true, $6, NOW(), NOW())",
+                uuid.uuid4(),
+                bank,
+                "tone",
+                "Always be concise.",
+                7,
+                ["style"],
+            )
+        await memory.create_mental_model(
+            bank,
+            name="Work model",
+            source_query="where do people work",
+            content="User tracks where people work.",
+            mental_model_id="mm-1",
+            tags=["people"],
+            request_context=request_context,
+        )
+
+        before = await _bank_content_snapshot(memory, bank)
+        # Sanity: the source genuinely has rich content in every section we carry.
+        assert before["facts"] and before["entities"] and before["links"]
+        assert before["webhooks"] and before["directives"] and before["mental_models"]
+        assert before["bank"][0] == "My Bank"
+
+        from hindsight_api.engine.transfer import export_bank
+
+        async with acquire_with_retry(backend) as conn:
+            archive = await export_bank(conn, bank)
+        # Delete then restore into the same id — exact round-trip, no PK collisions.
+        await memory.delete_bank(bank, request_context=request_context)
+        result = await memory.import_bank_async(archive, request_context)
+        assert result.bank_id == bank
+        assert result.webhooks_imported == 1
+        assert result.directives_imported == 1
+        assert result.mental_models_imported == 1
+
+        after = await _bank_content_snapshot(memory, bank)
+        # Semantic links are an ANN-approximate retrieval index regenerated from the
+        # (re-embedded) facts; their count depends on whether ANN runs incrementally
+        # per document (import) or as a final whole-bank pass (original retain), so
+        # compare them loosely. Everything else — source data and deterministic
+        # temporal links — must match exactly.
+        after_semantic = after["links"].pop("semantic", 0)
+        before["links"].pop("semantic", None)
+        assert after == before
+        assert after_semantic > 0, "semantic links should be regenerated on import"
+        # Facts were re-embedded on import (no NULL vectors).
+        assert after["null_embeddings"] == 0
+    finally:
+        await memory.delete_bank(bank, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_bank_roundtrip_carries_mental_model_history(memory, request_context):
+    """Mental-model refresh history survives export/import. Mental models keep a
+    stable (id, bank_id), so the dedicated mental_model_history rows are carried
+    (the surrogate id is dropped on export; the target reassigns it)."""
+    bank = _unique_bank("bank_mm_hist")
+    try:
+        await memory.get_bank_profile(bank, request_context=request_context)
+        await memory.create_mental_model(
+            bank,
+            name="Work model",
+            source_query="where do people work",
+            content="v1",
+            mental_model_id="mm-1",
+            request_context=request_context,
+        )
+        await memory.update_mental_model(
+            bank, mental_model_id="mm-1", content="v2", request_context=request_context
+        )
+        await memory.update_mental_model(
+            bank, mental_model_id="mm-1", content="v3", request_context=request_context
+        )
+        # Two refreshes → two snapshots (previous content v1 then v2), newest-first.
+        before = await memory.get_mental_model_history(bank, "mm-1", request_context=request_context)
+        assert [h["previous_content"] for h in before] == ["v2", "v1"]
+
+        from hindsight_api.engine.transfer import export_bank
+
+        backend = await memory._get_backend()
+        async with acquire_with_retry(backend) as conn:
+            archive = await export_bank(conn, bank)
+        await memory.delete_bank(bank, request_context=request_context)
+        result = await memory.import_bank_async(archive, request_context)
+        assert result.mental_model_history_imported == 2
+
+        after = await memory.get_mental_model_history(bank, "mm-1", request_context=request_context)
+        assert [h["previous_content"] for h in after] == ["v2", "v1"]
+    finally:
+        await memory.delete_bank(bank, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_bank_rejects_documents_archive(memory, request_context):
+    """A documents-only archive must be rejected by the bank importer."""
+    bank = _unique_bank("bank_reject")
+    try:
+        await _retain(memory, bank, "Alice works at Google.", request_context, "doc-1")
+        docs_archive = await memory.export_documents_async(bank, request_context)
+        with pytest.raises(ValueError, match="whole-bank archive"):
+            await memory.import_bank_async(docs_archive, request_context)
+    finally:
+        await memory.delete_bank(bank, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_bank_refuses_existing_bank(memory, request_context):
+    """import-bank restores a whole bank, not a merge — it must refuse an existing target."""
+    from hindsight_api.engine.transfer import export_bank
+
+    bank = _unique_bank("bank_exists")
+    try:
+        await _retain(memory, bank, "Alice works at Google.", request_context, "doc-1")
+        backend = await memory._get_backend()
+        async with acquire_with_retry(backend) as conn:
+            archive = await export_bank(conn, bank)
+        # The source bank still exists — importing the archive back must refuse
+        # (restoring into the same id after delete is covered by the exact round-trip test).
+        with pytest.raises(ValueError, match="already exists"):
+            await memory.import_bank_async(archive, request_context)
+    finally:
+        await memory.delete_bank(bank, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_export_import_roundtrip_without_llm(memory, request_context, monkeypatch):
+    """Export from one bank and import into another without re-running the LLM."""
+    src = _unique_bank("transfer_src")
+    dst = _unique_bank("transfer_dst")
+    try:
+        await _retain(
+            memory,
+            src,
+            "Alice works at Google. Bob works at Microsoft.",
+            request_context,
+            document_id="doc-1",
+        )
+
+        archive = await memory.export_documents_async(src, request_context)
+        assert isinstance(archive, bytes) and len(archive) > 0
+
+        parsed = parse_archive(archive)
+        assert parsed.manifest.source_bank_id == src
+        assert parsed.manifest.document_count == 1
+        assert parsed.manifest.fact_count > 0
+        # The archive must not carry embeddings or raw db ids (no "embedding" anywhere,
+        # now that the manifest no longer includes embedding model/dimension metadata).
+        assert "embedding" not in archive.decode("utf-8", errors="ignore")
+
+        exported_texts = {fact.text for doc in parsed.documents for fact in doc.facts}
+        assert exported_texts
+
+        # Importing must never call the LLM fact extractor — make it explode if it does.
+        def _boom(*args, **kwargs):
+            raise AssertionError("import must not invoke LLM fact extraction")
+
+        monkeypatch.setattr(
+            "hindsight_api.engine.retain.fact_extraction.extract_facts_from_contents",
+            _boom,
+        )
+
+        result = await _import(memory, dst, archive, request_context)
+        assert result["documents_imported"] == 1
+        assert result["documents_skipped"] == 0
+        assert result["facts_imported"] == parsed.manifest.fact_count
+
+        # Facts landed in the destination bank with matching text. Import triggers
+        # consolidation, which may synthesize observation units in the destination,
+        # so filter those out — the imported facts are world/experience only.
+        units = await memory.list_memory_units(dst, request_context=request_context)
+        imported_units = [item for item in units["items"] if item["fact_type"] != "observation"]
+        assert len(imported_units) == result["facts_imported"]
+        assert {item["text"] for item in imported_units} == exported_texts
+
+        # Entities were re-resolved in the destination bank.
+        entities = await memory.list_entities(dst, request_context=request_context)
+        entity_names = {e["canonical_name"].lower() for e in entities["items"]}
+        assert any("alice" in n for n in entity_names)
+        assert any("bob" in n for n in entity_names)
+
+        # Embeddings were regenerated locally (not null) in the destination.
+        backend = await memory._get_backend()
+        async with acquire_with_retry(backend) as conn:
+            null_embeddings = await conn.fetchval(
+                f"SELECT COUNT(*) FROM {fq_table('memory_units')} WHERE bank_id = $1 AND embedding IS NULL",
+                dst,
+            )
+        assert null_embeddings == 0
+
+        # And the imported memories are retrievable.
+        recall = await memory.recall_async(bank_id=dst, query="Where does Alice work?", request_context=request_context)
+        assert recall is not None
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+async def _bank_snapshot(memory, bank_id):
+    """Count everything persisted for a bank, for round-trip integrity comparison."""
+    backend = await memory._get_backend()
+    async with acquire_with_retry(backend) as conn:
+        docs = await conn.fetch(
+            f"SELECT id, COALESCE(length(original_text), 0) AS len FROM {fq_table('documents')} "
+            f"WHERE bank_id = $1",
+            bank_id,
+        )
+        chunks = await conn.fetch(
+            f"SELECT document_id, chunk_index, length(chunk_text) AS len FROM {fq_table('chunks')} "
+            f"WHERE bank_id = $1",
+            bank_id,
+        )
+        ftypes = await conn.fetch(
+            f"SELECT fact_type, count(*) AS c FROM {fq_table('memory_units')} WHERE bank_id = $1 GROUP BY fact_type",
+            bank_id,
+        )
+        links = await conn.fetch(
+            f"SELECT ml.link_type, count(*) AS c FROM {fq_table('memory_links')} ml "
+            f"JOIN {fq_table('memory_units')} m ON m.id = ml.from_unit_id "
+            f"WHERE m.bank_id = $1 GROUP BY ml.link_type",
+            bank_id,
+        )
+        unit_entities = await conn.fetchval(
+            f"SELECT count(*) FROM {fq_table('unit_entities')} ue "
+            f"JOIN {fq_table('memory_units')} m ON m.id = ue.unit_id WHERE m.bank_id = $1",
+            bank_id,
+        )
+        entities = await conn.fetchval(f"SELECT count(*) FROM {fq_table('entities')} WHERE bank_id = $1", bank_id)
+        facts_with_chunk = await conn.fetchval(
+            f"SELECT count(*) FROM {fq_table('memory_units')} WHERE bank_id = $1 AND chunk_id IS NOT NULL",
+            bank_id,
+        )
+    by_type = {r["fact_type"]: r["c"] for r in ftypes}
+    return {
+        "doc_count": len(docs),
+        "doc_lens": {r["id"]: r["len"] for r in docs},
+        "chunk_count": len(chunks),
+        # (document_id, chunk_index) -> chunk_text length: verifies attribution AND size.
+        "chunk_map": {(r["document_id"], r["chunk_index"]): r["len"] for r in chunks},
+        "world": by_type.get("world", 0),
+        "experience": by_type.get("experience", 0),
+        "observation": by_type.get("observation", 0),
+        "unit_entities": unit_entities,
+        "entities": entities,
+        "facts_with_chunk": facts_with_chunk,
+        "links_by_type": {r["link_type"]: r["c"] for r in links},
+        "links_total": sum(r["c"] for r in links),
+    }
+
+
+@pytest.mark.asyncio
+async def test_full_roundtrip_integrity(memory, request_context):
+    """Full export → import must reproduce every persisted artifact (counts + sizes)."""
+    src = _unique_bank("transfer_integ_src")
+    dst = _unique_bank("transfer_integ_dst")
+    try:
+        # A multi-chunk document (content > chunk_size) plus a short one, so chunk
+        # numbering and fact→chunk attribution across chunks are exercised.
+        long_doc = " ".join(f"Person{i} works at Company{i} in City{i}." for i in range(220))
+        await _retain(memory, src, long_doc, request_context, "doc-long")
+        await _retain(memory, src, "Carol moved to Berlin in 2024 and joined Acme.", request_context, "doc-short")
+
+        before = await _bank_snapshot(memory, src)
+        # Sanity: the fixture actually produced multiple chunks + links + observations.
+        assert before["chunk_count"] >= 2
+        assert before["links_total"] > 0
+        assert before["observation"] > 0
+
+        archive = await memory.export_documents_async(src, request_context, include_observations=True)
+        await _import(memory, dst, archive, request_context)
+        after = await _bank_snapshot(memory, dst)
+
+        # Documents: same count and same original_text sizes (by id).
+        assert after["doc_count"] == before["doc_count"]
+        assert after["doc_lens"] == before["doc_lens"]
+        # Chunks: same count, and same (document, chunk_index) -> size map. This is
+        # the chunk-attribution guarantee.
+        assert after["chunk_count"] == before["chunk_count"]
+        assert after["chunk_map"] == before["chunk_map"]
+        # Facts: same world/experience/observation counts, same chunk linkage count.
+        assert after["world"] == before["world"]
+        assert after["experience"] == before["experience"]
+        assert after["observation"] == before["observation"]
+        assert after["facts_with_chunk"] == before["facts_with_chunk"]
+        # Entities + entity links re-resolved to the same counts.
+        assert after["entities"] == before["entities"]
+        assert after["unit_entities"] == before["unit_entities"]
+        # Links are regenerated against the target bank; for the same facts/embeddings
+        # the deterministic temporal + causal links must match exactly.
+        for link_type in ("temporal", "caused_by"):
+            assert after["links_by_type"].get(link_type, 0) == before["links_by_type"].get(link_type, 0), (
+                link_type,
+                before["links_by_type"],
+                after["links_by_type"],
+            )
+        # And links overall must be present (semantic counts can vary slightly with
+        # ANN ordering, so we don't assert exact equality on the total).
+        assert after["links_total"] > 0
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_export_import_observations(memory, request_context):
+    """With include_observations, observations transfer and their sources re-link."""
+    src = _unique_bank("transfer_obs_src")
+    dst = _unique_bank("transfer_obs_dst")
+    try:
+        await _retain(memory, src, "Alice works at Google. Bob works at Microsoft.", request_context, "doc-1")
+        # Sources must be world/experience facts (not auto-consolidation observations).
+        units = await memory.list_memory_units(src, fact_type="world", request_context=request_context)
+        source_ids = [uuid.UUID(str(i["id"])) for i in units["items"][:2]]
+        assert len(source_ids) == 2
+
+        # Create a real observation over those source facts.
+        backend = await memory._get_backend()
+        async with acquire_with_retry(backend) as conn:
+            async with conn.transaction():
+                await _create_observation_directly(
+                    conn, memory, src, source_ids, "Alice and Bob are colleagues."
+                )
+
+        # Export WITHOUT observations -> none in the archive (the bank may also
+        # contain auto-consolidation observations; the flag is what gates them).
+        plain = parse_archive(await memory.export_documents_async(src, request_context))
+        assert plain.manifest.observation_count == 0
+        assert plain.observations == []
+
+        # Export WITH observations. (The mock LLM's auto-consolidation may have
+        # produced extra observations too, so assert on our specific one.)
+        archive = await memory.export_documents_async(src, request_context, include_observations=True)
+        parsed = parse_archive(archive)
+        assert parsed.manifest.observation_count == len(parsed.observations) >= 1
+        mine = next((o for o in parsed.observations if o.text == "Alice and Bob are colleagues."), None)
+        assert mine is not None
+        assert len(mine.sources) == 2  # both sources resolved within the export
+        assert "embedding" not in archive.decode("utf-8", errors="ignore")
+
+        # Import into a fresh bank. Every exported observation's sources are in
+        # the single exported document, so all import and none are skipped.
+        result = await _import(memory, dst, archive, request_context)
+        assert result["observations_imported"] == parsed.manifest.observation_count
+        assert result["observations_skipped"] == 0
+
+        # Our observation landed with source_memory_ids pointing at dst's facts,
+        # and those source facts are marked consolidated.
+        async with acquire_with_retry(backend) as conn:
+            obs_row = await conn.fetchrow(
+                f"SELECT source_memory_ids FROM {fq_table('memory_units')} "
+                f"WHERE bank_id = $1 AND fact_type = 'observation' AND text = $2",
+                dst,
+                "Alice and Bob are colleagues.",
+            )
+            assert obs_row is not None
+            dst_sources = list(obs_row["source_memory_ids"] or [])
+            assert len(dst_sources) == 2
+            consolidated = await conn.fetchval(
+                f"SELECT COUNT(*) FROM {fq_table('memory_units')} "
+                f"WHERE bank_id = $1 AND id = ANY($2) AND consolidated_at IS NOT NULL",
+                dst,
+                dst_sources,
+            )
+            assert consolidated == 2
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_triggers_consolidation(memory, request_context):
+    """Importing (without observations) triggers consolidation in the target bank,
+    so observations get generated there — same as a normal retain."""
+    src = _unique_bank("transfer_consol_src")
+    dst = _unique_bank("transfer_consol_dst")
+    try:
+        await _retain(memory, src, "Alice works at Google. Bob works at Microsoft.", request_context, "doc-1")
+        # Export WITHOUT observations: the archive carries only world/experience facts.
+        archive = await memory.export_documents_async(src, request_context)
+        assert parse_archive(archive).observations == []
+
+        # Import into a fresh bank. The post-import consolidation trigger runs
+        # inline (SyncTaskBackend) and the mock LLM produces observations.
+        await _import(memory, dst, archive, request_context)
+
+        obs = await memory.list_memory_units(dst, fact_type="observation", request_context=request_context)
+        assert obs["total"] > 0, "import should have triggered consolidation to generate observations"
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_fires_retain_complete_hook(memory, request_context):
+    """Import fires the post-retain extension hook once per imported document,
+    mirroring retain — with zero LLM tokens (import runs no extraction)."""
+    src = _unique_bank("transfer_hook_src")
+    dst = _unique_bank("transfer_hook_dst")
+    await _retain(memory, src, "Alice works at Google.", request_context, "doc-1")
+    await _retain(memory, src, "Bob works at Microsoft.", request_context, "doc-2")
+    archive = await memory.export_documents_async(src, request_context)
+
+    capture = _RetainResultCapture()
+    original_validator = memory._operation_validator
+    memory._operation_validator = capture
+    try:
+        result = await _import(memory, dst, archive, request_context)
+        assert result["documents_imported"] == 2
+
+        # One hook call per imported document.
+        assert len(capture.results) == 2
+        by_doc = {r.document_id: r for r in capture.results}
+        assert set(by_doc) == {"doc-1", "doc-2"}
+        for res in capture.results:
+            assert res.bank_id == dst
+            assert res.success is True
+            # Import runs no LLM extraction: token counts are zero and
+            # processed_content_tokens is 0 ("nothing went through extraction").
+            assert res.llm_input_tokens == 0
+            assert res.llm_output_tokens == 0
+            assert res.llm_total_tokens == 0
+            assert res.processed_content_tokens == 0
+            # unit_ids are reported per content item, with the created facts.
+            assert res.unit_ids and res.unit_ids[0]
+    finally:
+        memory._operation_validator = original_validator
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_queues_retain_webhook(memory, request_context):
+    """Import queues a retain.completed webhook delivery per document, like retain."""
+    src = _unique_bank("transfer_wh_src")
+    dst = _unique_bank("transfer_wh_dst")
+    webhook_id = uuid.uuid4()
+    await _retain(memory, src, "Carol lives in Paris.", request_context, "doc-wh")
+    archive = await memory.export_documents_async(src, request_context)
+
+    # The destination bank is created lazily by import; create it now so the
+    # webhook row's FK to banks is satisfied, then subscribe it to retain.completed.
+    backend = await memory._get_backend()
+    async with acquire_with_retry(backend) as conn:
+        await conn.execute(
+            f"INSERT INTO {fq_table('banks')} (bank_id, name) VALUES ($1, $2) ON CONFLICT DO NOTHING",
+            dst,
+            dst,
+        )
+        await conn.execute(
+            f"INSERT INTO {fq_table('webhooks')} "
+            f"(id, bank_id, url, secret, event_types, enabled, created_at, updated_at) "
+            f"VALUES ($1, $2, $3, NULL, $4, true, NOW(), NOW())",
+            webhook_id,
+            dst,
+            "https://example.com/retain-hook",
+            ["retain.completed"],
+        )
+
+    original_manager = memory._webhook_manager
+    memory._webhook_manager = WebhookManager(backend=memory._backend, global_webhooks=[])
+    try:
+        await _import(memory, dst, archive, request_context)
+
+        async with acquire_with_retry(backend) as conn:
+            rows = await conn.fetch(
+                f"SELECT task_payload FROM {fq_table('async_operations')} "
+                f"WHERE operation_type = 'webhook_delivery' AND bank_id = $1 "
+                f"AND task_payload->>'event_type' = 'retain.completed'",
+                dst,
+            )
+        assert len(rows) == 1, "import should queue one retain.completed delivery for the imported document"
+        payload = rows[0]["task_payload"]
+        if isinstance(payload, str):
+            payload = json.loads(payload)
+        inner = json.loads(payload["payload"])
+        assert inner.get("data", {}).get("document_id") == "doc-wh"
+    finally:
+        memory._webhook_manager = original_manager
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_include_observations_requires_whole_bank_export(memory, request_context):
+    """include_observations is only valid for a whole-bank export, not a subset."""
+    src = _unique_bank("transfer_obs_subset")
+    try:
+        await _retain(memory, src, "Alice works at Google.", request_context, "doc-1")
+        # Subset export (document_ids set) + observations must be rejected.
+        with pytest.raises(ValueError, match="whole bank"):
+            await memory.export_documents_async(
+                src, request_context, ["doc-1"], include_observations=True
+            )
+        # Whole-bank export with observations is fine; subset without observations is fine.
+        await memory.export_documents_async(src, request_context, include_observations=True)
+        await memory.export_documents_async(src, request_context, ["doc-1"])
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_on_conflict_modes(memory, request_context):
+    """skip leaves the document untouched; replace re-imports; new-id duplicates under a fresh id."""
+    src = _unique_bank("transfer_conf")
+    try:
+        await _retain(memory, src, "Carol lives in Paris.", request_context, document_id="doc-x")
+        archive = await memory.export_documents_async(src, request_context)
+
+        # Re-importing into the SAME bank with skip is a no-op.
+        skipped = await _import(memory, src, archive, request_context, on_conflict="skip")
+        assert skipped["documents_imported"] == 0
+        assert skipped["documents_skipped"] == 1
+        assert skipped["skipped_document_ids"] == ["doc-x"]
+
+        docs_after_skip = await memory.list_documents(src, request_context=request_context)
+        assert docs_after_skip["total"] == 1
+
+        # replace re-imports under the same id.
+        replaced = await _import(memory, src, archive, request_context, on_conflict="replace")
+        assert replaced["documents_imported"] == 1
+        assert replaced["documents_skipped"] == 0
+        docs_after_replace = await memory.list_documents(src, request_context=request_context)
+        assert docs_after_replace["total"] == 1
+
+        # new-id imports a copy under a freshly generated id.
+        remapped = await _import(memory, src, archive, request_context, on_conflict="new-id")
+        assert remapped["documents_imported"] == 1
+        assert "doc-x" in remapped["remapped_document_ids"]
+        docs_after_newid = await memory.list_documents(src, request_context=request_context)
+        assert docs_after_newid["total"] == 2
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_http_export_import_endpoints(api_client, memory, request_context):
+    """Round trip through the HTTP export (GET) and import (POST multipart) endpoints."""
+    src = _unique_bank("transfer_http_src")
+    dst = _unique_bank("transfer_http_dst")
+    try:
+        await _retain(memory, src, "Dana lives in Berlin.", request_context, document_id="doc-http")
+
+        export = await api_client.get(f"/v1/default/banks/{src}/document-transfer")
+        assert export.status_code == 200
+        assert export.headers["content-type"] == "application/zip"
+        archive = export.content
+        assert len(archive) > 0
+
+        # include_observations + a document_id subset is a 400.
+        bad = await api_client.get(
+            f"/v1/default/banks/{src}/document-transfer",
+            params={"document_id": "meeting-notes", "include_observations": "true"},
+        )
+        assert bad.status_code == 400
+
+        # Import is async: returns 202 + operation_id (runs inline under the
+        # SyncTaskBackend test fixture, so it's completed by the time we poll).
+        imported = await api_client.post(
+            f"/v1/default/banks/{dst}/document-transfer",
+            files={"file": ("transfer.zip", archive, "application/zip")},
+            params={"on_conflict": "skip"},
+        )
+        assert imported.status_code == 202
+        operation_id = imported.json()["operation_id"]
+
+        status = await api_client.get(f"/v1/default/banks/{dst}/operations/{operation_id}")
+        assert status.status_code == 200
+        op = status.json()
+        assert op["status"] == "completed"
+        assert op["result_metadata"]["documents_imported"] == 1
+        assert op["result_metadata"]["facts_imported"] >= 1
+
+        # Exporting a bank that does not exist is a 404.
+        missing = await api_client.get("/v1/default/banks/does-not-exist-bank/document-transfer")
+        assert missing.status_code == 404
+    finally:
+        await memory.delete_bank(src, request_context=request_context)
+        await memory.delete_bank(dst, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_endpoints_disabled_by_config(api_client, monkeypatch):
+    """When the feature flags are off, the endpoints return 404 and /version reports disabled."""
+    from hindsight_api.config import clear_config_cache
+
+    # The static config is a cached singleton; override via env + cache reset.
+    monkeypatch.setenv("HINDSIGHT_API_ENABLE_DOCUMENT_EXPORT_API", "false")
+    monkeypatch.setenv("HINDSIGHT_API_ENABLE_DOCUMENT_IMPORT_API", "false")
+    clear_config_cache()
+    try:
+        export = await api_client.get("/v1/default/banks/any-bank/document-transfer")
+        assert export.status_code == 404
+        assert "disabled" in export.json()["detail"].lower()
+
+        imported = await api_client.post(
+            "/v1/default/banks/any-bank/document-transfer",
+            files={"file": ("x.zip", b"not-a-zip", "application/zip")},
+        )
+        assert imported.status_code == 404
+        assert "disabled" in imported.json()["detail"].lower()
+
+        version = await api_client.get("/version")
+        features = version.json()["features"]
+        assert features["document_export_api"] is False
+        assert features["document_import_api"] is False
+    finally:
+        # Restore the cache so the reverted env is picked up by later tests.
+        clear_config_cache()
+
+
+@pytest.mark.asyncio
+async def test_import_rejects_unsupported_schema_version(memory, request_context):
+    """An archive with an unknown schema version is rejected before any writes."""
+    manifest = TransferManifest(schema_version=SCHEMA_VERSION + 999, source_bank_id="whatever")
+    buffer = io.BytesIO()
+    with zipfile.ZipFile(buffer, "w") as zf:
+        zf.writestr("manifest.json", manifest.model_dump_json())
+
+    with pytest.raises(ValueError, match="schema version"):
+        await memory.import_documents_async("any-bank", buffer.getvalue(), request_context)
+
+
+@pytest.mark.asyncio
+async def test_import_rejects_invalid_on_conflict(memory, request_context):
+    """An unknown on_conflict mode is rejected with a ValueError."""
+    manifest = TransferManifest(source_bank_id="whatever")
+    buffer = io.BytesIO()
+    with zipfile.ZipFile(buffer, "w") as zf:
+        zf.writestr("manifest.json", manifest.model_dump_json())
+
+    with pytest.raises(ValueError, match="on_conflict"):
+        await import_documents(
+            backend=await memory._get_backend(),
+            embeddings_model=memory.embeddings,
+            entity_resolver=memory.entity_resolver,
+            config=None,
+            format_date_fn=memory._format_readable_date,
+            bank_id="any-bank",
+            archive_bytes=buffer.getvalue(),
+            on_conflict="bogus",
+        )
@@ -635,6 +635,30 @@ class TestMemoryEngineTenantAuth:
                request_context=None,
            )

+    @pytest.mark.asyncio
+    async def test_list_audit_logs_fails_with_invalid_api_key(self, memory_with_tenant):
+        """Audit log listing goes through tenant auth like other ops."""
+        memory = memory_with_tenant
+
+        with pytest.raises(AuthenticationError) as exc_info:
+            await memory.list_audit_logs(
+                "test-bank",
+                request_context=RequestContext(api_key="wrong-key"),
+            )
+
+        assert "Invalid API key" in str(exc_info.value)
+
+    @pytest.mark.asyncio
+    async def test_audit_log_stats_fails_with_invalid_api_key(self, memory_with_tenant):
+        """Audit log stats goes through tenant auth like other ops."""
+        memory = memory_with_tenant
+
+        with pytest.raises(AuthenticationError):
+            await memory.audit_log_stats(
+                "test-bank",
+                request_context=RequestContext(api_key="wrong-key"),
+            )
+
    @pytest.mark.asyncio
    async def test_no_tenant_request_needed_without_extension(self, memory):
        """Operations work with empty RequestContext when no tenant extension configured."""
@@ -0,0 +1,42 @@
+"""Tests for per-source candidate capping before RRF fusion."""
+
+from hindsight_api.engine.search.fusion import cap_per_source
+from hindsight_api.engine.search.types import RetrievalResult
+
+
+def _results(n: int) -> list[RetrievalResult]:
+    return [RetrievalResult(id=str(i), text=f"r{i}", fact_type="world") for i in range(n)]
+
+
+def test_cap_truncates_to_top_n():
+    results = _results(10)
+    capped = cap_per_source(results, 3)
+    assert [r.id for r in capped] == ["0", "1", "2"]
+
+
+def test_cap_preserves_order():
+    """Capping must keep the caller's best-first ordering (it only slices)."""
+    results = _results(5)
+    capped = cap_per_source(results, 2)
+    assert capped == results[:2]
+
+
+def test_cap_zero_disables():
+    results = _results(5)
+    # 0 means "unlimited" — return the list untouched (same object, no copy).
+    assert cap_per_source(results, 0) is results
+
+
+def test_cap_negative_disables():
+    results = _results(5)
+    assert cap_per_source(results, -1) is results
+
+
+def test_cap_at_or_above_length_is_noop():
+    results = _results(4)
+    assert cap_per_source(results, 4) is results
+    assert cap_per_source(results, 10) is results
+
+
+def test_cap_empty_list():
+    assert cap_per_source([], 5) == []
@@ -0,0 +1,469 @@
+"""Unit tests for GeminiCacheManager.
+
+The SDK's caches.create call is replaced with a fake throughout — no
+network, no real Gemini calls. We assert:
+
+* Identical prefixes return identical fingerprints (cache hits).
+* Different prefixes return different fingerprints.
+* The first get_or_create for a fingerprint creates; the second within
+  the TTL window reuses without calling the SDK again.
+* "minimum token count" errors from Gemini surface as ``None`` (soft
+  fallback), not exceptions.
+* Other SDK errors also surface as ``None`` so callers don't crash on
+  transient creation failures.
+* The TTL refresh boundary recreates after the safety margin elapses.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+from hindsight_api.engine.providers.gemini_cache import GeminiCacheManager
+
+
+def _make_client(create_side_effect=None):
+    """Build a fake Gemini client whose ``aio.caches.create`` returns
+    a SimpleNamespace with ``.name`` (or raises the given exception)."""
+    create_mock = AsyncMock()
+    if isinstance(create_side_effect, Exception):
+        create_mock.side_effect = create_side_effect
+    elif callable(create_side_effect):
+        create_mock.side_effect = create_side_effect
+    else:
+        create_mock.return_value = SimpleNamespace(
+            name="cachedContents/test-cache-name-001"
+        )
+
+    client = MagicMock()
+    client.aio = MagicMock()
+    client.aio.caches = MagicMock()
+    client.aio.caches.create = create_mock
+    return client, create_mock
+
+
+# ---- Fingerprint properties ----------------------------------------------
+
+
+def test_fingerprint_is_stable():
+    fp1 = GeminiCacheManager.fingerprint(
+        model="gemini-3.1-flash-lite",
+        system_instruction="Extract facts.",
+        response_schema=None,
+    )
+    fp2 = GeminiCacheManager.fingerprint(
+        model="gemini-3.1-flash-lite",
+        system_instruction="Extract facts.",
+        response_schema=None,
+    )
+    assert fp1 == fp2
+
+
+def test_fingerprint_changes_with_model():
+    fp1 = GeminiCacheManager.fingerprint("gemini-3.1-flash-lite", "X", None)
+    fp2 = GeminiCacheManager.fingerprint("gemini-3.1-flash", "X", None)
+    assert fp1 != fp2
+
+
+def test_fingerprint_changes_with_system_instruction():
+    fp1 = GeminiCacheManager.fingerprint("m", "Extract facts.", None)
+    fp2 = GeminiCacheManager.fingerprint("m", "Extract entities.", None)
+    assert fp1 != fp2
+
+
+def test_fingerprint_handles_pydantic_schema():
+    """Two equivalent Pydantic schemas should fingerprint identically;
+    a different shape should not."""
+    from pydantic import BaseModel
+
+    class A(BaseModel):
+        x: int
+        y: str
+
+    class A_dup(BaseModel):
+        x: int
+        y: str
+
+    class B(BaseModel):
+        x: int
+        y: int  # different type
+
+    fp_a = GeminiCacheManager.fingerprint("m", "p", A)
+    fp_dup = GeminiCacheManager.fingerprint("m", "p", A_dup)
+    fp_b = GeminiCacheManager.fingerprint("m", "p", B)
+
+    # A and A_dup have the same JSON schema, even though they're distinct classes.
+    assert fp_a == fp_dup
+    assert fp_a != fp_b
+
+
+# ---- get_or_create lifecycle ---------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_first_call_creates_subsequent_reuses():
+    client, create_mock = _make_client()
+    mgr = GeminiCacheManager(client)
+
+    name1 = await mgr.get_or_create(
+        model="gemini-3.1-flash-lite",
+        system_instruction="Extract facts.",
+        response_schema=None,
+    )
+    name2 = await mgr.get_or_create(
+        model="gemini-3.1-flash-lite",
+        system_instruction="Extract facts.",
+        response_schema=None,
+    )
+
+    assert name1 == "cachedContents/test-cache-name-001"
+    assert name2 == name1
+    # Only ONE underlying create call — second was served from in-memory cache.
+    assert create_mock.call_count == 1
+
+
+@pytest.mark.asyncio
+async def test_different_prefixes_create_separately():
+    client, create_mock = _make_client(
+        create_side_effect=lambda *a, **kw: SimpleNamespace(
+            name=f"cachedContents/created-{create_mock.call_count}"
+        )
+    )
+    mgr = GeminiCacheManager(client)
+
+    name_a = await mgr.get_or_create(
+        model="m", system_instruction="A", response_schema=None
+    )
+    name_b = await mgr.get_or_create(
+        model="m", system_instruction="B", response_schema=None
+    )
+    assert name_a != name_b
+    assert create_mock.call_count == 2
+
+
+# ---- Failure / fallback handling -----------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_minimum_token_count_error_returns_none():
+    """When the prefix is too short, Gemini rejects with a 'minimum
+    token count' style message. Manager must surface this as None so
+    the caller transparently falls back to a non-cached call."""
+    err = Exception("Cached content must have at least 1024 input tokens (minimum)")
+    client, _ = _make_client(create_side_effect=err)
+    mgr = GeminiCacheManager(client)
+
+    result = await mgr.get_or_create(
+        model="m", system_instruction="tiny", response_schema=None
+    )
+    assert result is None
+
+
+@pytest.mark.asyncio
+async def test_other_sdk_errors_also_return_none():
+    """Transient errors (rate limits, 5xx, etc.) should fail soft so
+    a single bad create doesn't crash every retain call."""
+    err = RuntimeError("transient backend error 503")
+    client, _ = _make_client(create_side_effect=err)
+    mgr = GeminiCacheManager(client)
+
+    result = await mgr.get_or_create(
+        model="m", system_instruction="ok-sized prefix", response_schema=None
+    )
+    assert result is None
+
+
+@pytest.mark.asyncio
+async def test_failed_create_does_not_poison_cache():
+    """If create fails on first attempt, a retry should call create
+    again instead of returning a stale/None entry."""
+    call_log = []
+
+    async def maybe_fail(*args, **kwargs):
+        call_log.append(1)
+        if len(call_log) == 1:
+            raise RuntimeError("first call fails")
+        return SimpleNamespace(name="cachedContents/recovered")
+
+    client = MagicMock()
+    client.aio = MagicMock()
+    client.aio.caches = MagicMock()
+    client.aio.caches.create = maybe_fail
+
+    mgr = GeminiCacheManager(client)
+
+    first = await mgr.get_or_create(
+        model="m", system_instruction="prefix", response_schema=None
+    )
+    second = await mgr.get_or_create(
+        model="m", system_instruction="prefix", response_schema=None
+    )
+
+    assert first is None
+    assert second == "cachedContents/recovered"
+    assert len(call_log) == 2
+
+
+# ---- TTL behaviour --------------------------------------------------------
+
+
+@pytest.mark.asyncio
+async def test_refreshes_after_ttl_margin(monkeypatch):
+    """An entry created at t=0 with ttl=10 and margin=2 should be
+    treated as stale at t>=8 and trigger a recreate."""
+    client, create_mock = _make_client(
+        create_side_effect=lambda *a, **kw: SimpleNamespace(
+            name=f"cachedContents/v{create_mock.call_count}"
+        )
+    )
+    mgr = GeminiCacheManager(client, ttl_seconds=10, refresh_margin_seconds=2)
+
+    fake_now = {"t": 1000.0}
+    monkeypatch.setattr(
+        "hindsight_api.engine.providers.gemini_cache.time.monotonic",
+        lambda: fake_now["t"],
+    )
+
+    first = await mgr.get_or_create(
+        model="m", system_instruction="p", response_schema=None
+    )
+    assert first == "cachedContents/v1"
+
+    # Advance to just before the refresh boundary — should reuse.
+    fake_now["t"] = 1000.0 + 7.0
+    again = await mgr.get_or_create(
+        model="m", system_instruction="p", response_schema=None
+    )
+    assert again == "cachedContents/v1"
+    assert create_mock.call_count == 1
+
+    # Advance past the refresh boundary — should recreate.
+    fake_now["t"] = 1000.0 + 9.0
+    refreshed = await mgr.get_or_create(
+        model="m", system_instruction="p", response_schema=None
+    )
+    assert refreshed == "cachedContents/v2"
+    assert create_mock.call_count == 2
+
+
+# ---- Integration: feature flag + GeminiLLM accessor ----------------------
+
+
+@pytest.mark.asyncio
+async def test_gemini_llm_returns_none_when_cache_disabled():
+    """A directly-constructed GeminiLLM (no prompt_cache_enabled kwarg) does not
+    cache: ``get_or_create_cached_prefix`` returns None without ever building a
+    cache manager. The server-level default-on flows in via the kwarg (resolved
+    from config in LLMProvider), not via this constructor default."""
+    from hindsight_api.engine.providers.gemini_llm import GeminiLLM
+
+    llm = GeminiLLM(
+        provider="gemini",
+        api_key="not-real-key",
+        base_url="",
+        model="gemini-test",
+    )
+    # Constructor default is off; even with a stable prefix the cache stays disabled.
+    result = await llm.get_or_create_cached_prefix(
+        system_instruction="A reasonably long system prompt " * 50,
+        response_schema=None,
+    )
+    assert result is None
+    assert llm._cache_manager is None  # never built
+
+
+@pytest.mark.asyncio
+async def test_gemini_llm_uses_cache_when_enabled(monkeypatch):
+    """When the flag is on, the manager is constructed lazily and its
+    get_or_create is delegated to. We don't hit the real SDK; we replace
+    the client's caches.create with a fake."""
+    from types import SimpleNamespace
+    from unittest.mock import AsyncMock
+
+    from hindsight_api.engine.providers.gemini_llm import GeminiLLM
+
+    llm = GeminiLLM(
+        provider="gemini",
+        api_key="not-real-key",
+        base_url="",
+        model="gemini-test",
+        prompt_cache_enabled=True,
+    )
+
+    # Replace the SDK-shaped client with a fake whose caches.create returns
+    # a predictable name. The lazy import inside get_or_create_cached_prefix
+    # picks up the patched module-level GeminiCacheManager naturally.
+    fake_create = AsyncMock(
+        return_value=SimpleNamespace(name="cachedContents/from-llm-test")
+    )
+    llm._client = MagicMock()
+    llm._client.aio = MagicMock()
+    llm._client.aio.caches = MagicMock()
+    llm._client.aio.caches.create = fake_create
+
+    name = await llm.get_or_create_cached_prefix(
+        system_instruction="A long enough system prompt for caching",
+        response_schema=None,
+    )
+    assert name == "cachedContents/from-llm-test"
+    # The manager was lazy-built on first use.
+    assert llm._cache_manager is not None
+    # Second call within TTL → no new SDK call.
+    again = await llm.get_or_create_cached_prefix(
+        system_instruction="A long enough system prompt for caching",
+        response_schema=None,
+    )
+    assert again == name
+    assert fake_create.call_count == 1
+
+
+@pytest.mark.asyncio
+async def test_call_falls_back_to_uncached_when_cache_400s():
+    """A stale/invalid CachedContent makes the generate call 400. The provider
+    must drop the cache, invalidate the entry, and retry the SAME call inline
+    (prefix re-sent) so caching never breaks a request."""
+    import time
+    from types import SimpleNamespace
+    from unittest.mock import AsyncMock
+
+    from google.genai import errors as genai_errors
+
+    from hindsight_api.engine.providers.gemini_cache import GeminiCacheManager, _CacheEntry
+    from hindsight_api.engine.providers.gemini_llm import GeminiLLM
+
+    llm = GeminiLLM(provider="gemini", api_key="not-real-key", base_url="", model="gemini-test", prompt_cache_enabled=True)
+
+    # Seed a cache manager entry that maps to the (now invalid) cache name.
+    mgr = GeminiCacheManager(client=MagicMock())
+    mgr._entries["fp"] = _CacheEntry(name="cachedContents/stale", created_at=time.monotonic(), ttl_seconds=3300)
+    llm._cache_manager = mgr
+
+    captured = []
+
+    def _gen(*, model, contents, config):
+        captured.append(config)
+        if len(captured) == 1:
+            # First (cached) attempt — Gemini rejects the dead cache.
+            raise genai_errors.ClientError(
+                400, {"error": {"code": 400, "status": "INVALID_ARGUMENT", "message": "CachedContent not found"}}
+            )
+        # Retry without the cache succeeds.
+        return SimpleNamespace(
+            text="extracted",
+            usage_metadata=SimpleNamespace(
+                prompt_token_count=10, candidates_token_count=2, cached_content_token_count=0, thoughts_token_count=0
+            ),
+            candidates=[SimpleNamespace(finish_reason="STOP")],
+        )
+
+    llm._client = MagicMock()
+    llm._client.aio = MagicMock()
+    llm._client.aio.models = MagicMock()
+    llm._client.aio.models.generate_content = AsyncMock(side_effect=_gen)
+
+    result = await llm.call(
+        messages=[{"role": "system", "content": "SYSTEM PREFIX"}, {"role": "user", "content": "doc"}],
+        cached_prefix="cachedContents/stale",
+        max_retries=2,
+        temperature=0.1,
+    )
+
+    # The request succeeded via the uncached retry.
+    assert result == "extracted"
+    assert len(captured) == 2
+    # First attempt referenced the cache; the retry inlined the prefix instead.
+    assert captured[0].cached_content == "cachedContents/stale"
+    assert captured[1].cached_content is None
+    assert captured[1].system_instruction == "SYSTEM PREFIX"
+    # The dead entry was invalidated so the next operation recreates it.
+    assert mgr._entries == {}
+
+
+@pytest.mark.asyncio
+async def test_create_cache_times_out_and_falls_back():
+    """The create runs under the manager lock, so a hung caches.create would block
+    every concurrent caller (e.g. all chunks of a retain batch). It must time out
+    and return None so callers proceed uncached instead of stalling."""
+    import asyncio
+    from types import SimpleNamespace
+
+    from hindsight_api.engine.providers.gemini_cache import GeminiCacheManager
+
+    async def _hang(*args, **kwargs):
+        await asyncio.sleep(5)
+        return SimpleNamespace(name="never")
+
+    client = MagicMock()
+    client.aio = MagicMock()
+    client.aio.caches = MagicMock()
+    client.aio.caches.create = _hang
+
+    mgr = GeminiCacheManager(client, create_timeout_seconds=0.05)
+    result = await mgr.get_or_create(model="m", system_instruction="long enough prefix " * 20)
+    assert result is None
+    assert mgr._entries == {}
+
+
+# ---- Tools: cache key + create wiring -----------------------------------
+
+
+def test_fingerprint_changes_with_tools():
+    """Two prefixes that differ ONLY in tools must hash differently —
+    otherwise a loop that adds a tool would silently reuse a stale
+    cache that doesn't know about it."""
+    tools_a = [
+        {"type": "function", "function": {"name": "search", "description": "search", "parameters": {}}}
+    ]
+    tools_b = [
+        {"type": "function", "function": {"name": "search", "description": "search", "parameters": {}}},
+        {"type": "function", "function": {"name": "fetch", "description": "fetch", "parameters": {}}},
+    ]
+    fp_a = GeminiCacheManager.fingerprint("m", "sys", None, tools=tools_a)
+    fp_b = GeminiCacheManager.fingerprint("m", "sys", None, tools=tools_b)
+    assert fp_a != fp_b
+
+
+def test_fingerprint_stable_under_dict_reordering():
+    """The tools list contains dicts; iteration order of dict keys
+    must not affect the fingerprint (otherwise upstream re-serialisation
+    would produce phantom cache misses)."""
+    tools_1 = [{"type": "function", "function": {"description": "d", "name": "n", "parameters": {"a": 1, "b": 2}}}]
+    tools_2 = [{"function": {"parameters": {"b": 2, "a": 1}, "name": "n", "description": "d"}, "type": "function"}]
+    fp_1 = GeminiCacheManager.fingerprint("m", "sys", None, tools=tools_1)
+    fp_2 = GeminiCacheManager.fingerprint("m", "sys", None, tools=tools_2)
+    assert fp_1 == fp_2
+
+
+@pytest.mark.asyncio
+async def test_get_or_create_passes_tools_to_create():
+    """When tools are provided, the underlying caches.create call
+    must include them so the cached prefix actually contains the tool
+    definitions."""
+    captured = {}
+
+    async def fake_create(*, model, config):
+        captured["model"] = model
+        captured["config_dict"] = config.__dict__ if hasattr(config, "__dict__") else dict(config)
+        return SimpleNamespace(name="cachedContents/with-tools")
+
+    client = MagicMock()
+    client.aio = MagicMock()
+    client.aio.caches = MagicMock()
+    client.aio.caches.create = fake_create
+
+    mgr = GeminiCacheManager(client)
+    tools = [
+        {"type": "function", "function": {"name": "search", "description": "do a search", "parameters": {"type": "object"}}}
+    ]
+    name = await mgr.get_or_create(
+        model="gemini-3.1-flash-lite",
+        system_instruction="You are a helpful tool-using assistant.",
+        tools=tools,
+    )
+    assert name == "cachedContents/with-tools"
+    # The Gemini SDK's CreateCachedContentConfig accepted a `tools` list.
+    cfg = captured["config_dict"]
+    assert "tools" in cfg, f"tools should be in cache config; got keys: {list(cfg.keys())}"
+    assert cfg["tools"], "tools list should be non-empty"
@@ -0,0 +1,294 @@
+"""Measure the cached/input token ratio per operation against real Gemini.
+
+Each operation re-sends a large constant prefix and a small variable payload:
+- ``retain_extract_facts`` — fact-extraction system prompt + schema
+- ``reflect_tool_call`` — agent system prompt + tool definitions, reused across
+  every iteration of the tool loop
+- ``consolidation`` — the stable mission/rules/decision/output system prefix,
+  reused across every consolidation batch
+
+We run real Gemini, route every call through the LLM-request tracer (#1922), and
+read back recorded ``cached_tokens`` vs ``input_tokens`` per scope.
+
+Two modes:
+- Default (implicit only): Gemini's automatic caching — empirically ~0% for this
+  low-QPS access pattern. Structural assertions only; the ratio is a measurement.
+- ``HINDSIGHT_GEMINI_EXPLICIT_CACHE=1``: enables PR #1936's explicit CachedContent
+  caching. Then each operation must visibly engage the cache (cached_tokens > 0
+  and ratio above a conservative floor) — this is the regression guard that the
+  caching actually works end-to-end through the retain/reflect/consolidation paths.
+
+Gated on ``HINDSIGHT_RUN_GEMINI_EVALS=1`` plus a Gemini API key, since it costs
+money and needs network. The default model is ``gemini-2.5-flash`` (override with
+``HINDSIGHT_GEMINI_EVAL_MODEL``); explicit caching needs a >=2,048-token prefix.
+"""
+
+import asyncio
+import os
+import uuid
+
+import pytest
+
+from hindsight_api import MemoryEngine, RequestContext
+from hindsight_api.engine.consolidation.consolidator import run_consolidation_job
+from hindsight_api.engine.llm_trace import LLMRequestEntry
+from hindsight_api.engine.llm_wrapper import LLMConfig
+
+_GEMINI_API_KEY = (
+    os.getenv("HINDSIGHT_GEMINI_API_KEY") or os.getenv("GEMINI_API_KEY") or os.getenv("GOOGLE_API_KEY")
+)
+_RUN = os.getenv("HINDSIGHT_RUN_GEMINI_EVALS") == "1" and bool(_GEMINI_API_KEY)
+
+pytestmark = pytest.mark.skipif(
+    not _RUN,
+    reason=(
+        "Gemini implicit-cache measurement is gated. Set HINDSIGHT_RUN_GEMINI_EVALS=1 "
+        "and provide GEMINI_API_KEY/GOOGLE_API_KEY to run."
+    ),
+)
+
+# Number of retain "chunks": each is a separate retain call → a separate
+# retain_extract_facts LLM call that re-sends the same ~3k-token system prefix.
+# That repetition is the precondition for any caching (implicit or explicit) to
+# kick in. Override with HINDSIGHT_GEMINI_CACHE_CHUNKS.
+_CHUNKS = int(os.getenv("HINDSIGHT_GEMINI_CACHE_CHUNKS", "5"))
+
+# Distinct paragraphs so each retain extracts real, non-duplicate facts. The
+# *content* varies per call; the system prompt / schema prefix does not — which
+# is exactly the shape caching targets.
+_DOCS = [
+    "Ada Lovelace worked with Charles Babbage on the Analytical Engine in the 1840s. "
+    "She wrote what is often considered the first algorithm intended for a machine, "
+    "a method for computing Bernoulli numbers. She lived in London and corresponded "
+    "extensively with Babbage about the engine's capabilities.",
+    "Grace Hopper joined the Harvard Mark I team in 1944 and later developed the first "
+    "compiler, A-0, in 1952. She championed machine-independent programming languages, "
+    "which led to COBOL. She served in the US Navy and retired as a rear admiral.",
+    "Katherine Johnson computed orbital mechanics for NASA's first crewed spaceflights. "
+    "John Glenn personally asked her to verify the electronic computer's calculations "
+    "before his 1962 Friendship 7 orbit. She worked at Langley Research Center in Virginia.",
+    "Alan Turing formalized computation with the Turing machine in 1936 and worked at "
+    "Bletchley Park during World War II breaking the Enigma cipher. He proposed the "
+    "imitation game, now called the Turing test, in a 1950 paper on machine intelligence.",
+    "Margaret Hamilton led the software engineering team that wrote the onboard flight "
+    "software for the Apollo missions at MIT. Her error-detection code prevented an abort "
+    "during the Apollo 11 landing in 1969. She later coined the term 'software engineering'.",
+    "Barbara Liskov designed the CLU programming language in the 1970s and introduced data "
+    "abstraction. The Liskov substitution principle is named after her. She won the Turing "
+    "Award in 2008 for contributions to programming language and system design.",
+    "Tim Berners-Lee invented the World Wide Web in 1989 while at CERN, writing the first "
+    "browser and the HTTP protocol. He founded the World Wide Web Consortium in 1994 to "
+    "develop open web standards.",
+    "Radia Perlman invented the spanning-tree protocol while at Digital Equipment Corporation, "
+    "which made large bridged Ethernet networks possible. She is sometimes called the mother "
+    "of the internet, a title she has said she dislikes.",
+]
+
+
+# Explicit Gemini prompt caching (PR #1936) — opt-in. Set HINDSIGHT_GEMINI_EXPLICIT_CACHE=1
+# to turn it on for this run. On a branch without the feature the flag is simply
+# ignored, so the same test measures the implicit baseline there.
+_EXPLICIT_CACHE = os.getenv("HINDSIGHT_GEMINI_EXPLICIT_CACHE") == "1"
+
+
+async def _gemini_engine(memory_no_llm_verify: MemoryEngine) -> MemoryEngine:
+    """Point an engine at real Gemini and force-enable the LLM-request tracer.
+
+    The fixture builds the engine with tracing disabled (config default). The
+    recorder reads ``enabled`` once at construction, so we flip the flag directly
+    rather than rebuilding the engine — equivalent to running with
+    ``HINDSIGHT_API_LLM_TRACE_ENABLED=true``.
+
+    When ``HINDSIGHT_GEMINI_EXPLICIT_CACHE=1`` we also enable PR #1936's explicit
+    CachedContent caching the production way (env var + config-cache clear), so we
+    can compare its cached/input ratio against the implicit baseline. Otherwise the
+    only caching observed is Gemini's own implicit caching.
+    """
+    from hindsight_api.config import clear_config_cache
+
+    model = os.getenv("HINDSIGHT_GEMINI_EVAL_MODEL", "gemini-2.5-flash")
+    # Prompt caching is on by default now, so the implicit-baseline run must
+    # explicitly DISABLE it (not just leave it unset) to measure Gemini's own
+    # implicit caching. Set the flag in both modes and clear the config cache so
+    # the per-bank resolver re-reads it.
+    os.environ["HINDSIGHT_API_LLM_PROMPT_CACHE_ENABLED"] = "true" if _EXPLICIT_CACHE else "false"
+    clear_config_cache()
+    cfg = LLMConfig(
+        provider="gemini",
+        api_key=_GEMINI_API_KEY or "",
+        base_url="",
+        model=model,
+        prompt_cache_enabled=_EXPLICIT_CACHE,
+    )
+    memory_no_llm_verify._llm_config = cfg
+    memory_no_llm_verify._retain_llm_config = cfg
+    memory_no_llm_verify._reflect_llm_config = cfg
+    memory_no_llm_verify._consolidation_llm_config = cfg
+    memory_no_llm_verify._llm_recorder._enabled = True
+    mode = "EXPLICIT cache ON" if _EXPLICIT_CACHE else "implicit only"
+    print(f"\n[gemini-cache] provider=gemini model={model} chunks={_CHUNKS} mode={mode}")
+    return memory_no_llm_verify
+
+
+async def _drain_traces(mem: MemoryEngine) -> None:
+    """Wait for the recorder's fire-and-forget trace writes to land.
+
+    record_llm_call schedules each INSERT as a detached asyncio task tracked in
+    ``_pending`` (bucketed by trace_id). Gather them so the rows are queryable.
+    Loop a few times because consolidation's attach_memory_ids can spawn a
+    follow-up write after the first drain.
+    """
+    await mem.wait_for_background_tasks()
+    rec = mem._llm_recorder
+    for _ in range(10):
+        pending = [t for bucket in rec._pending.values() for t in bucket if not t.done()]
+        if not pending:
+            break
+        await asyncio.gather(*pending, return_exceptions=True)
+
+
+def _report(scope: str, rows: list[LLMRequestEntry]) -> float:
+    """Print the cached/input token ratio for a scope and return it.
+
+    Gemini's ``prompt_token_count`` (our ``input_tokens``) already includes the
+    cached prefix, so ``cached_tokens / input_tokens`` is the fraction of prompt
+    tokens billed at the cheaper cached rate — the number the PR's
+    ``cached_input / input`` dashboard would show.
+    """
+    input_total = sum((r.input_tokens or 0) for r in rows)
+    cached_total = sum((r.cached_tokens or 0) for r in rows)
+    output_total = sum((r.output_tokens or 0) for r in rows)
+    ratio = (cached_total / input_total) if input_total else 0.0
+    per_call = ", ".join(f"{(r.cached_tokens or 0)}/{(r.input_tokens or 0)}" for r in rows)
+    mode = "explicit cache ON" if _EXPLICIT_CACHE else "implicit only"
+    print(
+        f"\n[gemini-cache] scope={scope!r} calls={len(rows)}  ({mode})\n"
+        f"  input_tokens   = {input_total}\n"
+        f"  cached_tokens  = {cached_total}\n"
+        f"  output_tokens  = {output_total}\n"
+        f"  cached/input   = {ratio:.1%}\n"
+        f"  per-call cached/input: {per_call}"
+    )
+    return ratio
+
+
+@pytest.mark.hs_llm_core
+class TestGeminiCacheRatioPerOperation:
+    """Measure cached/input token ratio per operation (retain, reflect, consolidation).
+
+    Run with ``HINDSIGHT_GEMINI_EXPLICIT_CACHE=1`` to assert PR #1936's explicit
+    CachedContent caching actually engages (cached tokens > 0, ratio above a
+    conservative floor). Without it, the same tests record the implicit-caching
+    baseline (Gemini gives ~0% for this access pattern) without asserting a floor.
+    """
+
+    async def _fetch(self, mem: MemoryEngine, bank_id: str, rc: RequestContext, scope: str) -> list[LLMRequestEntry]:
+        resp = await mem.list_llm_requests(bank_id, request_context=rc, scope=scope, limit=200)
+        assert resp is not None, "bank should exist"
+        return [r for r in resp.items if r.status == "success"]
+
+    def _assert(self, scope: str, rows: list[LLMRequestEntry], *, min_calls: int, min_ratio: float) -> float:
+        """Common per-operation checks; returns the cached/input ratio.
+
+        ``min_ratio`` is per-operation because the achievable ratio differs by
+        design: retain re-sends a pure fixed prefix (~90%); consolidation's prefix
+        is fixed but the facts/observations payload is large (~30%); reflect can
+        only cache its ``auto`` iterations — Gemini forbids ``cached_content`` with
+        a per-request ``tool_config`` — and the tool-result context grows, so the
+        ratio is modest (~10%). The universal guarantee in explicit mode is simply
+        that caching engaged at all (cached_tokens > 0).
+        """
+        ratio = _report(scope, rows)
+        cached_total = sum((r.cached_tokens or 0) for r in rows)
+        assert len(rows) >= min_calls, f"expected >= {min_calls} {scope} calls, got {len(rows)}"
+        assert all((r.provider == "gemini") for r in rows)
+        assert sum((r.input_tokens or 0) for r in rows) > 0, "no input tokens recorded"
+        assert 0.0 <= ratio <= 1.0
+        if _EXPLICIT_CACHE:
+            assert cached_total > 0, f"{scope}: explicit cache ON but cached_tokens=0 (caching did not engage)"
+            assert ratio >= min_ratio, f"{scope}: cached/input {ratio:.1%} below floor {min_ratio:.0%}"
+        return ratio
+
+    async def test_retain_chunks_cached_ratio(self, memory_no_llm_verify, request_context):
+        """Retain N distinct chunks → N fact-extraction calls sharing one prefix."""
+        mem = await _gemini_engine(memory_no_llm_verify)
+        bank_id = f"gemini-cache-retain-{uuid.uuid4().hex[:8]}"
+        await mem.get_bank_profile(bank_id, request_context=request_context)
+
+        docs = [_DOCS[i % len(_DOCS)] for i in range(_CHUNKS)]
+        for i, content in enumerate(docs):
+            await mem.retain_batch_async(
+                bank_id=bank_id,
+                contents=[{"content": content}],
+                request_context=request_context,
+                document_id=f"doc-{i}",
+            )
+        await _drain_traces(mem)
+
+        rows = await self._fetch(mem, bank_id, request_context, "retain_extract_facts")
+        self._assert("retain_extract_facts", rows, min_calls=_CHUNKS, min_ratio=0.5)
+
+        await mem.delete_bank(bank_id, request_context=request_context)
+
+    async def test_reflect_tool_loop_cached_ratio(self, memory_no_llm_verify, request_context):
+        """Reflect runs an agentic tool loop; the system_prompt + tools prefix is
+        cached once and reused across every iteration (scope ``reflect_tool_call``)."""
+        mem = await _gemini_engine(memory_no_llm_verify)
+        bank_id = f"gemini-cache-reflect-{uuid.uuid4().hex[:8]}"
+        await mem.get_bank_profile(bank_id, request_context=request_context)
+
+        await mem.retain_batch_async(
+            bank_id=bank_id,
+            contents=[{"content": d} for d in _DOCS],
+            request_context=request_context,
+        )
+        await mem.wait_for_background_tasks()
+
+        # A broad question forces the agent to call recall/lookup tools, i.e. to
+        # iterate the tool loop more than once.
+        await mem.reflect_async(
+            bank_id=bank_id,
+            query="Who were the early pioneers of computing in these memories, and what is each one known for?",
+            request_context=request_context,
+        )
+        await _drain_traces(mem)
+
+        rows = await self._fetch(mem, bank_id, request_context, "reflect_tool_call")
+        if not rows:
+            # Diagnostic: dump every reflect_tool_call row (incl. errors) so a
+            # failure in the cached tool-loop path is visible, not silently skipped.
+            allresp = await mem.list_llm_requests(
+                bank_id, request_context=request_context, scope="reflect_tool_call", limit=200
+            )
+            for r in allresp.items if allresp else []:
+                print(f"\n[gemini-cache] reflect_tool_call status={r.status} error={r.error}")
+            pytest.skip("reflect made no SUCCESSFUL reflect_tool_call iterations for this seed")
+        self._assert("reflect_tool_call", rows, min_calls=1, min_ratio=0.0)
+
+        await mem.delete_bank(bank_id, request_context=request_context)
+
+    async def test_consolidation_cached_ratio(self, memory_no_llm_verify, request_context):
+        """Retain a batch, then consolidate; the stable system prefix is cached and
+        reused across every consolidation batch (scope ``consolidation``)."""
+        mem = await _gemini_engine(memory_no_llm_verify)
+        bank_id = f"gemini-cache-consol-{uuid.uuid4().hex[:8]}"
+        await mem.get_bank_profile(bank_id, request_context=request_context)
+
+        # Seed enough unconsolidated memories that consolidation makes several
+        # same-prefix LLM calls.
+        await mem.retain_batch_async(
+            bank_id=bank_id,
+            contents=[{"content": d} for d in _DOCS],
+            request_context=request_context,
+        )
+        await mem.wait_for_background_tasks()
+
+        await run_consolidation_job(mem, bank_id, request_context)
+        await _drain_traces(mem)
+
+        rows = await self._fetch(mem, bank_id, request_context, "consolidation")
+        if not rows:
+            pytest.skip("consolidation made no LLM calls for this seed (nothing to consolidate)")
+        self._assert("consolidation", rows, min_calls=1, min_ratio=0.15)
+
+        await mem.delete_bank(bank_id, request_context=request_context)
@@ -186,8 +186,8 @@ async def test_graph_document_filter_includes_observations_via_source_memories(
        # World fact tied to the document.
        await conn.execute(
            """
-            INSERT INTO memory_units (id, bank_id, text, fact_type, document_id, history)
-            VALUES ($1, $2, $3, 'world', $4, '[]'::jsonb)
+            INSERT INTO memory_units (id, bank_id, text, fact_type, document_id)
+            VALUES ($1, $2, $3, 'world', $4)
            """,
            fact_id,
            bank_id,
@@ -198,8 +198,8 @@ async def test_graph_document_filter_includes_observations_via_source_memories(
        # Unrelated fact NOT tied to the document.
        await conn.execute(
            """
-            INSERT INTO memory_units (id, bank_id, text, fact_type, history)
-            VALUES ($1, $2, $3, 'world', '[]'::jsonb)
+            INSERT INTO memory_units (id, bank_id, text, fact_type)
+            VALUES ($1, $2, $3, 'world')
            """,
            other_fact_id,
            bank_id,
@@ -210,9 +210,9 @@ async def test_graph_document_filter_includes_observations_via_source_memories(
        await conn.execute(
            """
            INSERT INTO memory_units (
-                id, bank_id, text, fact_type, source_memory_ids, history, proof_count
+                id, bank_id, text, fact_type, source_memory_ids, proof_count
            )
-            VALUES ($1, $2, $3, 'observation', $4::uuid[], '[]'::jsonb, 1)
+            VALUES ($1, $2, $3, 'observation', $4::uuid[], 1)
            """,
            observation_id,
            bank_id,
@@ -224,9 +224,9 @@ async def test_graph_document_filter_includes_observations_via_source_memories(
        await conn.execute(
            """
            INSERT INTO memory_units (
-                id, bank_id, text, fact_type, source_memory_ids, history, proof_count
+                id, bank_id, text, fact_type, source_memory_ids, proof_count
            )
-            VALUES ($1, $2, $3, 'observation', $4::uuid[], '[]'::jsonb, 1)
+            VALUES ($1, $2, $3, 'observation', $4::uuid[], 1)
            """,
            unrelated_observation_id,
            bank_id,
@@ -38,11 +38,17 @@ class MockTenantExtension(TenantExtension):
        return self.tenant_config


+class _FakeBankOps:
+    async def create_bank_vector_indexes(self, *args, **kwargs):
+        return None
+
+
 class FakeBankConfigBackend:
    """Minimal backend for ConfigResolver bank-config tests."""

    def __init__(self):
        self.config: dict[str, object] = {}
+        self.ops = _FakeBankOps()

    def acquire(self):
        return FakeBankConfigConnection(self)
@@ -61,6 +67,11 @@ class FakeBankConfigConnection:
    async def fetchrow(self, query, bank_id):
        return {"config": self.backend.config}

+    async def fetchval(self, query, *args):
+        # ensure_bank_exists INSERT ... ON CONFLICT DO NOTHING RETURNING bank_id.
+        # Return None to simulate the bank already existing (no index creation).
+        return None
+
    async def execute(self, query, updates_json, bank_id):
        self.backend.config.update(json.loads(updates_json))

@@ -231,7 +231,13 @@ async def test_horse_farm_observation_history(memory_real_llm: MemoryEngine, req
            async with pool.acquire() as conn:
                observations = await conn.fetch(
                    """
-                    SELECT id, text, proof_count, source_memory_ids, history
+                    SELECT id, text, proof_count, source_memory_ids,
+                           COALESCE((
+                               SELECT jsonb_agg(jsonb_build_object('previous_text', oh.content->>'previous_text')
+                                                ORDER BY oh.changed_at, oh.id)
+                               FROM observation_history oh
+                               WHERE oh.observation_id = memory_units.id
+                           ), '[]'::jsonb) AS history
                    FROM memory_units
                    WHERE bank_id = $1 AND fact_type = 'observation'
                    ORDER BY created_at
@@ -255,7 +261,13 @@ async def test_horse_farm_observation_history(memory_real_llm: MemoryEngine, req
    async with pool.acquire() as conn:
        observations = await conn.fetch(
            """
-            SELECT id, text, proof_count, source_memory_ids, history
+            SELECT id, text, proof_count, source_memory_ids,
+                   COALESCE((
+                       SELECT jsonb_agg(jsonb_build_object('previous_text', oh.content->>'previous_text')
+                                        ORDER BY oh.changed_at, oh.id)
+                       FROM observation_history oh
+                       WHERE oh.observation_id = memory_units.id
+                   ), '[]'::jsonb) AS history
            FROM memory_units
            WHERE bank_id = $1 AND fact_type = 'observation'
            ORDER BY created_at
@@ -1104,6 +1104,8 @@ async def test_version_endpoint_returns_correct_version(api_client):
    assert isinstance(features["observations"], bool)
    assert isinstance(features["mcp"], bool)
    assert isinstance(features["worker"], bool)
+    assert isinstance(features["audit_log"], bool)
+    assert isinstance(features["llm_trace"], bool)

    print(f"Version endpoint returned: api_version={result['api_version']}, features={features}")

@@ -1315,6 +1317,42 @@ async def test_unknown_params_not_rejected(api_client):
    assert "X-Ignored-Params" not in response.headers


+@pytest.mark.asyncio
+@pytest.mark.parametrize("field", ["enable_observations", "enable_auto_consolidation"])
+async def test_patch_config_persists_override_for_uncreated_bank(api_client, field):
+    """PATCH config must persist the override even when the bank was never retained.
+
+    Banks are created lazily on first retain, so a PATCH that precedes any
+    ingestion previously UPDATE-d zero rows and silently no-op'd while returning
+    200 (issue #1940). The endpoint must auto-create the bank and round-trip the
+    override in the same response.
+    """
+    test_bank_id = f"patch_uncreated_{field}_{datetime.now().timestamp()}"
+
+    # PATCH config without ever creating the bank (no PUT, no retain).
+    response = await api_client.patch(
+        f"/v1/default/banks/{test_bank_id}/config",
+        json={"updates": {field: False}},
+    )
+    assert response.status_code == 200, response.text
+    body = response.json()
+    assert body["config"][field] is False
+    assert body["overrides"].get(field) is False
+
+    # GET reads back the persisted override (proves it was written, not just echoed).
+    response = await api_client.get(f"/v1/default/banks/{test_bank_id}/config")
+    assert response.status_code == 200, response.text
+    body = response.json()
+    assert body["config"][field] is False
+    assert body["overrides"].get(field) is False
+
+    # The auto-created bank must have a name (defaults to bank_id). A NULL name
+    # would 500 the profile endpoint, whose response types name as a required str.
+    response = await api_client.get(f"/v1/default/banks/{test_bank_id}/profile")
+    assert response.status_code == 200, response.text
+    assert response.json()["name"] == test_bank_id
+
+
@pytest.mark.hs_llm_core
@pytest.mark.asyncio
 async def test_full_api_workflow_llm_quality(api_client_real_llm):
@@ -0,0 +1,61 @@
+"""Unit tests for round-robin interleave fusion (consolidation dedup recall)."""
+
+from hindsight_api.engine.search.fusion import interleave_fusion
+from hindsight_api.engine.search.types import RetrievalResult
+
+
+def _r(doc_id: str) -> RetrievalResult:
+    return RetrievalResult(id=doc_id, text=f"text-{doc_id}", fact_type="observation")
+
+
+def _ids(merged) -> list[str]:
+    return [mc.id for mc in merged]
+
+
+def test_round_robin_order_takes_each_arm_in_turn():
+    semantic = [_r("s1"), _r("s2"), _r("s3")]
+    bm25 = [_r("b1"), _r("b2")]
+    graph = [_r("g1")]
+
+    merged = interleave_fusion([semantic, bm25, graph])
+
+    # Round 0: s1, b1, g1 ; round 1: s2, b2 ; round 2: s3
+    assert _ids(merged) == ["s1", "b1", "g1", "s2", "b2", "s3"]
+
+
+def test_semantic_top_hit_is_always_first():
+    # The dedup twin: semantic #1 but absent from every other arm. Must still lead.
+    semantic = [_r("twin"), _r("s2")]
+    bm25 = [_r("b1"), _r("b2"), _r("b3")]
+    graph = [_r("g1")]
+
+    merged = interleave_fusion([semantic, bm25, graph])
+
+    assert merged[0].id == "twin"
+
+
+def test_dedup_keeps_first_occurrence_and_records_all_arm_ranks():
+    # "x" is semantic #1 and bm25 #2; it should appear once, at its first (semantic) slot,
+    # but carry ranks from every arm it appears in.
+    semantic = [_r("x"), _r("s2")]
+    bm25 = [_r("b1"), _r("x")]
+
+    merged = interleave_fusion([semantic, bm25])
+
+    assert _ids(merged) == ["x", "b1", "s2"]
+    x = next(mc for mc in merged if mc.id == "x")
+    assert x.source_ranks == {"semantic_rank": 1, "bm25_rank": 2}
+
+
+def test_rrf_score_strictly_decreasing_preserves_order_on_sort():
+    merged = interleave_fusion([[_r("a"), _r("b")], [_r("c")]])
+    scores = [mc.rrf_score for mc in merged]
+    assert scores == sorted(scores, reverse=True)
+    assert len(set(scores)) == len(scores)  # strictly decreasing, no ties
+    # rrf_rank reflects the interleave position
+    assert [mc.rrf_rank for mc in merged] == [1, 2, 3]
+
+
+def test_empty_inputs():
+    assert interleave_fusion([]) == []
+    assert interleave_fusion([[], []]) == []
@@ -104,6 +104,35 @@ class TestLiteLLMSDKCrossEncoder:
            assert len(call_args.kwargs["documents"]) == 3
            assert call_args.kwargs["api_key"] == "test_key"

+    def test_constructor_without_api_key(self):
+        """api_key is optional (e.g. AWS Bedrock reranker with ambient IAM creds)."""
+        encoder = LiteLLMSDKCrossEncoder(model="bedrock/cohere.rerank-v3-5:0")
+        assert encoder.api_key is None
+
+    @pytest.mark.asyncio
+    async def test_predict_omits_api_key_for_ambient_credentials(self):
+        """When no api_key is set, it must not be injected into the rerank call.
+
+        litellm maps an explicit ``api_key`` to ``aws_access_key_id`` for Bedrock,
+        which overrides ambient IAM/task-role credentials; omitting it lets litellm
+        resolve credentials from the environment (regression test for IAM auth).
+        """
+        encoder = LiteLLMSDKCrossEncoder(model="bedrock/cohere.rerank-v3-5:0")
+
+        mock_response = MagicMock()
+        mock_response.results = [{"index": 0, "relevance_score": 0.9}]
+
+        mock_litellm = MagicMock()
+        mock_litellm.arerank = AsyncMock(return_value=mock_response)
+
+        with patch.dict("sys.modules", {"litellm": mock_litellm}):
+            await encoder.initialize()
+            await encoder.predict([("query", "document")])
+
+            mock_litellm.arerank.assert_called_once()
+            call_kwargs = mock_litellm.arerank.call_args.kwargs
+            assert "api_key" not in call_kwargs
+
    @pytest.mark.asyncio
    async def test_predict_multiple_queries(self):
        """Test prediction with multiple different queries (grouped efficiently)."""
@@ -278,11 +307,11 @@ class TestFactoryFunction:
                assert encoder.model == "deepinfra/Qwen3-reranker-8B"

    @pytest.mark.asyncio
-    async def test_create_litellm_sdk_missing_api_key(self):
-        """Test that factory raises error when API key is missing."""
+    async def test_create_litellm_sdk_without_api_key(self):
+        """Test that litellm-sdk works without an API key (e.g. AWS Bedrock with IAM)."""
        env_vars = {
            "HINDSIGHT_API_RERANKER_PROVIDER": "litellm-sdk",
-            "HINDSIGHT_API_RERANKER_LITELLM_SDK_MODEL": "deepinfra/Qwen3-reranker-8B",
+            "HINDSIGHT_API_RERANKER_LITELLM_SDK_MODEL": "bedrock/cohere.rerank-v3-5:0",
        }

        with patch.dict(os.environ, env_vars, clear=False):
@@ -295,8 +324,11 @@ class TestFactoryFunction:
            config = HindsightConfig.from_env()

            with patch("hindsight_api.config.get_config", return_value=config):
-                with pytest.raises(ValueError, match="HINDSIGHT_API_RERANKER_LITELLM_SDK_API_KEY is required"):
-                    create_cross_encoder_from_env()
+                encoder = create_cross_encoder_from_env()
+
+                assert isinstance(encoder, LiteLLMSDKCrossEncoder)
+                assert encoder.api_key is None
+                assert encoder.model == "bedrock/cohere.rerank-v3-5:0"

    @pytest.mark.asyncio
    async def test_create_litellm_sdk_with_custom_api_base(self):
@@ -454,7 +454,7 @@ class TestLiteLLMSDKEmbeddings:
        ):
            emb = LiteLLMSDKEmbeddings(
                api_key="test_key",
-                model="gemini/gemini-embedding-2-preview",
+                model="gemini/gemini-embedding-2",
                encoding_format="",
            )
            await emb.initialize()
@@ -542,7 +542,7 @@ class TestLiteLLMSDKEmbeddingsFactory:
        mock_config = MagicMock()
        mock_config.embeddings_provider = "litellm-sdk"
        mock_config.embeddings_litellm_sdk_api_key = "test_key"
-        mock_config.embeddings_litellm_sdk_model = "gemini/gemini-embedding-2-preview"
+        mock_config.embeddings_litellm_sdk_model = "gemini/gemini-embedding-2"
        mock_config.embeddings_litellm_sdk_api_base = None
        mock_config.embeddings_litellm_sdk_output_dimensions = 768

@@ -0,0 +1,254 @@
+"""
+Tests for the ``llm_extra_body`` knob (env: ``HINDSIGHT_API_LLM_EXTRA_BODY``).
+
+The same JSON dict of extra request-body params is threaded into every API
+provider. Each provider merges it in its own native parameter space:
+
+- OpenAI-compatible / Fireworks: OpenAI SDK ``extra_body`` (already covered
+  elsewhere; the wiring predates this change).
+- Anthropic: the Anthropic SDK ``extra_body`` kwarg.
+- Gemini / VertexAI: seeded into ``GenerateContentConfig`` (the SDK's native
+  generation-param space — Gemini nests these in the request body).
+- LiteLLM (+ bedrock alias + router): merged as top-level ``acompletion`` kwargs
+  so LiteLLM normalizes/drops them per-provider.
+
+These are deterministic unit tests: the SDK client is mocked and we assert the
+params actually reach the call.
+"""
+
+import os
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+EXTRA_BODY = {"temperature": 0.2, "top_p": 0.9}
+
+
+# ─── config / env parsing ─────────────────────────────────────────────────────
+
+
+def test_extra_body_parsed_from_env():
+    """The JSON env var is parsed into ``HindsightConfig.llm_extra_body``."""
+    import json
+
+    from hindsight_api.config import ENV_LLM_EXTRA_BODY, HindsightConfig, clear_config_cache
+
+    with patch.dict(os.environ, {ENV_LLM_EXTRA_BODY: json.dumps(EXTRA_BODY)}, clear=False):
+        clear_config_cache()
+        config = HindsightConfig.from_env()
+        assert config.llm_extra_body == EXTRA_BODY
+        clear_config_cache()
+
+
+def test_extra_body_default_is_none():
+    """When the env var is unset, ``llm_extra_body`` defaults to None."""
+    from hindsight_api.config import ENV_LLM_EXTRA_BODY, HindsightConfig, clear_config_cache
+
+    env = {k: v for k, v in os.environ.items() if k != ENV_LLM_EXTRA_BODY}
+    with patch.dict(os.environ, env, clear=True):
+        clear_config_cache()
+        config = HindsightConfig.from_env()
+        assert config.llm_extra_body is None
+        clear_config_cache()
+
+
+# ─── Anthropic ────────────────────────────────────────────────────────────────
+
+
+def _make_anthropic_provider(extra_body=None):
+    with patch("anthropic.AsyncAnthropic") as mock_client_cls:
+        mock_client_cls.return_value = MagicMock()
+        from hindsight_api.engine.providers.anthropic_llm import AnthropicLLM
+
+        provider = AnthropicLLM(
+            provider="anthropic",
+            api_key="fake-key",
+            base_url="",
+            model="claude-sonnet-4-20250514",
+            extra_body=extra_body,
+        )
+    provider._client = MagicMock()
+    return provider
+
+
+def _fake_anthropic_response():
+    block = MagicMock()
+    block.type = "text"
+    block.text = "ok"
+    resp = MagicMock()
+    resp.content = [block]
+    resp.usage = MagicMock(input_tokens=5, output_tokens=2, cache_read_input_tokens=0)
+    resp.stop_reason = "end_turn"
+    return resp
+
+
+def test_anthropic_stores_extra_body():
+    provider = _make_anthropic_provider(extra_body=EXTRA_BODY)
+    assert provider._extra_body == EXTRA_BODY
+
+
+def test_anthropic_empty_extra_body_defaults_to_dict():
+    provider = _make_anthropic_provider(extra_body=None)
+    assert provider._extra_body == {}
+
+
+@pytest.mark.asyncio
+async def test_anthropic_call_passes_extra_body():
+    """``call()`` forwards extra_body via the Anthropic SDK ``extra_body`` kwarg."""
+    provider = _make_anthropic_provider(extra_body=EXTRA_BODY)
+    provider._client.messages.create = AsyncMock(return_value=_fake_anthropic_response())
+
+    with patch("hindsight_api.engine.providers.anthropic_llm.get_metrics_collector"):
+        await provider.call(messages=[{"role": "user", "content": "hi"}], scope="test", max_retries=0)
+
+    kwargs = provider._client.messages.create.call_args.kwargs
+    assert kwargs.get("extra_body") == EXTRA_BODY
+
+
+@pytest.mark.asyncio
+async def test_anthropic_no_extra_body_omits_key():
+    """``call()`` does not pass ``extra_body`` when none is configured."""
+    provider = _make_anthropic_provider(extra_body=None)
+    provider._client.messages.create = AsyncMock(return_value=_fake_anthropic_response())
+
+    with patch("hindsight_api.engine.providers.anthropic_llm.get_metrics_collector"):
+        await provider.call(messages=[{"role": "user", "content": "hi"}], scope="test", max_retries=0)
+
+    assert "extra_body" not in provider._client.messages.create.call_args.kwargs
+
+
+# ─── Gemini ───────────────────────────────────────────────────────────────────
+
+
+def _make_gemini_provider(extra_body=None):
+    pytest.importorskip("google.genai")
+    with patch("google.genai.Client") as mock_client_cls:
+        mock_client_cls.return_value = MagicMock()
+        from hindsight_api.engine.providers.gemini_llm import GeminiLLM
+
+        provider = GeminiLLM(
+            provider="gemini",
+            api_key="fake-key",
+            base_url="",
+            model="gemini-2.5-flash",
+            extra_body=extra_body,
+        )
+    provider._client = MagicMock()
+    return provider
+
+
+def _fake_gemini_response():
+    r = MagicMock()
+    r.text = "hello"
+    r.candidates = [MagicMock(finish_reason="STOP")]
+    r.usage_metadata = MagicMock(prompt_token_count=5, candidates_token_count=2)
+    return r
+
+
+def test_gemini_stores_extra_body():
+    provider = _make_gemini_provider(extra_body=EXTRA_BODY)
+    assert provider._extra_body == EXTRA_BODY
+
+
+@pytest.mark.asyncio
+async def test_gemini_call_applies_extra_body_to_generation_config():
+    """``call()`` seeds extra_body into GenerateContentConfig (temperature/top_p)."""
+    provider = _make_gemini_provider(extra_body=EXTRA_BODY)
+    provider._client.aio.models.generate_content = AsyncMock(return_value=_fake_gemini_response())
+
+    await provider.call(messages=[{"role": "user", "content": "hi"}], scope="test")
+
+    config_arg = provider._client.aio.models.generate_content.call_args.kwargs.get("config")
+    assert config_arg is not None
+    assert config_arg.temperature == 0.2
+    assert config_arg.top_p == 0.9
+
+
+@pytest.mark.asyncio
+async def test_gemini_explicit_temperature_overrides_extra_body():
+    """An explicit per-call temperature wins over the extra_body default."""
+    provider = _make_gemini_provider(extra_body={"temperature": 0.2})
+    provider._client.aio.models.generate_content = AsyncMock(return_value=_fake_gemini_response())
+
+    await provider.call(messages=[{"role": "user", "content": "hi"}], temperature=0.9, scope="test")
+
+    config_arg = provider._client.aio.models.generate_content.call_args.kwargs.get("config")
+    assert config_arg.temperature == 0.9
+
+
+# ─── LiteLLM ──────────────────────────────────────────────────────────────────
+
+
+def _make_litellm_provider(extra_body=None):
+    pytest.importorskip("litellm")
+    from hindsight_api.engine.providers.litellm_llm import LiteLLMLLM
+
+    return LiteLLMLLM(
+        provider="litellm",
+        api_key="fake-key",
+        base_url="",
+        model="gpt-4o",
+        extra_body=extra_body,
+    )
+
+
+def _fake_litellm_response():
+    msg = MagicMock()
+    msg.content = "ok"
+    choice = MagicMock()
+    choice.message = msg
+    choice.finish_reason = "stop"
+    resp = MagicMock()
+    resp.choices = [choice]
+    resp.usage = MagicMock(prompt_tokens=5, completion_tokens=2)
+    return resp
+
+
+def test_litellm_stores_extra_body():
+    provider = _make_litellm_provider(extra_body=EXTRA_BODY)
+    assert provider._extra_body == EXTRA_BODY
+
+
+@pytest.mark.asyncio
+async def test_litellm_call_merges_extra_body_as_top_level_kwargs():
+    """``call()`` merges extra_body into the acompletion kwargs."""
+    provider = _make_litellm_provider(extra_body=EXTRA_BODY)
+    provider._acompletion = AsyncMock(return_value=_fake_litellm_response())
+
+    with patch("hindsight_api.engine.providers.litellm_llm.get_metrics_collector"):
+        await provider.call(messages=[{"role": "user", "content": "hi"}], scope="test", max_retries=0)
+
+    kwargs = provider._acompletion.call_args.kwargs
+    assert kwargs.get("temperature") == 0.2
+    assert kwargs.get("top_p") == 0.9
+
+
+@pytest.mark.asyncio
+async def test_litellm_explicit_param_wins_over_extra_body():
+    """``setdefault`` semantics: an explicit per-call value is not overwritten."""
+    provider = _make_litellm_provider(extra_body={"temperature": 0.2})
+    provider._acompletion = AsyncMock(return_value=_fake_litellm_response())
+
+    with patch("hindsight_api.engine.providers.litellm_llm.get_metrics_collector"):
+        await provider.call(
+            messages=[{"role": "user", "content": "hi"}], temperature=0.9, scope="test", max_retries=0
+        )
+
+    assert provider._acompletion.call_args.kwargs.get("temperature") == 0.9
+
+
+def test_litellm_router_forwards_extra_body():
+    """The Router subclass forwards extra_body through to the shared LiteLLM base."""
+    pytest.importorskip("litellm")
+    from hindsight_api.engine.providers.litellm_router_llm import LiteLLMRouterLLM
+
+    config = {"model_list": [{"model_name": "m", "litellm_params": {"model": "gpt-4o", "api_key": "x"}}]}
+    provider = LiteLLMRouterLLM(
+        provider="litellmrouter",
+        api_key="",
+        base_url="",
+        model="m",
+        config=config,
+        extra_body=EXTRA_BODY,
+    )
+    assert provider._extra_body == EXTRA_BODY
@@ -41,7 +41,7 @@ def two_step_config() -> dict[str, Any]:
            {
                "model_name": "default",
                "litellm_params": {
-                    "model": "openai/MiniMax-M2.7",
+                    "model": "openai/MiniMax-M3",
                    "api_key": "sk-primary",
                    "api_base": "https://api.minimax.io/v1",
                },
@@ -0,0 +1,190 @@
+"""
+Tests for HINDSIGHT_API_LLM_STRICT_SCHEMA / config.llm_strict_schema.
+
+The flag asks every provider for its strongest structured-output mode so weaker
+self-hosted instruction-followers can't wedge retain/consolidation by emitting
+prose preambles, markdown ```json fences, or invalid JSON that fails parsing.
+
+It is resolved once in ``LLMProvider.call`` (OR-ed with the per-call
+``strict_schema`` argument) and passed down, so each provider honours it through
+its existing ``strict_schema`` handling:
+
+- OpenAI-compatible / LiteLLM: ``response_format`` ``json_schema`` with ``strict: true``
+- Gemini: already grammar-enforces its native ``response_schema`` (flag is a no-op)
+- Providers without a strict mode ignore it.
+
+The batch retain path builds its request body directly (bypassing ``call``), so
+it reads the config flag itself.
+"""
+
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, patch
+
+import pytest
+from pydantic import BaseModel
+
+from hindsight_api.config import ENV_LLM_STRICT_SCHEMA, HindsightConfig
+from hindsight_api.engine.llm_wrapper import LLMProvider
+from hindsight_api.engine.providers.openai_compatible_llm import OpenAICompatibleLLM
+
+
+class _Resp(BaseModel):
+    ok: bool
+
+
+def _config_with(strict: bool) -> object:
+    """A config proxy that overrides only llm_strict_schema (avoids recursion)."""
+    from hindsight_api.config import get_config
+
+    real = get_config()
+
+    class _Cfg:
+        llm_strict_schema = strict
+
+        def __getattr__(self, name):
+            return getattr(real, name)
+
+    return _Cfg()
+
+
+# --------------------------------------------------------------------------- #
+# config
+# --------------------------------------------------------------------------- #
+
+
+def test_env_var_enables_strict_schema(monkeypatch):
+    monkeypatch.setenv(ENV_LLM_STRICT_SCHEMA, "true")
+    assert HindsightConfig.from_env().llm_strict_schema is True
+
+
+def test_strict_schema_defaults_off(monkeypatch):
+    monkeypatch.delenv(ENV_LLM_STRICT_SCHEMA, raising=False)
+    assert HindsightConfig.from_env().llm_strict_schema is False
+
+
+# --------------------------------------------------------------------------- #
+# wrapper: resolves the flag for every provider
+# --------------------------------------------------------------------------- #
+
+
+async def _strict_passed_to_provider(*, config_flag: bool, call_arg: bool) -> bool:
+    """Return the strict_schema value the wrapper forwards to the provider impl."""
+    llm = LLMProvider(provider="anthropic", api_key="test-key", base_url="", model="claude-x")
+    impl = SimpleNamespace(call=AsyncMock(return_value=_Resp(ok=True)))
+    llm._provider_impl = impl
+
+    cfg = _config_with(config_flag)  # build before patching to avoid get_config recursion
+    with patch("hindsight_api.config.get_config", lambda: cfg):
+        await llm.call(
+            messages=[{"role": "user", "content": "hi"}],
+            response_format=_Resp,
+            strict_schema=call_arg,
+            max_retries=0,
+        )
+    return impl.call.call_args.kwargs["strict_schema"]
+
+
+@pytest.mark.asyncio
+async def test_wrapper_ors_in_config_flag():
+    # Config flag on, caller didn't ask → provider still gets strict.
+    assert await _strict_passed_to_provider(config_flag=True, call_arg=False) is True
+
+
+@pytest.mark.asyncio
+async def test_wrapper_off_by_default():
+    assert await _strict_passed_to_provider(config_flag=False, call_arg=False) is False
+
+
+@pytest.mark.asyncio
+async def test_wrapper_per_call_override_still_works_with_flag_off():
+    assert await _strict_passed_to_provider(config_flag=False, call_arg=True) is True
+
+
+# --------------------------------------------------------------------------- #
+# openai-compatible: strict_schema -> json_schema, else json_object
+# --------------------------------------------------------------------------- #
+
+
+def _openai_response(content: str = '{"ok": true}'):
+    choice = SimpleNamespace(
+        finish_reason="stop", message=SimpleNamespace(content=content, tool_calls=None, refusal=None)
+    )
+    return SimpleNamespace(error=None, usage=None, choices=[choice])
+
+
+async def _openai_response_format(*, strict: bool):
+    llm = OpenAICompatibleLLM(
+        provider="openai", api_key="test-key", base_url="https://example.test/v1", model="gpt-4o-mini"
+    )
+    create = AsyncMock(return_value=_openai_response())
+    llm._client.chat.completions.create = create
+    with patch("hindsight_api.engine.providers.openai_compatible_llm.get_metrics_collector"):
+        await llm.call(
+            messages=[{"role": "user", "content": "Return whether this worked."}],
+            response_format=_Resp,
+            strict_schema=strict,
+            max_retries=0,
+        )
+    return create.call_args.kwargs.get("response_format")
+
+
+@pytest.mark.asyncio
+async def test_openai_strict_uses_json_schema():
+    rf = await _openai_response_format(strict=True)
+    assert rf is not None and rf["type"] == "json_schema"
+    assert rf["json_schema"]["strict"] is True
+
+
+@pytest.mark.asyncio
+async def test_openai_soft_uses_json_object():
+    rf = await _openai_response_format(strict=False)
+    assert rf is not None and rf["type"] == "json_object"
+
+
+# --------------------------------------------------------------------------- #
+# litellm: strict_schema -> response_format strict flag
+# --------------------------------------------------------------------------- #
+
+
+@pytest.mark.parametrize("strict", [True, False])
+@pytest.mark.asyncio
+async def test_litellm_response_format_strict_follows_arg(strict):
+    from hindsight_api.engine.providers.litellm_llm import LiteLLMLLM
+
+    llm = LiteLLMLLM(provider="litellm", api_key="test-key", base_url="", model="gpt-4o-mini")
+    response = SimpleNamespace(
+        choices=[SimpleNamespace(finish_reason="stop", message=SimpleNamespace(content='{"ok": true}'))],
+        usage=SimpleNamespace(prompt_tokens=1, completion_tokens=1),
+    )
+    acompletion = AsyncMock(return_value=response)
+    with (
+        patch.object(llm, "_acompletion", acompletion),
+        patch("hindsight_api.engine.providers.litellm_llm.get_metrics_collector"),
+    ):
+        await llm.call(
+            messages=[{"role": "user", "content": "Return whether this worked."}],
+            response_format=_Resp,
+            strict_schema=strict,
+            max_retries=0,
+        )
+    rf = acompletion.call_args.kwargs["response_format"]
+    assert rf["type"] == "json_schema"
+    assert rf["json_schema"]["strict"] is strict
+
+
+# --------------------------------------------------------------------------- #
+# batch retain path: reads the config flag directly
+# --------------------------------------------------------------------------- #
+
+
+@pytest.mark.parametrize("strict", [True, False])
+def test_batch_request_body_strict_follows_config(strict):
+    from hindsight_api.engine.retain.fact_extraction import _build_request_body
+
+    llm_config = SimpleNamespace(model="gpt-4o-mini", provider="openai", _provider_impl=SimpleNamespace())
+    config = SimpleNamespace(retain_max_completion_tokens=None, llm_strict_schema=strict)
+    # provider != "openai" service-tier branch skipped via _provider_impl without attr
+    llm_config._provider_impl.openai_service_tier = None
+
+    body = _build_request_body(llm_config, config, "system prompt", "user message", _Resp)
+    assert body["response_format"]["json_schema"]["strict"] is strict
@@ -0,0 +1,545 @@
+"""Tests for per-bank LLM request tracing.
+
+Capture flows through the OTel GenAI recorder: providers call
+``record_llm_call`` on success, and the LLM wrapper forwards failures through
+the same recorder. The DB tracer (``LLMTraceRecorder``) is registered as one of
+those recorders. Covers serialization, record building, the wrapper
+success/error paths, and the HTTP read API (list / stats / tokens).
+"""
+
+import asyncio
+import json
+from datetime import datetime, timezone
+
+import httpx
+import pytest
+import pytest_asyncio
+from pydantic import BaseModel
+
+from hindsight_api import tracing
+from hindsight_api.api import create_app
+from hindsight_api.engine import llm_trace
+from hindsight_api.engine.llm_trace import (
+    LLMRequestRecord,
+    LLMTraceContext,
+    LLMTraceRecorder,
+    _safe_json,
+    current_trace_context,
+    set_trace_context,
+)
+from hindsight_api.engine.llm_wrapper import LLMProvider
+
+# ── serialization helpers ─────────────────────────────────────────────────────
+
+
+def test_safe_json_none_returns_none():
+    assert _safe_json(None, 1000) is None
+
+
+def test_safe_json_handles_datetime_uuid_set_and_pydantic():
+    import uuid
+
+    class Item(BaseModel):
+        name: str
+
+    data = {
+        "when": datetime(2026, 1, 1, tzinfo=timezone.utc),
+        "id": uuid.uuid4(),
+        "tags": {"a", "b"},
+        "model": Item(name="x"),
+        "raw": b"bytes",
+    }
+    out = json.loads(_safe_json(data, 100_000))
+    assert out["when"].startswith("2026-01-01")
+    assert isinstance(out["id"], str)
+    assert sorted(out["tags"]) == ["a", "b"]
+    assert out["model"] == {"name": "x"}
+    assert out["raw"] == "<bytes>"
+
+
+def test_safe_json_truncates_oversized_payload_to_valid_json():
+    big = {"text": "x" * 5000}
+    parsed = json.loads(_safe_json(big, max_chars=200))  # must stay valid JSON
+    assert parsed["_truncated"] is True
+    assert parsed["_original_chars"] > 200
+    assert len(parsed["preview"]) == 200
+
+
+def test_context_var_is_unset_by_default():
+    assert current_trace_context() is None
+
+
+# ── recorder: enable / allowlist ──────────────────────────────────────────────
+
+
+def _recorder(enabled=True, allowed=None):
+    return LLMTraceRecorder(
+        pool_getter=lambda: None,
+        schema_getter=lambda: "public",
+        enabled=enabled,
+        allowed_scopes=allowed or [],
+    )
+
+
+def test_recorder_disabled_is_not_enabled_for_any_scope():
+    assert _recorder(enabled=False).is_enabled("memory") is False
+
+
+def test_recorder_scope_allowlist():
+    r = _recorder(enabled=True, allowed=["reflect"])
+    assert r.is_enabled("reflect") is True
+    assert r.is_enabled("retain_extract_facts") is False
+
+
+# ── recorder: record_llm_call builds correct records ──────────────────────────
+
+
+class _CapturingRecorder(LLMTraceRecorder):
+    """Captures records synchronously instead of writing to the DB."""
+
+    def __init__(self, enabled=True, allowed=None):
+        super().__init__(
+            pool_getter=lambda: None,
+            schema_getter=lambda: "public",
+            enabled=enabled,
+            allowed_scopes=allowed or [],
+        )
+        self.records: list[LLMRequestRecord] = []
+
+    def _record_fire_and_forget(self, record: LLMRequestRecord) -> None:
+        self.records.append(record)
+
+
+def test_record_llm_call_success_with_context_and_tokens():
+    rec = _CapturingRecorder()
+    token = set_trace_context(LLMTraceContext(bank_id="bank-x", operation="retain", metadata={"k": "v"}))
+    try:
+        rec.record_llm_call(
+            provider="gemini",
+            model="gemini-2.5-flash-lite",
+            scope="retain_extract_facts",
+            messages=[{"role": "user", "content": "hi"}],
+            response_content="some output",
+            input_tokens=100,
+            output_tokens=20,
+            duration=0.5,
+            finish_reason="stop",
+            cached_tokens=40,
+        )
+    finally:
+        llm_trace.reset_trace_context(token)
+
+    assert len(rec.records) == 1
+    r = rec.records[0]
+    assert r.status == "success"
+    assert r.bank_id == "bank-x"
+    assert r.operation == "retain"
+    assert r.metadata == {"k": "v"}
+    assert r.input_tokens == 100
+    assert r.output_tokens == 20
+    assert r.cached_tokens == 40
+    assert r.total_tokens == 120
+    assert r.output == "some output"
+    assert r.llm_info["finish_reason"] == "stop"
+
+
+def test_record_llm_call_error_record():
+    rec = _CapturingRecorder()
+    rec.record_llm_call(
+        provider="mock",
+        model="mock",
+        scope="memory",
+        messages=[],
+        error=RuntimeError("boom"),
+        duration=0.1,
+    )
+    assert len(rec.records) == 1
+    r = rec.records[0]
+    assert r.status == "error"
+    assert r.error == "RuntimeError: boom"
+    assert r.output is None
+    assert r.bank_id is None
+
+
+def test_record_llm_call_disabled_scope_records_nothing():
+    rec = _CapturingRecorder(allowed=["reflect"])
+    rec.record_llm_call(provider="mock", model="mock", scope="memory", messages=[], duration=0.1)
+    assert rec.records == []
+
+
+# ── wrapper: success via provider, error forwarded ────────────────────────────
+
+
+@pytest.fixture
+def registered_recorder():
+    """Register a capturing recorder with the GenAI registry for one test."""
+    rec = _CapturingRecorder()
+    tracing.register_span_recorder(rec)
+    yield rec
+    tracing.unregister_span_recorder(rec)
+
+
+@pytest.mark.asyncio
+async def test_wrapper_success_recorded_by_provider(registered_recorder):
+    llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+    result = await llm.call(messages=[{"role": "user", "content": "hello"}], scope="memory")
+
+    assert result == "mock response"
+    assert len(registered_recorder.records) == 1
+    r = registered_recorder.records[0]
+    assert r.status == "success"
+    assert r.provider == "mock"
+    assert r.scope == "memory"
+
+
+@pytest.mark.asyncio
+async def test_wrapper_error_forwarded_and_reraised(registered_recorder):
+    llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+    llm._provider_impl.set_mock_exception(RuntimeError("kaboom"))
+
+    with pytest.raises(RuntimeError, match="kaboom"):
+        await llm.call(messages=[{"role": "user", "content": "x"}], scope="memory")
+
+    assert len(registered_recorder.records) == 1
+    r = registered_recorder.records[0]
+    assert r.status == "error"
+    assert "kaboom" in r.error
+
+
+@pytest.mark.asyncio
+async def test_configured_provider_binds_bank_context(registered_recorder):
+    llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+    class _Cfg:
+        llm_gemini_safety_settings = None
+
+    configured = llm.with_config(_Cfg(), bank_id="bank-42", operation="reflect")
+    await configured.call(messages=[{"role": "user", "content": "x"}], scope="memory")
+
+    assert len(registered_recorder.records) == 1
+    assert registered_recorder.records[0].bank_id == "bank-42"
+    assert registered_recorder.records[0].operation == "reflect"
+    assert current_trace_context() is None  # unwound after the call
+
+
+# ── HTTP read API (integration) ───────────────────────────────────────────────
+
+
+@pytest_asyncio.fixture
+async def trace_api_client(memory):
+    """Test client with LLM tracing enabled on the engine's recorder."""
+    memory._llm_recorder._enabled = True
+    memory._llm_recorder._allowed_scopes = None  # All scopes
+
+    app = create_app(memory, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        yield client
+
+    memory._llm_recorder._enabled = False
+
+
+@pytest.fixture
+def bank_id():
+    return f"llm_trace_test_{datetime.now().timestamp()}"
+
+
+@pytest.mark.asyncio
+async def test_list_empty(trace_api_client, bank_id):
+    await trace_api_client.put(f"/v1/default/banks/{bank_id}", json={"name": "Trace Bank"})
+    response = await trace_api_client.get(f"/v1/default/banks/{bank_id}/llm-requests")
+    assert response.status_code == 200
+    data = response.json()
+    assert data["items"] == []
+    assert data["total"] == 0
+
+
+@pytest.mark.asyncio
+async def test_retain_creates_trace_rows_with_tokens(trace_api_client, bank_id):
+    await trace_api_client.put(f"/v1/default/banks/{bank_id}", json={"name": "Trace Bank"})
+    response = await trace_api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": [{"content": "Alice likes cats", "context": "preferences"}]},
+    )
+    assert response.status_code == 200
+    # retain triggers fact extraction + consolidation (2 LLM calls → 2 trace
+    # writes); give the fire-and-forget tasks room under parallel test load.
+    await asyncio.sleep(1.5)
+
+    response = await trace_api_client.get(f"/v1/default/banks/{bank_id}/llm-requests")
+    assert response.status_code == 200
+    data = response.json()
+    assert data["total"] >= 1
+
+    operations = {item["operation"] for item in data["items"]}
+    assert "retain" in operations, f"Expected a 'retain' trace, got: {operations}"
+
+    retain_entry = next(it for it in data["items"] if it["operation"] == "retain")
+    assert retain_entry["status"] == "success"
+    assert retain_entry["provider"] == "mock"
+    assert retain_entry["bank_id"] == bank_id
+    # The mock provider reports token usage via record_llm_call.
+    assert retain_entry["input_tokens"] is not None
+    assert retain_entry["total_tokens"] is not None
+    # Requested params are captured (retain sets retain_max_completion_tokens=64000).
+    assert retain_entry["llm_info"]["request"]["max_completion_tokens"] == 64000
+
+    # The retain extraction call is attributed to its document, and the
+    # document_id filter returns that run's calls.
+    doc_id = retain_entry["metadata"]["document_id"]
+    assert doc_id
+    by_doc = (
+        await trace_api_client.get(f"/v1/default/banks/{bank_id}/llm-requests", params={"document_id": doc_id})
+    ).json()
+    assert by_doc["total"] >= 1
+    assert all(it["metadata"].get("document_id") == doc_id for it in by_doc["items"])
+
+    entry = data["items"][0]
+
+    # OTel-style grouping: every row carries trace ids, and each operation
+    # invocation (retain vs consolidation) gets its own trace_id, while a call's
+    # parent_span_id is its operation span.
+    by_op = {item["operation"]: item for item in data["items"]}
+    for item in data["items"]:
+        assert item["trace_id"] and item["span_id"] and item["parent_span_id"]
+    if "retain" in by_op and "consolidation" in by_op:
+        assert by_op["retain"]["trace_id"] != by_op["consolidation"]["trace_id"]
+
+    # Filtering by a trace_id returns only that operation run's calls.
+    a_trace = entry["trace_id"]
+    resp = await trace_api_client.get(
+        f"/v1/default/banks/{bank_id}/llm-requests", params={"trace_id": a_trace}
+    )
+    assert resp.status_code == 200
+    filtered = resp.json()
+    assert filtered["total"] >= 1
+    assert all(it["trace_id"] == a_trace for it in filtered["items"])
+
+    # group=true paginates by run: total counts distinct runs (here retain +
+    # consolidation = 2), and every traced row is still returned.
+    resp = await trace_api_client.get(f"/v1/default/banks/{bank_id}/llm-requests", params={"group": "true"})
+    assert resp.status_code == 200
+    grouped = resp.json()
+    distinct_traces = {it["trace_id"] for it in data["items"]}
+    assert grouped["total"] == len(distinct_traces)
+    assert len(grouped["items"]) >= len(data["items"])
+
+
+@pytest.mark.asyncio
+async def test_delta_reretain_binds_document_id(trace_api_client, bank_id):
+    """A delta re-retain (editing/appending a document) must tag its trace with
+    the document_id, so the document accrues one trace per retain — not just the
+    initial full retain. Regression: the delta path used a second extraction call
+    site that bypassed the document_id attribution, so edits were orphaned.
+    """
+    await trace_api_client.put(f"/v1/default/banks/{bank_id}", json={"name": "Trace Bank"})
+    document_id = "delta-doc-001"
+
+    # v1: a single self-contained chunk.
+    v1 = "Alice is a software engineer at Google. She works on search infrastructure."
+    resp = await trace_api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": [{"content": v1, "context": "people", "document_id": document_id}]},
+    )
+    assert resp.status_code == 200
+    await asyncio.sleep(1.5)
+
+    # v2: original content preserved + a new paragraph. The unchanged first chunk
+    # forces the delta path (it needs ≥1 unchanged chunk), and the new chunk is
+    # extracted via the instrumented delta extraction call site.
+    v2 = v1 + "\n\nBob joined Google as a product manager in 2024. He previously worked at Meta."
+    resp = await trace_api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": [{"content": v2, "context": "people", "document_id": document_id}]},
+    )
+    assert resp.status_code == 200
+    await asyncio.sleep(1.5)
+
+    # The document_id filter must now return both the full retain and the delta
+    # re-retain — i.e. ≥2 distinct retain trace_ids, every row tagged.
+    by_doc = (
+        await trace_api_client.get(
+            f"/v1/default/banks/{bank_id}/llm-requests",
+            params={"document_id": document_id, "operation": "retain"},
+        )
+    ).json()
+    assert all(it["metadata"].get("document_id") == document_id for it in by_doc["items"])
+    retain_traces = {it["trace_id"] for it in by_doc["items"]}
+    assert len(retain_traces) >= 2, f"expected full + delta retain bound to document, got {len(retain_traces)}"
+
+
+@pytest.mark.asyncio
+async def test_memory_ids_mapped_to_retain_and_consolidation(trace_api_client, bank_id):
+    """A retain trace maps to the facts it created; a consolidation trace maps to
+    the source memories it consumed (and any observations it produced). These are
+    attached to every row of the trace after the operation completes.
+    """
+    await trace_api_client.put(f"/v1/default/banks/{bank_id}", json={"name": "Trace Bank"})
+    resp = await trace_api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": [{"content": "Alice works at Google as a senior engineer.", "context": "people"}]},
+    )
+    assert resp.status_code == 200
+    # retain + the consolidation it triggers are fire-and-forget; give the trace
+    # writes and the post-operation memory_id UPDATE room under parallel load.
+    await asyncio.sleep(2.0)
+
+    data = (await trace_api_client.get(f"/v1/default/banks/{bank_id}/llm-requests")).json()
+    by_op = {item["operation"]: item for item in data["items"]}
+
+    # Retain maps to the created fact unit_ids (outputs).
+    retain = by_op["retain"]
+    created = retain["metadata"].get("memory_ids")
+    assert created, f"retain trace should map to created facts, metadata={retain['metadata']}"
+
+    # Consolidation maps to the source memories consumed (inputs). The retained
+    # fact is what gets consolidated, so it appears among the sources.
+    if "consolidation" in by_op:
+        sources = by_op["consolidation"]["metadata"].get("source_memory_ids")
+        assert sources, "consolidation trace should map to the source memories it consumed"
+
+    # Reverse lookup: ?memory_id=<created fact> returns every run touching it —
+    # the retain that produced it (memory_ids) and any consolidation that consumed
+    # it as a source (source_memory_ids).
+    by_mem = (
+        await trace_api_client.get(
+            f"/v1/default/banks/{bank_id}/llm-requests", params={"memory_id": created[0]}
+        )
+    ).json()
+    assert by_mem["total"] >= 1
+    for it in by_mem["items"]:
+        meta = it["metadata"]
+        assert created[0] in (meta.get("memory_ids") or []) or created[0] in (meta.get("source_memory_ids") or [])
+    assert retain["trace_id"] in {it["trace_id"] for it in by_mem["items"]}, "producing retain trace must be returned"
+
+
+@pytest.mark.asyncio
+async def test_filter_by_status_and_operation(trace_api_client, bank_id):
+    await trace_api_client.put(f"/v1/default/banks/{bank_id}", json={"name": "Trace Bank"})
+    await trace_api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": [{"content": "test content", "context": "test"}]},
+    )
+    await asyncio.sleep(1.0)
+
+    response = await trace_api_client.get(
+        f"/v1/default/banks/{bank_id}/llm-requests",
+        params={"status": "success", "operation": "retain"},
+    )
+    assert response.status_code == 200
+    data = response.json()
+    assert data["total"] >= 1
+    for item in data["items"]:
+        assert item["status"] == "success"
+        assert item["operation"] == "retain"
+
+    response = await trace_api_client.get(
+        f"/v1/default/banks/{bank_id}/llm-requests", params={"status": "error"}
+    )
+    assert response.json()["total"] == 0
+
+
+@pytest.mark.asyncio
+async def test_stats_endpoint_includes_tokens(trace_api_client, bank_id):
+    await trace_api_client.put(f"/v1/default/banks/{bank_id}", json={"name": "Trace Bank"})
+    await trace_api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": [{"content": "stats content", "context": "test"}]},
+    )
+    await asyncio.sleep(1.0)
+
+    response = await trace_api_client.get(
+        f"/v1/default/banks/{bank_id}/llm-requests/stats", params={"period": "1d"}
+    )
+    assert response.status_code == 200
+    data = response.json()
+    assert data["trunc"] == "day"
+    assert len(data["buckets"]) >= 1
+    bucket = data["buckets"][0]
+    assert "statuses" in bucket
+    assert "tokens" in bucket
+    assert set(bucket["tokens"].keys()) == {"input", "output", "cached", "total"}
+    assert bucket["total"] >= 1
+
+
+@pytest.mark.asyncio
+async def test_disabled_writes_no_rows(memory):
+    memory._llm_recorder._enabled = False
+
+    app = create_app(memory, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        bid = f"llm_trace_disabled_{datetime.now().timestamp()}"
+        await client.put(f"/v1/default/banks/{bid}", json={"name": "No Trace"})
+        await client.post(
+            f"/v1/default/banks/{bid}/memories",
+            json={"items": [{"content": "nope", "context": "x"}]},
+        )
+        await asyncio.sleep(0.5)
+
+        response = await client.get(f"/v1/default/banks/{bid}/llm-requests")
+        assert response.status_code == 200
+        assert response.json()["total"] == 0
+
+
+# ── real-LLM acceptance (provider matrix) ─────────────────────────────────────
+
+
+@pytest.mark.hs_llm_mat
+@pytest.mark.asyncio
+async def test_real_llm_retain_and_consolidation_traced(memory_real_llm):
+    """A real retain produces traced retain *and* consolidation calls with real
+    token usage. Runs across providers in the hs_llm_mat matrix to confirm the
+    GenAI record_llm_call path reports tokens for each provider."""
+    memory_real_llm._llm_recorder._enabled = True
+    memory_real_llm._llm_recorder._allowed_scopes = None  # All scopes
+
+    app = create_app(memory_real_llm, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test", timeout=120.0) as client:
+        bank_id = f"llm_trace_real_{datetime.now().timestamp()}"
+        await client.put(f"/v1/default/banks/{bank_id}", json={"name": "Real Trace"})
+
+        resp = await client.post(
+            f"/v1/default/banks/{bank_id}/memories",
+            json={
+                "items": [
+                    {
+                        "content": "Alice is a data engineer from Turin who loves hiking in the Dolomites.",
+                        "context": "profile",
+                    }
+                ]
+            },
+        )
+        assert resp.status_code == 200
+
+        # Real LLM latency + fire-and-forget writes: poll until both operations
+        # land (retain fact-extraction, then consolidation over the new facts).
+        by_op: dict[str, list[dict]] = {}
+        for _ in range(30):
+            await asyncio.sleep(1.0)
+            data = (await client.get(f"/v1/default/banks/{bank_id}/llm-requests?limit=50")).json()
+            by_op = {}
+            for item in data["items"]:
+                by_op.setdefault(item["operation"], []).append(item)
+            if "retain" in by_op and "consolidation" in by_op:
+                break
+
+        assert "retain" in by_op, f"expected a retain trace, got operations: {list(by_op)}"
+        assert "consolidation" in by_op, f"expected a consolidation trace, got operations: {list(by_op)}"
+
+        for operation in ("retain", "consolidation"):
+            entry = by_op[operation][0]
+            assert entry["status"] == "success", f"{operation} trace not successful: {entry}"
+            assert entry["provider"] == memory_real_llm._llm_config.provider
+            assert entry["input_tokens"] and entry["input_tokens"] > 0, f"no input tokens for {operation}"
+            assert entry["output_tokens"] and entry["output_tokens"] > 0, f"no output tokens for {operation}"
+            assert entry["total_tokens"] == entry["input_tokens"] + entry["output_tokens"]
+            assert entry["input"] is not None  # the prompt messages were captured
+
+        # The operations map to the memory_units they touched: retain to the
+        # facts it created, consolidation to the source memories it consumed.
+        assert by_op["retain"][0]["metadata"].get("memory_ids"), "retain trace missing created memory_ids"
+        assert by_op["consolidation"][0]["metadata"].get("source_memory_ids"), (
+            "consolidation trace missing source_memory_ids"
+        )
@@ -81,7 +81,9 @@ def _make_lmstudio_llm() -> OpenAICompatibleLLM:
    )


-def _lmstudio_400_error(msg: str = "Tool choice of type 'function' is not supported. Use 'auto', 'none', or 'required'.") -> APIStatusError:
+def _lmstudio_400_error(
+    msg: str = "Tool choice of type 'function' is not supported. Use 'auto', 'none', or 'required'.",
+) -> APIStatusError:
    """Simulate the HTTP 400 LM Studio returns for unsupported tool_choice format."""
    mock_response = MagicMock()
    mock_response.status_code = 400
@@ -151,7 +153,11 @@ class TestLMStudioNamedToolChoiceBug:
        assert result.tool_calls[0].name == "search_mental_models"

        sent_kwargs = mock_create.call_args.kwargs
-        assert sent_kwargs["tool_choice"] == "required"
+        # The named dict is normalized to "required" + a single filtered tool,
+        # then "required" is downgraded to auto (omitted) because LM Studio
+        # silently drops it (#1563/#1179/#1877). The single filtered tool keeps
+        # the call forced in practice. See test_tool_choice_required_downgrade.py.
+        assert "tool_choice" not in sent_kwargs
        assert len(sent_kwargs["tools"]) == 1
        assert sent_kwargs["tools"][0]["function"]["name"] == "search_mental_models"

@@ -182,11 +188,12 @@ class TestLMStudioNamedToolChoiceBug:
            assert exc_info.value.status_code == 400

    @pytest.mark.asyncio
-    async def test_lmstudio_string_tool_choice_works_fine(self):
+    async def test_lmstudio_string_required_is_downgraded(self):
        """
-        String tool_choice values ("auto", "none", "required") ARE supported by LM Studio.
-        Only the dict format {"type": "function", "function": {"name": "..."}} fails.
-        This test confirms the control case works.
+        LM Studio does not honour the string ``tool_choice="required"`` either:
+        it silently returns an empty tool_calls array (#1563/#1179/#1877). So
+        "required" is downgraded to auto (omitted) for LM Studio. See the
+        dedicated suite in test_tool_choice_required_downgrade.py.
        """
        llm = _make_lmstudio_llm()
        success_response = _make_tool_call_response("search_mental_models", {"query": "user name"})
@@ -197,16 +204,16 @@ class TestLMStudioNamedToolChoiceBug:
            result = await llm.call_with_tools(
                messages=[{"role": "user", "content": "What is the user's name?"}],
                tools=REFLECT_TOOLS,
-                tool_choice="required",  # string form — LM Studio accepts this
+                tool_choice="required",
                max_retries=0,
            )

        assert len(result.tool_calls) == 1
        assert result.tool_calls[0].name == "search_mental_models"

-        # Confirm "required" was sent, not a dict
+        # "required" is omitted (downgraded to auto) rather than sent verbatim.
        sent_kwargs = mock_create.call_args.kwargs
-        assert sent_kwargs["tool_choice"] == "required"
+        assert "tool_choice" not in sent_kwargs


 class TestExpectedFixBehavior:
@@ -225,10 +232,11 @@ class TestExpectedFixBehavior:
    """

    @pytest.mark.asyncio
-    async def test_fix_converts_named_tool_choice_to_required(self):
+    async def test_fix_converts_named_tool_choice_and_downgrades(self):
        """
-        After fix: named tool_choice dict is converted to "required" for lmstudio.
-        The API receives tool_choice="required" instead of the unsupported dict.
+        Named tool_choice dict → normalized to "required" + a single filtered
+        tool → "required" downgraded to auto (omitted) for lmstudio. The API
+        never sees the unsupported dict nor the silently-dropped "required".
        """
        llm = _make_lmstudio_llm()
        named_tool_choice = {"type": "function", "function": {"name": "search_mental_models"}}
@@ -248,11 +256,9 @@ class TestExpectedFixBehavior:
        assert result.tool_calls[0].name == "search_mental_models"

        sent_kwargs = mock_create.call_args.kwargs
-        # Fix: dict was converted to "required"
-        assert sent_kwargs["tool_choice"] == "required", (
-            f"Expected tool_choice='required', got {sent_kwargs['tool_choice']!r}"
-        )
-        # Fix: tools filtered to just the requested one
+        # Fix: dict was normalized then "required" downgraded to auto (omitted)
+        assert "tool_choice" not in sent_kwargs
+        # Fix: tools filtered to just the requested one (keeps the call forced)
        assert len(sent_kwargs["tools"]) == 1
        assert sent_kwargs["tools"][0]["function"]["name"] == "search_mental_models"

@@ -281,7 +287,9 @@ class TestExpectedFixBehavior:
            )

        sent_kwargs = mock_create.call_args.kwargs
-        assert sent_kwargs["tool_choice"] == "required"
+        # "required" is downgraded to auto (omitted) for lmstudio; the single
+        # filtered tool keeps the call forced.
+        assert "tool_choice" not in sent_kwargs
        assert len(sent_kwargs["tools"]) == 1
        assert sent_kwargs["tools"][0]["function"]["name"] == forced_tool_name

@@ -290,7 +298,9 @@ class TestExpectedFixBehavior:
        """
        The fix is generalized: all providers convert named tool_choice to
        "required" + filtered tools.  OpenAI natively supports the dict format
-        too, so the behaviour is semantically identical either way.
+        too, so the behaviour is semantically identical either way. The real
+        OpenAI API (no base_url override) honours "required", so unlike the
+        self-hosted providers it is NOT downgraded.
        """
        from hindsight_api.engine.providers.openai_compatible_llm import OpenAICompatibleLLM

@@ -0,0 +1,130 @@
+"""Regression tests for DataAccessOps.lock_document_for_write.
+
+Covers vectorize-io/hindsight#1944: the retain document-ownership gate used a
+single ``INSERT ... ON CONFLICT DO UPDATE ... RETURNING`` upsert. PostgreSQL
+runs it as-is, but the Oracle adapter rewrites ``ON CONFLICT DO UPDATE`` to a
+``MERGE``, which cannot carry a ``RETURNING`` clause — so every retain 500'd
+with ``DPY-1003: the executed statement does not return rows``.
+
+The lock-and-read step now lives behind ``ops.lock_document_for_write`` so each
+backend implements it natively (PG: one upsert; Oracle: idempotent insert +
+``SELECT ... FOR UPDATE``). These tests pin the contract on PG (run in the
+default suite) and the Oracle rewrite limitation that motivated the split.
+"""
+
+from datetime import datetime, timezone
+
+import pytest
+
+from hindsight_api.engine.memory_engine import fq_table
+
+
+def _ts() -> float:
+    return datetime.now(timezone.utc).timestamp()
+
+
+async def _seed_bank(conn, bank_id: str) -> None:
+    await conn.execute(
+        "INSERT INTO banks (bank_id, name) VALUES ($1, $2) ON CONFLICT DO NOTHING",
+        bank_id,
+        bank_id,
+    )
+
+
+@pytest.mark.asyncio
+async def test_lock_document_for_write_returns_pending_then_existing_hash(memory):
+    """Fresh row → '__pending__'; existing row → its stored hash."""
+    bank_id = f"test_lock_doc_{_ts()}"
+    doc_id = "doc-lock-regression"
+    documents = fq_table("documents")
+
+    backend = await memory._get_backend()
+    ops = backend.ops
+    async with backend.acquire() as conn:
+        await _seed_bank(conn, bank_id)
+
+        # First call creates the row and reports the placeholder hash.
+        async with conn.transaction():
+            first = await ops.lock_document_for_write(conn, documents, doc_id, bank_id)
+        assert first == "__pending__"
+
+        # Promote the placeholder to a real content hash, as the real retain
+        # flow does immediately after taking the lock.
+        await conn.execute(
+            f"UPDATE {documents} SET content_hash = $1 WHERE id = $2 AND bank_id = $3",
+            "real-hash-v1",
+            doc_id,
+            bank_id,
+        )
+
+        # A subsequent writer sees the committed hash (and re-takes the lock).
+        async with conn.transaction():
+            second = await ops.lock_document_for_write(conn, documents, doc_id, bank_id)
+        assert second == "real-hash-v1"
+
+
+@pytest.mark.asyncio
+async def test_lock_document_for_write_isolates_by_bank(memory):
+    """The lock/read is scoped to (id, bank_id) — same doc_id in another bank
+    is a distinct row and starts at '__pending__'."""
+    suffix = _ts()
+    bank_a = f"test_lock_doc_a_{suffix}"
+    bank_b = f"test_lock_doc_b_{suffix}"
+    doc_id = "shared-doc-id"
+    documents = fq_table("documents")
+
+    backend = await memory._get_backend()
+    ops = backend.ops
+    async with backend.acquire() as conn:
+        await _seed_bank(conn, bank_a)
+        await _seed_bank(conn, bank_b)
+
+        async with conn.transaction():
+            assert await ops.lock_document_for_write(conn, documents, doc_id, bank_a) == "__pending__"
+        await conn.execute(
+            f"UPDATE {documents} SET content_hash = $1 WHERE id = $2 AND bank_id = $3",
+            "bank-a-hash",
+            doc_id,
+            bank_a,
+        )
+
+        async with conn.transaction():
+            assert await ops.lock_document_for_write(conn, documents, doc_id, bank_b) == "__pending__"
+
+
+def test_oracle_merge_rewrite_cannot_carry_returning():
+    """Root cause of #1944: PG's single-statement upsert-and-lock rewrites to an
+    Oracle MERGE, and MERGE can't RETURNING — so a ``fetchval`` on it gets no
+    rows back (DPY-1003). The dialect split in lock_document_for_write exists to
+    avoid emitting this form on Oracle."""
+    from hindsight_api.engine.db.oracle import _rewrite_pg_to_oracle
+
+    query, ignore_dup, returning_cols = _rewrite_pg_to_oracle(
+        "INSERT INTO documents (id, bank_id, original_text, content_hash) "
+        "VALUES ($1, $2, '', '__pending__') "
+        "ON CONFLICT (id, bank_id) DO UPDATE SET content_hash = documents.content_hash "
+        "RETURNING content_hash"
+    )
+
+    assert query.lstrip().upper().startswith("MERGE")
+    # The RETURNING clause is silently dropped by the MERGE rewrite — this is
+    # exactly why the old single-statement form returned no rows on Oracle.
+    assert "RETURNING" not in query.upper()
+    assert returning_cols is None
+    assert not ignore_dup
+
+
+def test_oracle_select_for_update_hash_translates_cleanly():
+    """The Oracle fallback reads the hash with a plain SELECT ... FOR UPDATE,
+    which translates without a MERGE so the scalar fetch returns the column."""
+    from hindsight_api.engine.db.oracle import _rewrite_pg_to_oracle
+
+    query, ignore_dup, returning_cols = _rewrite_pg_to_oracle(
+        "SELECT content_hash FROM documents WHERE id = $1 AND bank_id = $2 FOR UPDATE"
+    )
+
+    assert "MERGE" not in query.upper()
+    assert "FOR UPDATE" in query.upper()
+    assert ":1" in query and ":2" in query
+    assert not ignore_dup
+    assert returning_cols is None
--- a/Show More
+++ b/Show More