blog: co-brand Copilot cover with the official GitHub Copilot mark

Add the GitHub Copilot logo (top-left lockup + terminal title bar) so the
cover reads as a Copilot x Hindsight co-brand.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.claude/skills/code-review/SKILL.md

@@ -159,6 +159,18 @@ If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
 - Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
 - Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
 
+### 7a. Check TS/Python wrapper-client parity
+
+Two of the generated SDKs ship a **hand-written, maintained convenience wrapper** on top of the auto-generated low-level client — and *only* these two:
+- **TypeScript**: `hindsight-clients/typescript/src/index.ts` (`HindsightClient`)
+- **Python**: `hindsight-clients/python/hindsight_client/hindsight_client.py` (`Hindsight`)
+
+(The Rust/Go/etc. clients are generated-only — no wrapper to keep in sync.)
+
+These wrappers are what most third-party consumers actually call, and they must expose the same surface. **If a change touches one wrapper's method — adds/removes a parameter, changes a default, forwards a new query/body field — the equivalent method in the *other* wrapper must get the same change in the same (or an immediately-following) PR.** A parameter that exists in the generated SDK but is dropped by one wrapper silently strips it for every consumer of that language (this is exactly what #2975 / #3042 fixed for `detail`/`tags_match`/`limit`/`offset` on `listMentalModels`/`getMentalModel`). **Should fix** — flag any wrapper method that gains capabilities in one language but not the other, and add a matching mapping regression test on both sides.
+
+Note: the `client-coverage-check` CI tool only validates **request-body** fields, not GET **query** parameters — so query-param parity gaps are *not* caught automatically and must be checked by hand here.
+
 ### 7b. Check API-layer data-access boundary
 
 For each changed handler in `hindsight-api-slim/hindsight_api/api/` (e.g. `http.py`, `mcp.py`):

.env.example

@@ -31,6 +31,10 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # HINDSIGHT_API_LLM_STRICT_SCHEMA_REFLECT=true
 # HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION=true
 
+# Some backends, including Bedrock Converse, reject JSON Schema maxItems.
+# Disable it only for those backends; consolidation still enforces the cap.
+# HINDSIGHT_API_LLM_SUPPORTS_MAX_ITEMS=true
+
 # Diagnostic: on any LLM 4xx, log the exact assembled request ([LLM_4XX_DUMP]) --
 # serialized request config (message bodies stripped) + capped per-message previews.
 # For debugging otherwise-unreproducible rejected calls. Off by default.
@@ -108,6 +112,12 @@ HINDSIGHT_API_LOG_LEVEL=info
 # 'failed' (not 'completed'), surfacing silently-dropped facts. Default false.
 # HINDSIGHT_API_FAIL_ON_EXTRACTION_ERRORS=false
 
+# Wall-clock ceiling (seconds) for one retain task in the worker. A retain that
+# blocks indefinitely is cancelled and marked 'failed' — and so becomes
+# retryable — instead of holding its worker slot until the process restarts.
+# Set well above your slowest healthy retain; 0 disables. Default 3600.
+# HINDSIGHT_API_RETAIN_WALL_TIMEOUT=3600
+
 # Dry-run extraction preview endpoint (POST /memories/dry-run-extract). Enabled by default; it makes
 # a real LLM call but stores nothing. Set to false to remove the endpoint (returns 404).
 # HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT=true
@@ -171,6 +181,9 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=BAAI/bge-small-en-v1.5
 # Force CPU if local embeddings hit MPS/XPC instability on macOS:
 # HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=false
+# Opt in to the Apple Silicon MPS GPU (off by default: MPS leaks memory under
+# variable-length workloads). CUDA/XPU still auto-select regardless:
+# HINDSIGHT_API_EMBEDDINGS_LOCAL_ALLOW_MPS=false
 # For ONNX provider (local CPU embeddings without an Ollama/TEI sidecar):
 # HINDSIGHT_API_EMBEDDINGS_PROVIDER=onnx
 # HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_ID=intfloat/multilingual-e5-small
@@ -207,6 +220,15 @@ HINDSIGHT_API_LOG_LEVEL=info
 # DeepSeek note: DeepSeek is supported for LLM calls, but not for embeddings.
 # If using DeepSeek as LLM provider, keep embeddings on local/openai/cohere/google/etc.
 
+# Embedding similarity thresholds. These defaults preserve the behavior calibrated
+# for BAAI/bge-small-en-v1.5. Recalibrate each threshold independently when changing
+# embedding models because cosine-similarity distributions are model-dependent.
+# HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY=0.3
+# HINDSIGHT_API_GRAPH_SEED_MIN_SIMILARITY=0.3
+# HINDSIGHT_API_TEMPORAL_SEMANTIC_MIN_SIMILARITY=0.1
+# HINDSIGHT_API_SEMANTIC_LINK_MIN_SIMILARITY=0.7
+# HINDSIGHT_API_CONSOLIDATION_DEDUP_THRESHOLD=0.97
+
 # Reranker Configuration (Optional - uses local by default)
 # Provider: "local" (default) or "tei" (HuggingFace Text Embeddings Inference)
 # HINDSIGHT_API_RERANKER_PROVIDER=local
@@ -217,6 +239,9 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_RERANKER_LOCAL_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
 # Force CPU if the local reranker hits MPS/XPC instability on macOS:
 # HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=false
+# Opt in to the Apple Silicon MPS GPU (off by default: MPS leaks memory under
+# variable-length workloads). CUDA/XPU still auto-select regardless:
+# HINDSIGHT_API_RERANKER_LOCAL_ALLOW_MPS=false
 # For TEI provider:
 # HINDSIGHT_API_RERANKER_TEI_URL=http://localhost:8081
 
@@ -238,6 +263,16 @@ HINDSIGHT_API_LOG_LEVEL=info
 # Expose async-operation queue + consolidation-backlog gauges on /metrics.
 # Runs periodic per-schema COUNT queries on a background task (disabled by default).
 # HINDSIGHT_API_METRICS_BACKLOG_ENABLED=true
+#
+# Runtime-stall observability (enabled by default). When a liveness probe fails,
+# these tell you WHY: a blocked event loop vs DB connection-pool exhaustion.
+# The loop watchdog logs the offending stack when the loop is unresponsive; the
+# DB-pool acquire timing logs (and exposes hindsight.db.pool.waiting) when
+# callers queue for a connection. Both are cheap; tune or disable if needed.
+# HINDSIGHT_API_LOOP_WATCHDOG_ENABLED=false
+# HINDSIGHT_API_LOOP_WATCHDOG_STALL_THRESHOLD_MS=1000
+# HINDSIGHT_API_LOOP_WATCHDOG_POLL_INTERVAL_MS=250
+# HINDSIGHT_API_DB_ACQUIRE_WARN_THRESHOLD_MS=1000
 
 # -----------------------------------------------------------------------------
 # Control Plane (Optional)

.github/workflows/test.yml

@@ -42,6 +42,7 @@ jobs:
       integrations-continue: ${{ steps.filter.outputs.integrations-continue }}
       integrations-cursor-cli: ${{ steps.filter.outputs.integrations-cursor-cli }}
       integrations-zcode: ${{ steps.filter.outputs.integrations-zcode }}
+      integrations-copilot-cli: ${{ steps.filter.outputs.integrations-copilot-cli }}
       integrations-crewai: ${{ steps.filter.outputs.integrations-crewai }}
       integrations-litellm: ${{ steps.filter.outputs.integrations-litellm }}
       integrations-pydantic-ai: ${{ steps.filter.outputs.integrations-pydantic-ai }}
@@ -153,6 +154,8 @@ jobs:
             - 'hindsight-integrations/continue/**'
           integrations-cursor-cli:
             - 'hindsight-integrations/cursor-cli/**'
+          integrations-copilot-cli:
+            - 'hindsight-integrations/copilot-cli/**'
           integrations-crewai:
             - 'hindsight-integrations/crewai/**'
           integrations-litellm:
@@ -286,6 +289,18 @@ jobs:
       working-directory: ./hindsight-api-slim
       run: uv build
 
+    # `uv build` only packages the source; it does not prove the dependency set
+    # resolves or that the code imports on this interpreter. Install into a fresh
+    # env and run a byte-compile + import smoke test so the matrix actually
+    # exercises each Python version (notably 3.14).
+    - name: Install and smoke-test on Python ${{ matrix.python-version }}
+      working-directory: ./hindsight-api-slim
+      run: |
+        uv venv --python ${{ matrix.python-version }} .venv-smoke
+        VIRTUAL_ENV=.venv-smoke uv pip install .
+        .venv-smoke/bin/python -m compileall -q hindsight_api
+        .venv-smoke/bin/python -c "import hindsight_api, hindsight_api.main, hindsight_api.config; from hindsight_api.engine import memory_engine, llm_wrapper; print('import OK')"
+
   build-typescript-client:
     needs: [detect-changes]
     if: >-
@@ -1871,6 +1886,27 @@ jobs:
       with:
         ref: ${{ github.event.pull_request.head.sha || '' }}
 
+    # The Oracle 23ai `free` service image is large; combined with the Python ML
+    # deps (torch) it exhausts the runner's ~14 GB root disk, so uv fails to
+    # extract a wheel with "No space left on device (os error 28)".
+    # Only the cheap, high-yield reclaims are enabled — the Android/.NET/Haskell
+    # dirs plus swap are a few `rm -rf`s worth ~16-21 GB, which is ample headroom:
+    #   - large-packages runs apt-get remove and costs minutes for little gain;
+    #   - tool-cache would delete the preinstalled Python that actions/setup-python
+    #     then has to re-download, making the job slower, not faster;
+    #   - docker-images is pointless here (the Oracle service container is already
+    #     running, so its image is in use and cannot be pruned anyway).
+    - name: Free Disk Space
+      uses: jlumbroso/free-disk-space@main
+      with:
+        tool-cache: false
+        android: true
+        dotnet: true
+        haskell: true
+        large-packages: false
+        docker-images: false
+        swap-storage: true
+
     - name: Setup Oracle test user
       # The SYSTEM tablespace uses manual segment space management which
       # doesn't support VECTOR types. Create an ASSM tablespace and a
@@ -2231,6 +2267,27 @@ jobs:
       with:
         ref: ${{ github.event.pull_request.head.sha || '' }}
 
+    # The Oracle 23ai `free` service image is large; combined with the Python ML
+    # deps (torch) it exhausts the runner's ~14 GB root disk, so uv fails to
+    # extract a wheel with "No space left on device (os error 28)".
+    # Only the cheap, high-yield reclaims are enabled — the Android/.NET/Haskell
+    # dirs plus swap are a few `rm -rf`s worth ~16-21 GB, which is ample headroom:
+    #   - large-packages runs apt-get remove and costs minutes for little gain;
+    #   - tool-cache would delete the preinstalled Python that actions/setup-python
+    #     then has to re-download, making the job slower, not faster;
+    #   - docker-images is pointless here (the Oracle service container is already
+    #     running, so its image is in use and cannot be pruned anyway).
+    - name: Free Disk Space
+      uses: jlumbroso/free-disk-space@main
+      with:
+        tool-cache: false
+        android: true
+        dotnet: true
+        haskell: true
+        large-packages: false
+        docker-images: false
+        swap-storage: true
+
     - name: Setup Oracle test user
       run: |
         pip install oracledb
@@ -2391,6 +2448,27 @@ jobs:
       with:
         ref: ${{ github.event.pull_request.head.sha || '' }}
 
+    # The Oracle 23ai `free` service image is large; combined with the Python ML
+    # deps (torch) it exhausts the runner's ~14 GB root disk, so uv fails to
+    # extract a wheel with "No space left on device (os error 28)".
+    # Only the cheap, high-yield reclaims are enabled — the Android/.NET/Haskell
+    # dirs plus swap are a few `rm -rf`s worth ~16-21 GB, which is ample headroom:
+    #   - large-packages runs apt-get remove and costs minutes for little gain;
+    #   - tool-cache would delete the preinstalled Python that actions/setup-python
+    #     then has to re-download, making the job slower, not faster;
+    #   - docker-images is pointless here (the Oracle service container is already
+    #     running, so its image is in use and cannot be pruned anyway).
+    - name: Free Disk Space
+      uses: jlumbroso/free-disk-space@main
+      with:
+        tool-cache: false
+        android: true
+        dotnet: true
+        haskell: true
+        large-packages: false
+        docker-images: false
+        swap-storage: true
+
     - name: Setup Oracle test user
       run: |
         pip install oracledb
@@ -3516,6 +3594,43 @@ jobs:
       # (requires_real_llm) needs a live Hindsight server and runs separately.
       run: uv run pytest tests -v -m "not requires_real_llm"
 
+  test-copilot-cli-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-copilot-cli == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build copilot-cli integration
+      working-directory: ./hindsight-integrations/copilot-cli
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/copilot-cli
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/copilot-cli
+      run: uv run pytest tests -v
+
   test-crewai-integration:
     needs: [detect-changes]
     if: >-

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.8.4
-appVersion: "0.8.4"
+version: 0.8.6
+appVersion: "0.8.6"
 keywords:
   - ai
   - memory

helm/hindsight/templates/worker-statefulset.yaml

@@ -60,13 +60,13 @@ spec:
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
-        {{- /* Inherit LLM config from api.env */}}
-        {{- range $key, $value := .Values.api.env }}
-        - name: {{ $key }}
-          value: {{ $value | quote }}
-        {{- end }}
-        {{- /* Worker-specific env vars */}}
-        {{- range $key, $value := .Values.worker.env }}
+        {{- /* Explicitly set port to override K8s service discovery env var (HINDSIGHT_API_PORT) */}}
+        - name: HINDSIGHT_API_PORT
+          value: {{ .Values.worker.service.targetPort | quote }}
+        {{- /* Inherit LLM config from api.env, then apply worker-specific env.
+               Merge (worker.env wins) so a key set in both does not emit a
+               duplicate env entry, which server-side apply rejects. */}}
+        {{- range $key, $value := merge (deepCopy (.Values.worker.env | default dict)) (.Values.api.env | default dict) }}
         - name: {{ $key }}
           value: {{ $value | quote }}
         {{- end }}

hindsight-all-npm/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vectorize-io/hindsight-all",
-  "version": "0.8.4",
+  "version": "0.8.6",
   "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",

hindsight-all-slim/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "hindsight-all-slim"
-version = "0.8.4"
+version = "0.8.6"
 description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim==0.8.4",
+    "hindsight-api-slim==0.8.6",
     "hindsight-client>=0.0.7",
     "hindsight-embed>=0.1.0",
 ]

hindsight-all/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-all"
-version = "0.8.4"
+version = "0.8.6"
 description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim[all]==0.8.4",
+    "hindsight-api-slim[all]==0.8.6",
     "hindsight-client>=0.0.7",
     "hindsight-embed>=0.1.0",
 ]
@@ -21,7 +21,7 @@ hindsight-embed = { workspace = true }
 
 [project.optional-dependencies]
 local-llm = [
-    "hindsight-api-slim[local-llm]==0.8.4",
+    "hindsight-api-slim[local-llm]==0.8.6",
 ]
 test = [
     "pytest>=7.0.0",

hindsight-api-slim/hindsight_api/__init__.py

@@ -53,4 +53,4 @@ __all__ = [
     "RemoteTEICrossEncoder",
     "LLMConfig",
 ]
-__version__ = "0.8.4"
+__version__ = "0.8.6"

hindsight-api-slim/hindsight_api/admin/cli.py

@@ -9,6 +9,7 @@ import io
 import json
 import logging
 import zipfile
+from dataclasses import dataclass
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any
@@ -16,7 +17,7 @@ from typing import Any
 import asyncpg
 import typer
 
-from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig
+from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig, load_dotenv_for_entrypoint
 from ..engine.memory_engine import _current_schema
 from ..engine.retain.bank_utils import _vector_index_clause
 from ..engine.schema import fq_table_explicit as _fq_table
@@ -67,7 +68,88 @@ BACKUP_TABLES = [
     "graph_maintenance_queue",
 ]
 
-MANIFEST_VERSION = "1"
+MANIFEST_VERSION = "2"
+
+
+@dataclass(frozen=True)
+class BackupColumn:
+    """A PostgreSQL column shape required to decode a binary COPY stream."""
+
+    name: str
+    type_name: str
+
+
+async def _table_columns(conn: asyncpg.Connection, schema: str, table: str) -> list[BackupColumn]:
+    rows = await conn.fetch(
+        """
+        SELECT a.attname AS name, pg_catalog.format_type(a.atttypid, a.atttypmod) AS type_name
+        FROM pg_catalog.pg_attribute AS a
+        JOIN pg_catalog.pg_class AS c ON c.oid = a.attrelid
+        JOIN pg_catalog.pg_namespace AS n ON n.oid = c.relnamespace
+        WHERE n.nspname = $1 AND c.relname = $2 AND a.attnum > 0 AND NOT a.attisdropped
+          AND a.attgenerated = ''
+        ORDER BY a.attnum
+        """,
+        schema,
+        table,
+    )
+    return [BackupColumn(name=row["name"], type_name=row["type_name"]) for row in rows]
+
+
+async def _validate_restore_schema(
+    conn: asyncpg.Connection, manifest: dict[str, Any], schema: str
+) -> dict[str, list[str]]:
+    """Validate every COPY stream against the target before destructive work starts.
+
+    Type equality is an exact ``format_type`` string match. This is deliberately
+    stricter than binary-COPY wire compatibility (e.g. ``varchar`` and ``text``
+    share a binary format yet compare unequal here): we would rather fail a
+    genuinely-restorable backup with a clear, actionable error than silently risk
+    a subtle binary mismatch. Restores blocked this way can be recovered by
+    aligning the target schema.
+    """
+    restore_columns: dict[str, list[str]] = {}
+    errors: list[str] = []
+    for table, table_manifest in manifest["tables"].items():
+        source_columns = [BackupColumn(**column) for column in table_manifest["columns"]]
+        target_by_name = {column.name: column for column in await _table_columns(conn, schema, table)}
+        missing = [column.name for column in source_columns if column.name not in target_by_name]
+        mismatched = [
+            f"{column.name} ({column.type_name} in backup, {target_by_name[column.name].type_name} in target)"
+            for column in source_columns
+            if column.name in target_by_name and target_by_name[column.name].type_name != column.type_name
+        ]
+        if missing:
+            errors.append(f"{table}: target is missing backup columns {', '.join(missing)}")
+        if mismatched:
+            errors.append(f"{table}: incompatible column types: {', '.join(mismatched)}")
+        restore_columns[table] = [column.name for column in source_columns]
+
+    if errors:
+        details = "; ".join(errors)
+        raise ValueError(f"Backup schema is incompatible with target schema '{schema}': {details}")
+    return restore_columns
+
+
+def _effective_backup_tables() -> list[str]:
+    """Core backup tables plus any bank-scoped tables a loaded extension declares.
+
+    ``BACKUP_TABLES`` covers only the tables core owns. An extension that
+    provisions its own bank-scoped tables (via ``TenantExtension``) declares
+    them through ``extra_bank_tables()`` so they aren't dropped on restore.
+    Extension tables are appended *after* the core set so restore's forward
+    COPY inserts them after their FK parents (e.g. ``banks``) and the reversed
+    TRUNCATE clears them before those parents.
+    """
+    tables = list(BACKUP_TABLES)
+    tenant_extension = load_extension("TENANT", TenantExtension)
+    if tenant_extension is not None:
+        seen = set(tables)
+        for spec in tenant_extension.extra_bank_tables():
+            if spec.include_in_backup and spec.name not in seen:
+                tables.append(spec.name)
+                seen.add(spec.name)
+    return tables
 
 
 async def _admin_connect(db_url: str) -> asyncpg.Connection:
@@ -88,8 +170,18 @@ async def _admin_connect(db_url: str) -> asyncpg.Connection:
     return conn
 
 
-async def _backup(database_url: str, output_path: Path, schema: str = "public") -> dict[str, Any]:
-    """Backup all tables to a zip file using binary COPY protocol."""
+async def _backup(
+    database_url: str,
+    output_path: Path,
+    schema: str = "public",
+    backup_tables: list[str] | None = None,
+) -> dict[str, Any]:
+    """Backup all tables to a zip file using binary COPY protocol.
+
+    ``backup_tables`` defaults to the core ``BACKUP_TABLES``; callers pass the
+    extension-augmented list from ``_effective_backup_tables()``.
+    """
+    backup_tables = backup_tables if backup_tables is not None else BACKUP_TABLES
     conn = await asyncpg.connect(database_url)
     try:
         tables: dict[str, Any] = {}
@@ -106,14 +198,24 @@ async def _backup(database_url: str, output_path: Path, schema: str = "public")
         # entities table was backed up.
         async with conn.transaction(isolation="repeatable_read"):
             with zipfile.ZipFile(output_path, "w", zipfile.ZIP_DEFLATED) as zf:
-                for i, table in enumerate(BACKUP_TABLES, 1):
-                    typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] Backing up {table}...", nl=False)
+                for i, table in enumerate(backup_tables, 1):
+                    typer.echo(f"  [{i}/{len(backup_tables)}] Backing up {table}...", nl=False)
 
                     buffer = io.BytesIO()
 
-                    # Use binary COPY for exact type preservation
+                    columns = await _table_columns(conn, schema, table)
+
+                    # Pin the ordered columns into both the stream and manifest.
+                    # PostgreSQL binary COPY does not encode column identities, so
+                    # restore must validate this shape before truncating any data.
                     # asyncpg requires schema_name as separate parameter
-                    await conn.copy_from_table(table, schema_name=schema, output=buffer, format="binary")
+                    await conn.copy_from_table(
+                        table,
+                        schema_name=schema,
+                        columns=[column.name for column in columns],
+                        output=buffer,
+                        format="binary",
+                    )
 
                     data = buffer.getvalue()
                     zf.writestr(f"{table}.bin", data)
@@ -124,6 +226,7 @@ async def _backup(database_url: str, output_path: Path, schema: str = "public")
                     tables[table] = {
                         "rows": row_count,
                         "size_bytes": len(data),
+                        "columns": [{"name": column.name, "type_name": column.type_name} for column in columns],
                     }
 
                     typer.echo(f" {row_count} rows")
@@ -135,8 +238,20 @@ async def _backup(database_url: str, output_path: Path, schema: str = "public")
         await conn.close()
 
 
-async def _restore(database_url: str, input_path: Path, schema: str = "public") -> dict[str, Any]:
-    """Restore all tables from a zip file using binary COPY protocol."""
+async def _restore(
+    database_url: str,
+    input_path: Path,
+    schema: str = "public",
+    backup_tables: list[str] | None = None,
+) -> dict[str, Any]:
+    """Restore all tables from a zip file using binary COPY protocol.
+
+    ``backup_tables`` defaults to the core ``BACKUP_TABLES``; callers pass the
+    extension-augmented list from ``_effective_backup_tables()``. Tables named
+    here but absent from the archive are truncated then skipped for restore, so
+    a stale extension registration never leaves pre-restore rows behind.
+    """
+    backup_tables = backup_tables if backup_tables is not None else BACKUP_TABLES
     conn = await asyncpg.connect(database_url)
     try:
         with zipfile.ZipFile(input_path, "r") as zf:
@@ -145,29 +260,40 @@ async def _restore(database_url: str, input_path: Path, schema: str = "public")
             if manifest.get("version") != MANIFEST_VERSION:
                 raise ValueError(f"Unsupported backup version: {manifest.get('version')}")
 
+            # Complete the compatibility check before entering the transaction
+            # that truncates tables. This turns historical schema drift into an
+            # actionable error without risking the target's existing data.
+            restore_columns = await _validate_restore_schema(conn, manifest, schema)
+
             # Use a transaction for atomic restore - either all tables are
             # restored or none are, preventing partial/inconsistent state.
             async with conn.transaction():
                 typer.echo("  Clearing existing data...")
                 # Truncate tables in reverse order (respects FK constraints)
-                for table in reversed(BACKUP_TABLES):
+                for table in reversed(backup_tables):
                     qualified_table = _fq_table(table, schema)
                     await conn.execute(f"TRUNCATE TABLE {qualified_table} CASCADE")
 
                 # Restore tables in forward order
-                for i, table in enumerate(BACKUP_TABLES, 1):
+                for i, table in enumerate(backup_tables, 1):
                     filename = f"{table}.bin"
                     if filename not in zf.namelist():
-                        typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] {table}: skipped (not in backup)")
+                        typer.echo(f"  [{i}/{len(backup_tables)}] {table}: skipped (not in backup)")
                         continue
 
                     expected_rows = manifest["tables"].get(table, {}).get("rows", "?")
-                    typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] Restoring {table}... {expected_rows} rows")
+                    typer.echo(f"  [{i}/{len(backup_tables)}] Restoring {table}... {expected_rows} rows")
 
                     data = zf.read(filename)
                     buffer = io.BytesIO(data)
                     # asyncpg requires schema_name as separate parameter
-                    await conn.copy_to_table(table, schema_name=schema, source=buffer, format="binary")
+                    await conn.copy_to_table(
+                        table,
+                        schema_name=schema,
+                        columns=restore_columns[table],
+                        source=buffer,
+                        format="binary",
+                    )
 
                 # Refresh materialized view
                 typer.echo("  Refreshing materialized views...")
@@ -185,7 +311,7 @@ async def _run_backup(db_url: str, output: Path, schema: str = "public") -> dict
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
-    return await _backup(resolved_url, output, schema)
+    return await _backup(resolved_url, output, schema, backup_tables=_effective_backup_tables())
 
 
 async def _run_restore(db_url: str, input_file: Path, schema: str = "public") -> dict[str, Any]:
@@ -195,7 +321,7 @@ async def _run_restore(db_url: str, input_file: Path, schema: str = "public") ->
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
-    return await _restore(resolved_url, input_file, schema)
+    return await _restore(resolved_url, input_file, schema, backup_tables=_effective_backup_tables())
 
 
 @app.command()
@@ -219,7 +345,7 @@ def backup(
     manifest = asyncio.run(_run_backup(config.database_url, output, schema))
 
     total_rows = sum(t["rows"] for t in manifest["tables"].values())
-    typer.echo(f"Backed up {total_rows} rows across {len(BACKUP_TABLES)} tables")
+    typer.echo(f"Backed up {total_rows} rows across {len(manifest['tables'])} tables")
     typer.echo(f"Backup saved to {output}")
 
 
@@ -252,7 +378,7 @@ def restore(
     manifest = asyncio.run(_run_restore(config.database_url, input_file, schema))
 
     total_rows = sum(t["rows"] for t in manifest["tables"].values())
-    typer.echo(f"Restored {total_rows} rows across {len(BACKUP_TABLES)} tables")
+    typer.echo(f"Restored {total_rows} rows across {len(manifest['tables'])} tables")
     typer.echo("Restore complete")
 
 
@@ -273,11 +399,10 @@ async def _run_migration(
     resolved_url = await resolve_database_url(db_url)
 
     config = HindsightConfig.from_env()
+    tenant_extension = load_extension("TENANT", TenantExtension)
     if schema:
         schemas = [schema]
     else:
-        tenant_extension = load_extension("TENANT", TenantExtension)
-
         schemas = [base_schema or DEFAULT_DATABASE_SCHEMA]
         if tenant_extension:
             tenants = await tenant_extension.list_tenants()
@@ -302,9 +427,36 @@ async def _run_migration(
         ensure_extensions=ensure_extensions,
     )
 
+    # After core migrations, provision any extension-owned bank-scoped tables
+    # per schema so extension schema evolves on the same lifecycle as core
+    # schema (rather than via a lazy first-request path).
+    if tenant_extension is not None:
+        await _provision_extra_bank_tables(resolved_url, schemas, tenant_extension)
+
     return schemas
 
 
+async def _provision_extra_bank_tables(
+    resolved_url: str, schemas: list[str], tenant_extension: TenantExtension
+) -> None:
+    """Run the tenant extension's table provisioner for each migrated schema.
+
+    Fires after core migrations complete so extension-owned bank tables are
+    created/evolved on the same lifecycle as core schema. A failure aborts the
+    migration command (and names the offending schema) rather than being
+    swallowed — provisioning is idempotent, so the operator can fix and re-run.
+    """
+    for schema in schemas:
+        conn = await asyncpg.connect(resolved_url)
+        try:
+            await tenant_extension.provision_bank_tables(conn, schema)
+        except Exception as e:
+            typer.echo(f"  Failed to provision extension tables for schema '{schema}': {e}", err=True)
+            raise
+        finally:
+            await conn.close()
+
+
 @app.command(name="run-db-migration")
 def run_db_migration(
     schema: str | None = typer.Option(
@@ -808,6 +960,7 @@ def worker_status(
 
 
 def main():
+    load_dotenv_for_entrypoint()
     app()
 
 

hindsight-api-slim/hindsight_api/alembic/versions/c7d1e9a4b3f2_add_archive_causal_links.py

@@ -0,0 +1,90 @@
+"""Add ``causal_links`` to the curation archive (invalidated_memory_units).
+
+Causal edges (``caused_by`` and the historical ``causes``/``enables``/
+``prevents``) are retain-time extraction output: unlike temporal and semantic
+links they cannot be recomputed from dates or embeddings, and graph maintenance
+never rebuilds them. Invalidation MOVES a fact out of ``memory_units``, so the
+``memory_links → memory_units`` FK cascade deletes every incident edge — and
+revert had no way to bring the causal ones back (#2864).
+
+This column parks the descriptors of the causal edges incident to an archived
+fact — ``[{"from_unit_id", "to_unit_id", "link_type", "weight"}, ...]`` — so
+revert can rematerialize them. It is deliberately unindexed and lives only on
+the archive: live facts keep their causal edges in ``memory_links`` (curation
+edits no longer delete them), and the archive is small, cold, and only read by
+low-frequency curation operations.
+
+Revision ID: c7d1e9a4b3f2
+Revises: d7b2f8a1c934
+Create Date: 2026-07-24
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "c7d1e9a4b3f2"
+down_revision: str | Sequence[str] | None = "d7b2f8a1c934"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _pg_schema_prefix() -> str:
+    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    schema = _pg_schema_prefix()
+    # NOT NULL DEFAULT is metadata-only on PG 11+, so this is cheap even on a
+    # large archive. Existing rows read as "no causal edges captured" — edges
+    # lost before this migration cannot be reconstructed and are not guessed.
+    op.execute(
+        f"ALTER TABLE {schema}invalidated_memory_units "
+        f"ADD COLUMN IF NOT EXISTS causal_links JSONB NOT NULL DEFAULT '[]'::jsonb"
+    )
+
+
+def _pg_downgrade() -> None:
+    schema = _pg_schema_prefix()
+    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS causal_links")
+
+
+def _oracle_upgrade() -> None:
+    # Kept in sync with PG for schema parity (curation itself is PostgreSQL-only
+    # today — it introspects pg_attribute to move rows between the two tables).
+    # Swallow ORA-01430 (column already exists) so the migration is idempotent.
+    op.execute(
+        """
+        BEGIN
+            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units ADD (causal_links CLOB DEFAULT ''[]''
+                CONSTRAINT imu_causal_links_json CHECK (causal_links IS JSON))';
+        EXCEPTION WHEN OTHERS THEN
+            IF SQLCODE != -1430 THEN RAISE; END IF;
+        END;
+        """
+    )
+
+
+def _oracle_downgrade() -> None:
+    # Swallow ORA-00904 (column does not exist).
+    op.execute(
+        """
+        BEGIN
+            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units DROP COLUMN causal_links';
+        EXCEPTION WHEN OTHERS THEN
+            IF SQLCODE != -904 THEN RAISE; END IF;
+        END;
+        """
+    )
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/api/http.py

@@ -27,7 +27,7 @@ from hindsight_api.engine.audit import (
     AuditLogStatsResponse,
 )
 from hindsight_api.engine.llm_trace import LLMRequestListResponse, LLMRequestStatsResponse
-from hindsight_api.extensions import AuthenticationError, PrecheckOperation
+from hindsight_api.extensions import AuthenticationError, BankWriteOperation, PrecheckOperation
 
 
 def _parse_metadata(metadata: Any) -> dict[str, Any]:
@@ -149,8 +149,15 @@ def FieldWithDefault(default_factory: Callable, **kwargs) -> Any:
 
 
 from hindsight_api.config import get_config
-from hindsight_api.engine.memory_engine import Budget, _current_schema, _get_tiktoken_encoding
+from hindsight_api.engine.interface import BankTemplateImportWrite
+from hindsight_api.engine.memory_engine import (
+    Budget,
+    RetainOperationConflictError,
+    _current_schema,
+    _get_tiktoken_encoding,
+)
 from hindsight_api.engine.providers.none_llm import LLMNotAvailableError
+from hindsight_api.engine.reflect import ReflectToolCallError
 from hindsight_api.engine.response_models import (
     VALID_RECALL_FACT_TYPES,
     DryRunExtractionResult,
@@ -722,6 +729,25 @@ class RetainRequest(BaseModel):
         description="Deprecated. Use item-level tags instead.",
         deprecated=True,
     )
+    operation_id: str | None = Field(
+        default=None,
+        description=(
+            "Optional client-supplied UUID used as the identity of an async retain operation. "
+            "Re-submitting with the same operation_id returns the original operation and creates no new "
+            "work, so retrying after a lost or timed-out acknowledgement will not enqueue a duplicate. "
+            "Reusing an id that belongs to a different operation returns HTTP 409. Ignored for synchronous retain."
+        ),
+    )
+
+    @field_validator("operation_id")
+    @classmethod
+    def validate_operation_id(cls, value: str | None) -> str | None:
+        if value is None:
+            return None
+        try:
+            return str(uuid.UUID(value))
+        except (ValueError, AttributeError, TypeError) as exc:
+            raise ValueError("operation_id must be a valid UUID") from exc
 
 
 class FileRetainMetadata(BaseModel):
@@ -770,6 +796,26 @@ class FileRetainRequest(BaseModel):
         description="Metadata for each file (optional, must match number of files if provided)",
     )
 
+    @model_validator(mode="before")
+    @classmethod
+    def reject_misplaced_file_metadata(cls, data: Any) -> Any:
+        if not isinstance(data, dict):
+            return data
+
+        misplaced = sorted(name for name in _FILE_RETAIN_PER_FILE_FIELDS if data.get(name) is not None)
+        errors = []
+        if misplaced:
+            fields = ", ".join(misplaced)
+            errors.append(f"Per-file fields ({fields}) must be placed in the corresponding 'files_metadata' entry")
+        if data.get("update_mode") is not None:
+            errors.append("'update_mode' is not supported by /files/retain, which always processes asynchronously")
+        if errors:
+            raise ValueError("; ".join(errors))
+        return data
+
+
+_FILE_RETAIN_PER_FILE_FIELDS = frozenset(FileRetainMetadata.model_fields) - frozenset(FileRetainRequest.model_fields)
+
 
 class RetainResponse(BaseModel):
     """Response model for retain endpoint."""
@@ -910,6 +956,12 @@ class ReflectRequest(BaseModel):
         description="Compound tag filter using boolean groups. Groups in the list are AND-ed. "
         "Each group is a leaf {tags, match} or compound {and: [...]}, {or: [...]}, {not: ...}.",
     )
+    apply_all_directives: bool = Field(
+        default=False,
+        description="Apply every active directive regardless of tags. By default directives are "
+        "scoped like memories: untagged directives always apply, and tagged directives apply only "
+        "when the request's tags match them. Set true to apply all active directives, ignoring tag scope.",
+    )
     fact_types: list[Literal["world", "experience", "observation"]] | None = Field(
         default=None,
         description="Filter which fact types are retrieved during reflect. None means all types (world, experience, observation).",
@@ -2297,6 +2349,14 @@ class BankTemplateConfig(BaseModel):
     recall_budget_max: int | None = Field(
         default=None, description="Ceiling for the adaptive function (after clamping)"
     )
+    audit_log_enabled: bool | None = Field(
+        default=None, description="Enable audit logging for this bank (overrides the server default)"
+    )
+    store_document_text: bool | None = Field(
+        default=None,
+        description="Persist raw source text (documents.original_text / chunks.chunk_text). "
+        "Set false to keep only derived facts.",
+    )
 
     def get_config_updates(self) -> dict[str, Any]:
         """Return only the fields that were explicitly set (non-None)."""
@@ -2442,50 +2502,222 @@ def validate_bank_template(manifest: "BankTemplateManifest") -> list[str]:
         if bank.retain_custom_instructions and bank.retain_extraction_mode != "custom":
             errors.append("bank.retain_custom_instructions: requires retain_extraction_mode='custom'")
     if manifest.mental_models:
+        seen_mental_model_ids: set[str] = set()
         for i, mm in enumerate(manifest.mental_models):
             if not mm.name.strip():
                 errors.append(f"mental_models[{i}].name: must not be empty")
             if not mm.source_query.strip():
                 errors.append(f"mental_models[{i}].source_query: must not be empty")
+            if mm.id in seen_mental_model_ids:
+                errors.append(f"mental_models[{i}].id: duplicate id '{mm.id}'")
+            seen_mental_model_ids.add(mm.id)
     if manifest.directives:
+        seen_directive_names: set[str] = set()
         for i, d in enumerate(manifest.directives):
             if not d.name.strip():
                 errors.append(f"directives[{i}].name: must not be empty")
             if not d.content.strip():
                 errors.append(f"directives[{i}].content: must not be empty")
+            if d.name in seen_directive_names:
+                errors.append(f"directives[{i}].name: duplicate name '{d.name}'")
+            seen_directive_names.add(d.name)
     return errors
 
 
+def load_default_bank_template_manifest() -> "BankTemplateManifest | None":
+    """Parse and semantically validate the configured default bank template."""
+    template_dict = get_config().default_bank_template
+    if not template_dict:
+        return None
+    manifest = BankTemplateManifest.model_validate(template_dict)
+    semantic_errors = validate_bank_template(manifest)
+    if semantic_errors:
+        raise ValueError("; ".join(semantic_errors))
+    return manifest
+
+
 async def apply_bank_template_manifest(
-    memory,
+    memory: MemoryEngine,
     bank_id: str,
     manifest: "BankTemplateManifest",
     request_context: "RequestContext",
 ) -> "BankTemplateImportResponse":
-    """Apply a validated BankTemplateManifest to an existing bank.
+    """Apply a client-provided BankTemplateManifest to a bank.
 
-    Shared by the /import endpoint and the default-template-on-create hook
-    driven by HINDSIGHT_API_DEFAULT_BANK_TEMPLATE. The bank MUST already
-    exist; caller is responsible for validation (Pydantic + validate_bank_template).
+    The authorization context creates a missing bank after validating every
+    requested operation. The caller remains responsible for manifest validation
+    (Pydantic + validate_bank_template). Server-owned defaults use
+    ``apply_default_bank_template_resources`` instead, so this function always
+    owns persistence of its client-provided config.
     """
-    config_applied = False
-    if manifest.bank:
-        config_updates = manifest.bank.get_config_updates()
-        if config_updates:
-            await memory._config_resolver.update_bank_config(bank_id, config_updates, request_context)
-            config_applied = True
+    config_updates = manifest.bank.get_config_updates() if manifest.bank else {}
+    bank_exists = (
+        await memory.get_bank_profile(
+            bank_id,
+            request_context=request_context,
+            create_if_missing=False,
+        )
+        is not None
+    )
 
+    # A missing bank receives the server-owned default template during
+    # provisioning. Project those resources into the authorization decision so
+    # the client's import is authorized as an update when the default owns the
+    # same key, while still keeping every client check before bank creation.
+    default_manifest: BankTemplateManifest | None = None
+    if not bank_exists:
+        try:
+            default_manifest = load_default_bank_template_manifest()
+        except (ValueError, ValidationError):
+            # Provisioning owns error logging and the best-effort fallback for a
+            # malformed server template. Client authorization must not change it.
+            pass
+    imported_mental_model_ids = {item.id for item in manifest.mental_models or []}
+    imported_directive_names = {item.name for item in manifest.directives or []}
+    default_mental_models = (default_manifest.mental_models or []) if default_manifest else []
+    default_directives = (default_manifest.directives or []) if default_manifest else []
+    projected_mental_model_ids = {item.id for item in default_mental_models} & imported_mental_model_ids
+    projected_directive_names = {item.name for item in default_directives} & imported_directive_names
+
+    existing_by_id: dict[str, dict[str, Any]] = {}
+    if bank_exists and manifest.mental_models:
+        existing = await memory.list_mental_models(bank_id=bank_id, request_context=request_context)
+        existing_by_id = {m["id"]: m for m in existing}
+
+    existing_by_name: dict[str, dict[str, Any]] = {}
+    if bank_exists and manifest.directives:
+        existing_directives = await memory.list_directives(
+            bank_id=bank_id, active_only=False, request_context=request_context
+        )
+        existing_by_name = {d["name"]: d for d in existing_directives}
+
+    bank_writes: list[BankTemplateImportWrite] = []
+    if config_updates:
+        bank_writes.append(BankTemplateImportWrite(BankWriteOperation.UPDATE_BANK_CONFIG))
+    for mental_model in manifest.mental_models or []:
+        if mental_model.id in existing_by_id:
+            bank_writes.append(BankTemplateImportWrite(BankWriteOperation.UPDATE_MENTAL_MODEL, mental_model.id))
+        elif mental_model.id in projected_mental_model_ids:
+            # Default-template application is best-effort. Authorize both
+            # outcomes before provisioning so a failed default create can
+            # safely fall back to the client's create operation.
+            bank_writes.extend(
+                [
+                    BankTemplateImportWrite(BankWriteOperation.UPDATE_MENTAL_MODEL, mental_model.id),
+                    BankTemplateImportWrite(BankWriteOperation.CREATE_MENTAL_MODEL, mental_model.id),
+                ]
+            )
+        else:
+            bank_writes.append(BankTemplateImportWrite(BankWriteOperation.CREATE_MENTAL_MODEL, mental_model.id))
+    for directive in manifest.directives or []:
+        if directive.name in existing_by_name:
+            bank_writes.append(BankTemplateImportWrite(BankWriteOperation.UPDATE_DIRECTIVE, directive.name))
+        elif directive.name in projected_directive_names:
+            bank_writes.extend(
+                [
+                    BankTemplateImportWrite(BankWriteOperation.UPDATE_DIRECTIVE, directive.name),
+                    BankTemplateImportWrite(BankWriteOperation.CREATE_DIRECTIVE, directive.name),
+                ]
+            )
+        else:
+            bank_writes.append(BankTemplateImportWrite(BankWriteOperation.CREATE_DIRECTIVE, directive.name))
+
+    async with memory.bank_template_import_authorization(
+        bank_id,
+        config_updates=config_updates,
+        bank_writes=bank_writes,
+        mental_model_ids=[mental_model.id for mental_model in manifest.mental_models or []],
+        bank_exists=bank_exists,
+        request_context=request_context,
+    ):
+        if projected_mental_model_ids:
+            provisioned = await memory.list_mental_models(
+                bank_id=bank_id,
+                request_context=request_context,
+            )
+            provisioned_by_id = {item["id"]: item for item in provisioned}
+            existing_by_id.update(
+                {
+                    item_id: provisioned_by_id[item_id]
+                    for item_id in projected_mental_model_ids & provisioned_by_id.keys()
+                }
+            )
+
+        if projected_directive_names:
+            provisioned = await memory.list_directives(
+                bank_id=bank_id,
+                active_only=False,
+                request_context=request_context,
+            )
+            provisioned_by_name = {item["name"]: item for item in provisioned}
+            existing_by_name.update(
+                {name: provisioned_by_name[name] for name in projected_directive_names & provisioned_by_name.keys()}
+            )
+
+        if config_updates:
+            await memory.update_bank_config(bank_id, config_updates, request_context=request_context)
+
+        return await _apply_bank_template_resources(
+            memory,
+            bank_id,
+            manifest,
+            existing_by_id,
+            existing_by_name,
+            request_context,
+            config_applied=bool(config_updates),
+        )
+
+
+async def apply_default_bank_template_resources(
+    memory: MemoryEngine,
+    bank_id: str,
+    manifest: "BankTemplateManifest",
+    request_context: "RequestContext",
+) -> None:
+    """Apply only the resources from a server-owned default template."""
+    existing_by_id: dict[str, dict[str, Any]] = {}
+    if manifest.mental_models:
+        existing = await memory.list_mental_models(bank_id=bank_id, request_context=request_context)
+        existing_by_id = {model["id"]: model for model in existing}
+
+    existing_by_name: dict[str, dict[str, Any]] = {}
+    if manifest.directives:
+        existing_directives = await memory.list_directives(
+            bank_id=bank_id,
+            active_only=False,
+            request_context=request_context,
+        )
+        existing_by_name = {directive["name"]: directive for directive in existing_directives}
+
+    await _apply_bank_template_resources(
+        memory,
+        bank_id,
+        manifest,
+        existing_by_id,
+        existing_by_name,
+        request_context,
+        config_applied=False,
+    )
+
+
+async def _apply_bank_template_resources(
+    memory: MemoryEngine,
+    bank_id: str,
+    manifest: "BankTemplateManifest",
+    existing_mental_models: dict[str, dict[str, Any]],
+    existing_directives: dict[str, dict[str, Any]],
+    request_context: "RequestContext",
+    *,
+    config_applied: bool,
+) -> "BankTemplateImportResponse":
+    """Apply template resources after the caller has handled config and access."""
     created_ids: list[str] = []
     updated_ids: list[str] = []
     operation_ids: list[str] = []
 
     if manifest.mental_models:
-        # Fetch existing mental models to decide create vs update
-        existing = await memory.list_mental_models(bank_id=bank_id, request_context=request_context)
-        existing_by_id = {m["id"]: m for m in existing}
-
         for mm in manifest.mental_models:
-            if mm.id in existing_by_id:
+            if mm.id in existing_mental_models:
                 await memory.update_mental_model(
                     bank_id=bank_id,
                     mental_model_id=mm.id,
@@ -2527,16 +2759,12 @@ async def apply_bank_template_manifest(
     directives_updated: list[str] = []
 
     if manifest.directives:
-        existing_directives = await memory.list_directives(
-            bank_id=bank_id, active_only=False, request_context=request_context
-        )
-        existing_by_name = {d["name"]: d for d in existing_directives}
-
         for directive in manifest.directives:
-            if directive.name in existing_by_name:
+            if directive.name in existing_directives:
                 await memory.update_directive(
                     bank_id=bank_id,
-                    directive_id=existing_by_name[directive.name]["id"],
+                    directive_id=existing_directives[directive.name]["id"],
+                    name=directive.name,
                     content=directive.content,
                     priority=directive.priority,
                     is_active=directive.is_active,
@@ -3102,6 +3330,7 @@ def create_app(
         config = get_config()
         poller = None
         poller_task = None
+        loop_watchdog = None
 
         # Initialize OpenTelemetry metrics
         try:
@@ -3145,6 +3374,12 @@ def create_app(
                 metrics_collector.set_db_pool(memory._pool)
                 logging.info("DB pool metrics configured")
 
+        # Start the event-loop stall watchdog (logs the culprit stack if a task
+        # blocks the loop, so a failing /health can be told apart from pool exhaustion).
+        from ..loop_watchdog import start_loop_watchdog
+
+        loop_watchdog = start_loop_watchdog(asyncio.get_running_loop())
+
         # Start worker poller if the backend supports it.
         # All current backends (PostgreSQL, Oracle) support async worker/poller.
         if config.worker_enabled and memory._backend.supports_worker_poller:
@@ -3189,6 +3424,10 @@ def create_app(
 
         yield
 
+        # Stop the loop watchdog first so it doesn't fire during teardown.
+        if loop_watchdog is not None:
+            loop_watchdog.stop()
+
         # Shutdown worker poller if running
         if poller is not None:
             await poller.shutdown_graceful(timeout=30.0)
@@ -3600,7 +3839,7 @@ def _register_routes(app: FastAPI):
         "/v1/default/banks/{bank_id}/memories/list",
         response_model=ListMemoryUnitsResponse,
         summary="List memory units",
-        description="List memory units with pagination and optional full-text search. Supports filtering by type. Results are sorted by most recent first (mentioned_at DESC, then created_at DESC).",
+        description="List memory units with pagination and optional full-text search. Supports filtering by type, source document, and linked entity ID. Results are sorted by most recent first (mentioned_at DESC, then created_at DESC).",
         operation_id="list_memories",
         tags=["Memory"],
     )
@@ -3611,6 +3850,7 @@ def _register_routes(app: FastAPI):
         consolidation_state: str | None = None,
         state: str | None = None,
         document_id: str | None = None,
+        entity_id: str | None = None,
         tags: list[str] | None = Query(default=None),
         tags_match: TagsMatch = Query(default="any"),
         limit: int = Query(default=100, ge=0),
@@ -3629,6 +3869,10 @@ def _register_routes(app: FastAPI):
             q: Search query for full-text search (searches text and context)
             consolidation_state: Filter by consolidation state for source memories
                 (world/experience). One of 'failed', 'pending', or 'done'.
+            document_id: Filter to a single source document.
+            entity_id: Filter to memory units linked to this entity ID (via stored
+                entity links, not text/semantic match). Combining with
+                state='invalidated' returns no results (the archive has no links).
             tags: Optional list of tag names to filter by.
             tags_match: How to combine tags: 'any' (OR, default) or 'all' (AND) both
                 also include untagged memories; 'any_strict'/'all_strict' exclude
@@ -3644,6 +3888,7 @@ def _register_routes(app: FastAPI):
                 consolidation_state=consolidation_state,
                 state=state,
                 document_id=document_id,
+                entity_id=entity_id,
                 tags=tags,
                 tags_match=tags_match,
                 limit=limit,
@@ -4110,6 +4355,7 @@ def _register_routes(app: FastAPI):
                         tags=request.tags,
                         tags_match=request.tags_match,
                         tag_groups=request.tag_groups,
+                        apply_all_directives=request.apply_all_directives,
                         fact_types=request.fact_types,
                         exclude_mental_models=request.exclude_mental_models,
                         exclude_mental_model_ids=request.exclude_mental_model_ids,
@@ -4193,6 +4439,12 @@ def _register_routes(app: FastAPI):
             raise
         except LLMNotAvailableError as e:
             raise HTTPException(status_code=400, detail=str(e))
+        except ReflectToolCallError as e:
+            # The configured model/transport can't drive reflect's tool-calling loop.
+            # The request itself is fine, so this is a server-side (500) failure, not a
+            # 4xx -- but log at warning, not error: it's a misconfiguration, not a bug.
+            logger.warning("Reflect tool-calling failure in bank %s: %s", bank_id, e)
+            raise HTTPException(status_code=500, detail=str(e))
         except TimeoutError as e:
             logger.error("Timeout in /v1/default/banks/%s/reflect: %s", bank_id, e)
             raise HTTPException(
@@ -5732,28 +5984,15 @@ def _register_routes(app: FastAPI):
     ):
         """Create or update an agent with disposition and mission."""
         try:
-            # Ensure bank exists, validating create_bank only when this call
-            # actually creates a missing bank.
-            await app.state.memory._ensure_bank_exists(
-                bank_id,
-                request_context,
-            )
-
-            # Update name if provided (stored in DB for display only, deprecated)
-            if request.name is not None:
-                await app.state.memory.update_bank(
-                    bank_id,
-                    name=request.name,
-                    request_context=request_context,
-                )
-
-            # Apply all config overrides (includes reflect_mission, disposition, retain settings)
+            # The engine validates and authorizes all requested changes before
+            # creating a missing bank.
             config_updates = request.get_config_updates()
-            if config_updates:
-                await app.state.memory._config_resolver.update_bank_config(bank_id, config_updates, request_context)
-
-            # Get final profile
-            final_profile = await app.state.memory.get_bank_profile(bank_id, request_context=request_context)
+            final_profile = await app.state.memory.update_bank(
+                bank_id,
+                name=request.name,
+                config_updates=config_updates or None,
+                request_context=request_context,
+            )
             disposition_dict = (
                 final_profile["disposition"].model_dump()
                 if hasattr(final_profile["disposition"], "model_dump")
@@ -5769,6 +6008,8 @@ def _register_routes(app: FastAPI):
             )
         except OperationValidationError as e:
             raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except ValueError as e:
+            raise HTTPException(status_code=400, detail=str(e))
         except (AuthenticationError, HTTPException):
             raise
         except Exception as e:
@@ -5792,33 +6033,16 @@ def _register_routes(app: FastAPI):
     ):
         """Partially update an agent's profile (name, mission, disposition)."""
         try:
-            # PATCH is update-only; missing banks must not be created as a
-            # side effect of reading the profile.
-            existing_profile = await app.state.memory.get_bank_profile(
-                bank_id, request_context=request_context, create_if_missing=False
-            )
-            if existing_profile is None:
-                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
-
-            # Update name if provided (stored in DB for display only, deprecated)
-            if request.name is not None:
-                await app.state.memory.update_bank(
-                    bank_id,
-                    name=request.name,
-                    request_context=request_context,
-                )
-
-            # Apply all config overrides (includes reflect_mission, disposition, retain settings)
+            # Update every requested field through one engine call so all
+            # authorization and validation completes before either write.
             config_updates = request.get_config_updates()
-            if config_updates:
-                await app.state.memory._config_resolver.update_bank_config(bank_id, config_updates, request_context)
-
-            # Get final profile
-            final_profile = await app.state.memory.get_bank_profile(
-                bank_id, request_context=request_context, create_if_missing=False
+            final_profile = await app.state.memory.update_bank(
+                bank_id,
+                name=request.name,
+                config_updates=config_updates or None,
+                create_if_missing=False,
+                request_context=request_context,
             )
-            if final_profile is None:
-                raise HTTPException(status_code=404, detail=f"Bank '{bank_id}' not found")
             disposition_dict = (
                 final_profile["disposition"].model_dump()
                 if hasattr(final_profile["disposition"], "model_dump")
@@ -5834,6 +6058,8 @@ def _register_routes(app: FastAPI):
             )
         except OperationValidationError as e:
             raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except ValueError as e:
+            raise HTTPException(status_code=400, detail=str(e))
         except (AuthenticationError, HTTPException):
             raise
         except Exception as e:
@@ -5929,13 +6155,6 @@ def _register_routes(app: FastAPI):
                     dry_run=True,
                 )
 
-            # Ensure bank exists, validating create_bank only when this import
-            # actually creates a missing target bank.
-            await app.state.memory._ensure_bank_exists(
-                bank_id,
-                request_context,
-            )
-
             return await apply_bank_template_manifest(
                 memory=app.state.memory,
                 bank_id=bank_id,
@@ -6310,25 +6529,8 @@ def _register_routes(app: FastAPI):
                 detail="Bank configuration API is disabled. Set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true to re-enable.",
             )
         try:
-            # Authenticate and set schema context for multi-tenant DB queries
-            await app.state.memory._authenticate_tenant(request_context)
-            if app.state.memory._operation_validator:
-                from hindsight_api.extensions import BankReadContext, BankReadOperation
-
-                ctx = BankReadContext(
-                    bank_id=bank_id, operation=BankReadOperation.GET_BANK_CONFIG, request_context=request_context
-                )
-                await app.state.memory._validate_operation(
-                    app.state.memory._operation_validator.validate_bank_read(ctx)
-                )
-
-            # Get resolved config from config resolver
-            config_dict = await app.state.memory._config_resolver.get_bank_config(bank_id, request_context)
-
-            # Get bank-specific overrides only
-            bank_overrides = await app.state.memory._config_resolver._load_bank_config(bank_id)
-
-            return BankConfigResponse(bank_id=bank_id, config=config_dict, overrides=bank_overrides)
+            state = await app.state.memory.get_bank_config(bank_id, request_context=request_context)
+            return BankConfigResponse(bank_id=bank_id, config=state.config, overrides=state.overrides)
         except OperationValidationError as e:
             raise HTTPException(status_code=e.status_code, detail=e.reason)
         except (AuthenticationError, HTTPException):
@@ -6360,35 +6562,12 @@ def _register_routes(app: FastAPI):
                 detail="Bank configuration API is disabled. Set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true to re-enable.",
             )
         try:
-            # Authenticate and set schema context for multi-tenant DB queries
-            await app.state.memory._authenticate_tenant(request_context)
-            if app.state.memory._operation_validator:
-                from hindsight_api.extensions import BankWriteContext, BankWriteOperation
-
-                ctx = BankWriteContext(
-                    bank_id=bank_id, operation=BankWriteOperation.UPDATE_BANK_CONFIG, request_context=request_context
-                )
-                await app.state.memory._validate_operation(
-                    app.state.memory._operation_validator.validate_bank_write(ctx)
-                )
-
-            # Validate Memory Defense policy shape before persisting.
-            if "memory_defense" in request.updates and request.updates["memory_defense"] is not None:
-                from hindsight_api.extensions.memory_defense import parse_policy
-
-                try:
-                    parse_policy(request.updates["memory_defense"])
-                except ValueError as exc:
-                    raise HTTPException(status_code=422, detail=f"invalid memory_defense policy: {exc}")
-
-            # Update config via config resolver (validates configurable fields and permissions)
-            await app.state.memory._config_resolver.update_bank_config(bank_id, request.updates, request_context)
-
-            # Return updated config
-            config_dict = await app.state.memory._config_resolver.get_bank_config(bank_id, request_context)
-            bank_overrides = await app.state.memory._config_resolver._load_bank_config(bank_id)
-
-            return BankConfigResponse(bank_id=bank_id, config=config_dict, overrides=bank_overrides)
+            state = await app.state.memory.update_bank_config(
+                bank_id,
+                request.updates,
+                request_context=request_context,
+            )
+            return BankConfigResponse(bank_id=bank_id, config=state.config, overrides=state.overrides)
         except OperationValidationError as e:
             raise HTTPException(status_code=e.status_code, detail=e.reason)
         except ValueError as e:
@@ -6421,26 +6600,8 @@ def _register_routes(app: FastAPI):
                 detail="Bank configuration API is disabled. Set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true to re-enable.",
             )
         try:
-            # Authenticate and set schema context for multi-tenant DB queries
-            await app.state.memory._authenticate_tenant(request_context)
-            if app.state.memory._operation_validator:
-                from hindsight_api.extensions import BankWriteContext, BankWriteOperation
-
-                ctx = BankWriteContext(
-                    bank_id=bank_id, operation=BankWriteOperation.RESET_BANK_CONFIG, request_context=request_context
-                )
-                await app.state.memory._validate_operation(
-                    app.state.memory._operation_validator.validate_bank_write(ctx)
-                )
-
-            # Reset config via config resolver
-            await app.state.memory._config_resolver.reset_bank_config(bank_id)
-
-            # Return updated config (should match defaults now)
-            config_dict = await app.state.memory._config_resolver.get_bank_config(bank_id, request_context)
-            bank_overrides = await app.state.memory._config_resolver._load_bank_config(bank_id)
-
-            return BankConfigResponse(bank_id=bank_id, config=config_dict, overrides=bank_overrides)
+            state = await app.state.memory.reset_bank_config(bank_id, request_context=request_context)
+            return BankConfigResponse(bank_id=bank_id, config=state.config, overrides=state.overrides)
         except OperationValidationError as e:
             raise HTTPException(status_code=e.status_code, detail=e.reason)
         except (AuthenticationError, HTTPException):
@@ -6833,6 +6994,11 @@ def _register_routes(app: FastAPI):
                 strategy_groups[effective].append(content_dict)
 
             if request.async_:
+                if request.operation_id is not None and len(strategy_groups) != 1:
+                    raise HTTPException(
+                        status_code=400,
+                        detail="operation_id requires all retain items to resolve to a single strategy",
+                    )
                 # Async processing: one submit per strategy group
                 all_operation_ids = []
                 total_items_count = 0
@@ -6843,6 +7009,7 @@ def _register_routes(app: FastAPI):
                         document_tags=request.document_tags,
                         strategy=group_strategy,
                         request_context=request_context,
+                        operation_id=request.operation_id,
                     )
                     all_operation_ids.append(result["operation_id"])
                     total_items_count += result["items_count"]
@@ -6909,6 +7076,10 @@ def _register_routes(app: FastAPI):
                 )
         except OperationValidationError as e:
             raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except RetainOperationConflictError as e:
+            # Caller reused an async retain operation_id that already belongs to
+            # a different operation.
+            raise HTTPException(status_code=409, detail=str(e))
         except (AuthenticationError, HTTPException):
             raise
         except ValueError as e:

hindsight-api-slim/hindsight_api/banner.py

@@ -66,11 +66,6 @@ def color_end(text: str) -> str:
     return color(text, 1.0)
 
 
-def color_mid(text: str) -> str:
-    """Color text with gradient middle color."""
-    return color(text, 0.5)
-
-
 def dim(text: str) -> str:
     """Dim/gray text."""
     return f"\033[38;2;128;128;128m{text}\033[0m"

hindsight-api-slim/hindsight_api/config.py

@@ -19,12 +19,28 @@ from ._pg_search import normalize_pg_search_tokenizer
 from ._vector_index import validate_extension
 from .utils import mask_network_location
 
-# Load .env file, searching current and parent directories (overrides existing env vars)
-load_dotenv(find_dotenv(usecwd=True), override=True)
-
 logger = logging.getLogger(__name__)
 
 
+def load_dotenv_for_entrypoint() -> None:
+    """Load a discovered ``.env`` file for Hindsight's own entry points.
+
+    Importing ``hindsight_api`` (or anything that pulls it in) must NOT mutate
+    the host application's ``os.environ``. See issue #2961: doing so at module
+    scope let an upward ``.env`` walk from the process cwd silently overwrite an
+    embedding application's own configuration.
+
+    This helper is therefore called explicitly from Hindsight's standalone entry
+    points only — the API server (CLI and ``hindsight_api.server:app``), the
+    worker, and the admin CLI. ``override=True`` is deliberate: it preserves the
+    exact precedence those entry points have always had (a discovered ``.env``
+    is authoritative over the ambient process environment). Because a library
+    import never reaches this code path, that precedence no longer leaks into
+    embedders.
+    """
+    load_dotenv(find_dotenv(usecwd=True), override=True)
+
+
 class ConfigFieldAccessError(AttributeError):
     """Raised when trying to access a bank-configurable field from global config."""
 
@@ -154,6 +170,7 @@ ENV_LLM_STRICT_SCHEMA = "HINDSIGHT_API_LLM_STRICT_SCHEMA"
 ENV_LLM_STRICT_SCHEMA_RETAIN = "HINDSIGHT_API_LLM_STRICT_SCHEMA_RETAIN"
 ENV_LLM_STRICT_SCHEMA_REFLECT = "HINDSIGHT_API_LLM_STRICT_SCHEMA_REFLECT"
 ENV_LLM_STRICT_SCHEMA_CONSOLIDATION = "HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION"
+ENV_LLM_SUPPORTS_MAX_ITEMS = "HINDSIGHT_API_LLM_SUPPORTS_MAX_ITEMS"
 ENV_LLM_SEND_BANK_AS_USER = "HINDSIGHT_API_LLM_SEND_BANK_AS_USER"
 ENV_LLM_OLLAMA_NUM_CTX = "HINDSIGHT_API_LLM_OLLAMA_NUM_CTX"
 
@@ -272,6 +289,20 @@ def _resolve_operation_strict_schema(operation_env: str) -> bool:
     return raw.strip().lower() in ("true", "1")
 
 
+def _parse_boolean_env(env_name: str, default: bool) -> bool:
+    """Parse a boolean environment variable, rejecting ambiguous values."""
+    raw = os.getenv(env_name)
+    if raw is None:
+        return default
+
+    normalized = raw.strip().lower()
+    if normalized in ("true", "1"):
+        return True
+    if normalized in ("false", "0"):
+        return False
+    raise ValueError(f"Invalid {env_name} value {raw!r}: expected true, false, 1, or 0")
+
+
 # Per-operation LLM configuration (optional, falls back to global LLM config)
 ENV_RETAIN_LLM_PROVIDER = "HINDSIGHT_API_RETAIN_LLM_PROVIDER"
 ENV_RETAIN_LLM_API_KEY = "HINDSIGHT_API_RETAIN_LLM_API_KEY"
@@ -283,6 +314,7 @@ ENV_RETAIN_LLM_INITIAL_BACKOFF = "HINDSIGHT_API_RETAIN_LLM_INITIAL_BACKOFF"
 ENV_RETAIN_LLM_MAX_BACKOFF = "HINDSIGHT_API_RETAIN_LLM_MAX_BACKOFF"
 ENV_RETAIN_LLM_TIMEOUT = "HINDSIGHT_API_RETAIN_LLM_TIMEOUT"
 ENV_RETAIN_LLM_LITELLMROUTER_CONFIG = "HINDSIGHT_API_RETAIN_LLM_LITELLMROUTER_CONFIG"
+ENV_RETAIN_LLM_REASONING_EFFORT = "HINDSIGHT_API_RETAIN_LLM_REASONING_EFFORT"
 
 # Fireworks AI batch inference. Fireworks' batch API is a proprietary
 # account-scoped dataset/job REST API on a control-plane host, distinct from the
@@ -305,6 +337,7 @@ ENV_REFLECT_LLM_INITIAL_BACKOFF = "HINDSIGHT_API_REFLECT_LLM_INITIAL_BACKOFF"
 ENV_REFLECT_LLM_MAX_BACKOFF = "HINDSIGHT_API_REFLECT_LLM_MAX_BACKOFF"
 ENV_REFLECT_LLM_TIMEOUT = "HINDSIGHT_API_REFLECT_LLM_TIMEOUT"
 ENV_REFLECT_LLM_LITELLMROUTER_CONFIG = "HINDSIGHT_API_REFLECT_LLM_LITELLMROUTER_CONFIG"
+ENV_REFLECT_LLM_REASONING_EFFORT = "HINDSIGHT_API_REFLECT_LLM_REASONING_EFFORT"
 
 ENV_CONSOLIDATION_LLM_PROVIDER = "HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER"
 ENV_CONSOLIDATION_LLM_API_KEY = "HINDSIGHT_API_CONSOLIDATION_LLM_API_KEY"
@@ -316,10 +349,12 @@ ENV_CONSOLIDATION_LLM_INITIAL_BACKOFF = "HINDSIGHT_API_CONSOLIDATION_LLM_INITIAL
 ENV_CONSOLIDATION_LLM_MAX_BACKOFF = "HINDSIGHT_API_CONSOLIDATION_LLM_MAX_BACKOFF"
 ENV_CONSOLIDATION_LLM_TIMEOUT = "HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT"
 ENV_CONSOLIDATION_LLM_LITELLMROUTER_CONFIG = "HINDSIGHT_API_CONSOLIDATION_LLM_LITELLMROUTER_CONFIG"
+ENV_CONSOLIDATION_LLM_REASONING_EFFORT = "HINDSIGHT_API_CONSOLIDATION_LLM_REASONING_EFFORT"
 
 ENV_EMBEDDINGS_PROVIDER = "HINDSIGHT_API_EMBEDDINGS_PROVIDER"
 ENV_EMBEDDINGS_LOCAL_MODEL = "HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL"
 ENV_EMBEDDINGS_LOCAL_FORCE_CPU = "HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"
+ENV_EMBEDDINGS_LOCAL_ALLOW_MPS = "HINDSIGHT_API_EMBEDDINGS_LOCAL_ALLOW_MPS"
 ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE = "HINDSIGHT_API_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE"
 ENV_EMBEDDINGS_ONNX_MODEL_ID = "HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_ID"
 ENV_EMBEDDINGS_ONNX_MODEL_PATH = "HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH"
@@ -410,6 +445,7 @@ ENV_RERANKER_PROVIDER = "HINDSIGHT_API_RERANKER_PROVIDER"
 ENV_RERANKER_SEND_BANK_AS_HEADER = "HINDSIGHT_API_RERANKER_SEND_BANK_AS_HEADER"
 ENV_RERANKER_LOCAL_MODEL = "HINDSIGHT_API_RERANKER_LOCAL_MODEL"
 ENV_RERANKER_LOCAL_FORCE_CPU = "HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"
+ENV_RERANKER_LOCAL_ALLOW_MPS = "HINDSIGHT_API_RERANKER_LOCAL_ALLOW_MPS"
 ENV_RERANKER_LOCAL_MAX_CONCURRENT = "HINDSIGHT_API_RERANKER_LOCAL_MAX_CONCURRENT"
 ENV_RERANKER_LOCAL_TRUST_REMOTE_CODE = "HINDSIGHT_API_RERANKER_LOCAL_TRUST_REMOTE_CODE"
 ENV_RERANKER_LOCAL_FP16 = "HINDSIGHT_API_RERANKER_LOCAL_FP16"
@@ -429,6 +465,9 @@ ENV_RERANKER_LITELLM_SDK_TIMEOUT = "HINDSIGHT_API_RERANKER_LITELLM_SDK_TIMEOUT"
 ENV_RERANKER_GOOGLE_TIMEOUT = "HINDSIGHT_API_RERANKER_GOOGLE_TIMEOUT"
 ENV_RERANKER_MAX_CANDIDATES = "HINDSIGHT_API_RERANKER_MAX_CANDIDATES"
 ENV_SEMANTIC_MIN_SIMILARITY = "HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY"
+ENV_GRAPH_SEED_MIN_SIMILARITY = "HINDSIGHT_API_GRAPH_SEED_MIN_SIMILARITY"
+ENV_TEMPORAL_SEMANTIC_MIN_SIMILARITY = "HINDSIGHT_API_TEMPORAL_SEMANTIC_MIN_SIMILARITY"
+ENV_SEMANTIC_LINK_MIN_SIMILARITY = "HINDSIGHT_API_SEMANTIC_LINK_MIN_SIMILARITY"
 ENV_RERANKER_FLASHRANK_MODEL = "HINDSIGHT_API_RERANKER_FLASHRANK_MODEL"
 ENV_RERANKER_FLASHRANK_CACHE_DIR = "HINDSIGHT_API_RERANKER_FLASHRANK_CACHE_DIR"
 ENV_RERANKER_FLASHRANK_CPU_MEM_ARENA = "HINDSIGHT_API_RERANKER_FLASHRANK_CPU_MEM_ARENA"
@@ -493,6 +532,12 @@ ENV_OTEL_DEPLOYMENT_ENVIRONMENT = "HINDSIGHT_API_OTEL_DEPLOYMENT_ENVIRONMENT"
 ENV_METRICS_INCLUDE_BANK_ID = "HINDSIGHT_API_METRICS_INCLUDE_BANK_ID"
 ENV_METRICS_BACKLOG_ENABLED = "HINDSIGHT_API_METRICS_BACKLOG_ENABLED"
 
+# Runtime-stall observability (loop watchdog + DB pool acquire instrumentation)
+ENV_LOOP_WATCHDOG_ENABLED = "HINDSIGHT_API_LOOP_WATCHDOG_ENABLED"
+ENV_LOOP_WATCHDOG_STALL_THRESHOLD_MS = "HINDSIGHT_API_LOOP_WATCHDOG_STALL_THRESHOLD_MS"
+ENV_LOOP_WATCHDOG_POLL_INTERVAL_MS = "HINDSIGHT_API_LOOP_WATCHDOG_POLL_INTERVAL_MS"
+ENV_DB_ACQUIRE_WARN_THRESHOLD_MS = "HINDSIGHT_API_DB_ACQUIRE_WARN_THRESHOLD_MS"
+
 # Vertex AI configuration
 ENV_LLM_VERTEXAI_PROJECT_ID = "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID"
 ENV_LLM_VERTEXAI_REGION = "HINDSIGHT_API_LLM_VERTEXAI_REGION"
@@ -633,21 +678,64 @@ ENV_WORKER_MAX_SLOTS = "HINDSIGHT_API_WORKER_MAX_SLOTS"
 ENV_OPERATION_RETENTION_DAYS = "HINDSIGHT_API_OPERATION_RETENTION_DAYS"
 ENV_OPERATION_CLEANUP_BATCH_SIZE = "HINDSIGHT_API_OPERATION_CLEANUP_BATCH_SIZE"
 
-# Per-operation-type slot reservations. Each entry maps an operation_type
-# (as stored in async_operations.operation_type) to its env var and default.
-# Adding a new operation type here is the ONLY change needed to make it
-# reservable via env var — config fields, from_env(), and the
-# worker_slot_reservations property all derive from this dict.
-WORKER_SLOT_RESERVATION_TYPES: dict[str, tuple[str, int]] = {
-    "consolidation": ("HINDSIGHT_API_WORKER_CONSOLIDATION_MAX_SLOTS", 2),
-    "retain": ("HINDSIGHT_API_WORKER_RETAIN_MAX_SLOTS", 0),
-    "file_convert_retain": ("HINDSIGHT_API_WORKER_FILE_CONVERT_RETAIN_MAX_SLOTS", 0),
-    "refresh_mental_model": ("HINDSIGHT_API_WORKER_REFRESH_MENTAL_MODEL_MAX_SLOTS", 0),
-    "graph_maintenance": ("HINDSIGHT_API_WORKER_GRAPH_MAINTENANCE_MAX_SLOTS", 0),
-    "import_documents": ("HINDSIGHT_API_WORKER_IMPORT_DOCUMENTS_MAX_SLOTS", 0),
+
+# Per-operation-type worker slot reservations: op_type -> default reserved count.
+# Each entry reserves a guaranteed *minimum* number of slots (a floor) for that
+# operation type within the global WORKER_MAX_SLOTS pool, so a saturated pool can't
+# starve it. Remaining capacity (WORKER_MAX_SLOTS - sum of reservations) is a shared
+# pool usable by any type, so a reservation does NOT cap the type — it may overflow
+# the shared pool. Adding a type here is the only change needed to make it reservable.
+#
+# op_type matches the value stored in async_operations.operation_type; the env var
+# names are derived from it (see _parse_worker_slot_reservations).
+WORKER_SLOT_TYPE_DEFAULTS: dict[str, int] = {
+    "consolidation": 2,
+    "retain": 0,
+    "file_convert_retain": 0,
+    "refresh_mental_model": 0,
+    "graph_maintenance": 0,
+    "import_documents": 0,
 }
+
+
+def _parse_worker_slot_reservations() -> dict[str, int]:
+    """Parse per-type RESERVED_SLOTS (and the deprecated _MAX_SLOTS alias).
+
+    ``HINDSIGHT_API_WORKER_<TYPE>_MAX_SLOTS`` is a deprecated alias for
+    ``..._RESERVED_SLOTS`` — despite its name it always set the reservation floor,
+    never a ceiling. It still works but logs a warning. Returns op_type -> reserved
+    floor for entries with a reservation > 0.
+    """
+    reservations: dict[str, int] = {}
+    for op_type, default in WORKER_SLOT_TYPE_DEFAULTS.items():
+        reserved_env = f"HINDSIGHT_API_WORKER_{op_type.upper()}_RESERVED_SLOTS"
+        legacy_env = f"HINDSIGHT_API_WORKER_{op_type.upper()}_MAX_SLOTS"
+        raw_reserved = os.getenv(reserved_env)
+        raw_legacy = os.getenv(legacy_env)
+        if raw_reserved is not None and raw_legacy is not None:
+            raise ValueError(
+                f"Both {reserved_env} and the deprecated {legacy_env} are set; "
+                f"they configure the same value. Keep only {reserved_env}."
+            )
+        if raw_legacy is not None:
+            logger.warning(
+                "%s is deprecated and will be removed in a future release. Despite its name it "
+                "reserves a *minimum* (floor), not a maximum. Rename it to %s.",
+                legacy_env,
+                reserved_env,
+            )
+        reserved_source = raw_reserved if raw_reserved is not None else raw_legacy
+        reserved = int(reserved_source) if reserved_source is not None else default
+        if reserved < 0:
+            raise ValueError(f"{reserved_env} must be >= 0, got {reserved}")
+        if reserved > 0:
+            reservations[op_type] = reserved
+    return reservations
+
+
 ENV_WORKER_CONSOLIDATION_BANK_PRIORITY = "HINDSIGHT_API_WORKER_CONSOLIDATION_BANK_PRIORITY"
 ENV_RETAIN_MAX_CONCURRENT = "HINDSIGHT_API_RETAIN_MAX_CONCURRENT"
+ENV_RETAIN_WALL_TIMEOUT = "HINDSIGHT_API_RETAIN_WALL_TIMEOUT"
 
 # Reflect agent settings
 ENV_REFLECT_MAX_ITERATIONS = "HINDSIGHT_API_REFLECT_MAX_ITERATIONS"
@@ -767,6 +855,7 @@ DEFAULT_LLAMACPP_EXTRA_ARGS = None  # Space-separated extra CLI args for llama.c
 # (prose preambles, markdown fences, invalid JSON) — wedging retain/consolidation
 # on parse retries.
 DEFAULT_LLM_STRICT_SCHEMA = False
+DEFAULT_LLM_SUPPORTS_MAX_ITEMS = True
 
 DEFAULT_LLM_MAX_CONCURRENT = 32
 DEFAULT_LLM_MAX_RETRIES = 3  # Max retry attempts for LLM API calls
@@ -787,6 +876,9 @@ DEFAULT_LLM_GEMINI_SAFETY_SETTINGS = None  # None = use Gemini default safety se
 DEFAULT_EMBEDDINGS_PROVIDER = "local"
 DEFAULT_EMBEDDINGS_LOCAL_MODEL = "BAAI/bge-small-en-v1.5"
 DEFAULT_EMBEDDINGS_LOCAL_FORCE_CPU = False  # Force CPU mode for local embeddings
+# Apple Silicon MPS is opt-in: it leaks memory under variable-length workloads
+# (unbounded per-shape kernel/allocator cache). CUDA/XPU still auto-select.
+DEFAULT_EMBEDDINGS_LOCAL_ALLOW_MPS = False
 DEFAULT_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE = False  # Security: disabled by default, required for some models
 DEFAULT_EMBEDDINGS_ONNX_MODEL_ID = "intfloat/multilingual-e5-small"
 DEFAULT_EMBEDDINGS_ONNX_FILE = "onnx/model.onnx"
@@ -806,6 +898,9 @@ DEFAULT_RERANKER_PROVIDER = "local"
 DEFAULT_RERANKER_SEND_BANK_AS_HEADER = False
 DEFAULT_RERANKER_LOCAL_MODEL = "cross-encoder/ms-marco-MiniLM-L-6-v2"
 DEFAULT_RERANKER_LOCAL_FORCE_CPU = False  # Force CPU mode for local reranker
+# Apple Silicon MPS is opt-in: it leaks memory under variable-length workloads
+# (unbounded per-shape kernel/allocator cache). CUDA/XPU still auto-select.
+DEFAULT_RERANKER_LOCAL_ALLOW_MPS = False
 DEFAULT_RERANKER_LOCAL_MAX_CONCURRENT = 4  # Limit concurrent CPU-bound reranking to prevent thrashing
 DEFAULT_RERANKER_LOCAL_TRUST_REMOTE_CODE = (
     False  # Security: disabled by default, required for some models like jina-reranker-v2
@@ -828,6 +923,9 @@ DEFAULT_RERANKER_LITELLM_SDK_TIMEOUT = 60.0
 DEFAULT_RERANKER_GOOGLE_TIMEOUT = 60.0
 DEFAULT_RERANKER_MAX_CANDIDATES = 300
 DEFAULT_SEMANTIC_MIN_SIMILARITY = 0.3
+DEFAULT_GRAPH_SEED_MIN_SIMILARITY = 0.3
+DEFAULT_TEMPORAL_SEMANTIC_MIN_SIMILARITY = 0.1
+DEFAULT_SEMANTIC_LINK_MIN_SIMILARITY = 0.7
 # Minimum BM25 score a row must exceed to enter fusion. 0.0 gates out
 # zero-score (non-matching) rows on backends — notably VectorChord — whose
 # operator ranks every document rather than pre-filtering to term matches.
@@ -1116,6 +1214,14 @@ DEFAULT_WORKER_MAX_SLOTS = 10  # Total concurrent tasks per worker
 DEFAULT_OPERATION_RETENTION_DAYS = 0
 DEFAULT_OPERATION_CLEANUP_BATCH_SIZE = 1000
 DEFAULT_RETAIN_MAX_CONCURRENT = 4  # Max concurrent retain DB phases (HNSW reads + writes). Limits I/O contention.
+# Wall-clock ceiling for one retain task in the worker (0 disables). This is a
+# deadlock/wedge backstop, not a latency target: a retain that blocks forever on
+# a lock, an LLM permit or a queue put would otherwise hold its worker slot until
+# the process restarts, and 'processing' is neither retryable nor cancellable
+# through the API. Set well above any healthy retain so it only ever fires on a
+# genuine wedge — the per-attempt LLM timeout and the retry budget already bound
+# the normal slow path.
+DEFAULT_RETAIN_WALL_TIMEOUT = 3600  # seconds (1 hour)
 
 # Reflect agent settings
 DEFAULT_REFLECT_MAX_ITERATIONS = 10  # Max tool call iterations before forcing response
@@ -1158,6 +1264,16 @@ DEFAULT_OTEL_DEPLOYMENT_ENVIRONMENT = "development"
 DEFAULT_METRICS_INCLUDE_BANK_ID = False  # Disabled by default to avoid high-cardinality OTel metric growth
 DEFAULT_METRICS_BACKLOG_ENABLED = False  # Disabled by default: runs periodic per-schema COUNT queries
 
+# Runtime-stall observability defaults. Both are cheap and on by default: the
+# watchdog is a single background thread pinging the loop; the DB-pool acquire
+# timing is a monotonic() delta per acquire. They turn a failing liveness probe
+# from "pod restarted, cause unknown" into a logged root cause (blocked loop vs
+# pool exhaustion).
+DEFAULT_LOOP_WATCHDOG_ENABLED = True
+DEFAULT_LOOP_WATCHDOG_STALL_THRESHOLD_MS = 1000  # log a stall once the loop is unresponsive this long
+DEFAULT_LOOP_WATCHDOG_POLL_INTERVAL_MS = 250  # how often the watchdog thread pings the loop
+DEFAULT_DB_ACQUIRE_WARN_THRESHOLD_MS = 1000  # log a warning when a pool acquire waits this long
+
 # Audit log defaults
 DEFAULT_AUDIT_LOG_ENABLED = False  # Disabled by default
 DEFAULT_AUDIT_LOG_ACTIONS = ""  # Empty = audit all eligible actions
@@ -1684,6 +1800,10 @@ class HindsightConfig:
     llm_strict_schema_retain: bool
     llm_strict_schema_reflect: bool
     llm_strict_schema_consolidation: bool
+    llm_supports_max_items: bool = field(
+        default=DEFAULT_LLM_SUPPORTS_MAX_ITEMS,
+        kw_only=True,
+    )  # Whether structured-output schemas accept JSON Schema maxItems
     # Tags outbound OpenAI-compatible LLM + embedding calls with `user=<bank_id>` for
     # per-bank cost attribution. Downstream cost gateways (OpenRouter usage accounting,
     # LiteLLM, Helicone) key attribution on the OpenAI `user` field. Opt-in; never
@@ -1742,6 +1862,7 @@ class HindsightConfig:
     retain_llm_max_backoff: float | None
     retain_llm_timeout: float | None
     retain_llm_litellmrouter_config: dict | None
+    retain_llm_reasoning_effort: str | None
 
     # Fireworks AI batch inference (static, server-level)
     fireworks_account_id: str | None
@@ -1758,6 +1879,7 @@ class HindsightConfig:
     reflect_llm_max_backoff: float | None
     reflect_llm_timeout: float | None
     reflect_llm_litellmrouter_config: dict | None
+    reflect_llm_reasoning_effort: str | None
 
     consolidation_llm_provider: str | None
     consolidation_llm_api_key: str | None
@@ -1769,11 +1891,13 @@ class HindsightConfig:
     consolidation_llm_max_backoff: float | None
     consolidation_llm_timeout: float | None
     consolidation_llm_litellmrouter_config: dict | None
+    consolidation_llm_reasoning_effort: str | None
 
     # Embeddings
     embeddings_provider: str
     embeddings_local_model: str
     embeddings_local_force_cpu: bool
+    embeddings_local_allow_mps: bool
     embeddings_local_trust_remote_code: bool
     embeddings_onnx_model_id: str
     embeddings_onnx_model_path: str | None
@@ -1819,6 +1943,7 @@ class HindsightConfig:
     reranker_send_bank_as_header: bool
     reranker_local_model: str
     reranker_local_force_cpu: bool
+    reranker_local_allow_mps: bool
     reranker_local_max_concurrent: int
     reranker_local_trust_remote_code: bool
     reranker_local_fp16: bool
@@ -1830,6 +1955,9 @@ class HindsightConfig:
     reranker_tei_http_timeout: float
     reranker_max_candidates: int
     semantic_min_similarity: float
+    graph_seed_min_similarity: float
+    temporal_semantic_min_similarity: float
+    semantic_link_min_similarity: float
     bm25_min_score: float
     recall_max_candidates_per_source: int
     recall_strategy_boosts: dict[str, str]
@@ -2033,6 +2161,7 @@ class HindsightConfig:
     operation_retention_days: int
     operation_cleanup_batch_size: int
     retain_max_concurrent: int
+    retain_wall_timeout: int
 
     # Reflect agent settings
     reflect_max_iterations: int
@@ -2049,6 +2178,12 @@ class HindsightConfig:
     metrics_include_bank_id: bool
     metrics_backlog_enabled: bool
 
+    # Runtime-stall observability (static, server-level only)
+    loop_watchdog_enabled: bool
+    loop_watchdog_stall_threshold_ms: int
+    loop_watchdog_poll_interval_ms: int
+    db_acquire_warn_threshold_ms: int
+
     # Audit log configuration
     # audit_log_enabled is hierarchical (env -> tenant -> bank): a deployment can
     # audit some banks and not others. The actions allowlist and retention window
@@ -2173,6 +2308,10 @@ class HindsightConfig:
         # Audit logging on/off, per bank. The actions allowlist and retention
         # window remain server-level and are deliberately not configurable.
         "audit_log_enabled",
+        # Persist raw source text (documents.original_text / chunks.chunk_text).
+        # Per-bank so a data-minimizing bank can keep only derived facts while
+        # others retain the raw source for expansion/re-extraction.
+        "store_document_text",
         # Retention settings (behavioral)
         "retain_chunk_size",
         "retain_structured_chunk_size",
@@ -2311,10 +2450,15 @@ class HindsightConfig:
             self.text_search_extension_pg_search_tokenizer
         )
 
-        if not 0.0 <= self.semantic_min_similarity <= 1.0:
-            raise ValueError(
-                f"Invalid semantic_min_similarity: {self.semantic_min_similarity}. Must be between 0.0 and 1.0"
-            )
+        for field_name in (
+            "semantic_min_similarity",
+            "graph_seed_min_similarity",
+            "temporal_semantic_min_similarity",
+            "semantic_link_min_similarity",
+        ):
+            value = getattr(self, field_name)
+            if not 0.0 <= value <= 1.0:
+                raise ValueError(f"Invalid {field_name}: {value}. Must be between 0.0 and 1.0")
 
         if self.bm25_max_query_terms < 0:
             raise ValueError(f"Invalid bm25_max_query_terms: {self.bm25_max_query_terms}. Must be >= 0")
@@ -2422,6 +2566,9 @@ class HindsightConfig:
         llm_provider = os.getenv(ENV_LLM_PROVIDER, DEFAULT_LLM_PROVIDER)
         llm_model = os.getenv(ENV_LLM_MODEL) or _get_default_model_for_provider(llm_provider)
 
+        # Parse per-type worker slot reservations (floors) once.
+        worker_slot_reservations = _parse_worker_slot_reservations()
+
         config = cls(
             # Database
             database_backend=os.getenv(ENV_DATABASE_BACKEND, DEFAULT_DATABASE_BACKEND).lower(),
@@ -2467,6 +2614,10 @@ class HindsightConfig:
             llm_strict_schema_retain=_resolve_operation_strict_schema(ENV_LLM_STRICT_SCHEMA_RETAIN),
             llm_strict_schema_reflect=_resolve_operation_strict_schema(ENV_LLM_STRICT_SCHEMA_REFLECT),
             llm_strict_schema_consolidation=_resolve_operation_strict_schema(ENV_LLM_STRICT_SCHEMA_CONSOLIDATION),
+            llm_supports_max_items=_parse_boolean_env(
+                ENV_LLM_SUPPORTS_MAX_ITEMS,
+                DEFAULT_LLM_SUPPORTS_MAX_ITEMS,
+            ),
             llm_send_bank_as_user=os.getenv(ENV_LLM_SEND_BANK_AS_USER, str(DEFAULT_LLM_SEND_BANK_AS_USER)).lower()
             in ("true", "1"),
             llm_ollama_num_ctx=_parse_optional_positive_int(
@@ -2536,6 +2687,7 @@ class HindsightConfig:
             else None,
             retain_llm_timeout=float(os.getenv(ENV_RETAIN_LLM_TIMEOUT)) if os.getenv(ENV_RETAIN_LLM_TIMEOUT) else None,
             retain_llm_litellmrouter_config=_parse_llm_router_config(ENV_RETAIN_LLM_LITELLMROUTER_CONFIG),
+            retain_llm_reasoning_effort=os.getenv(ENV_RETAIN_LLM_REASONING_EFFORT) or None,
             reflect_llm_provider=os.getenv(ENV_REFLECT_LLM_PROVIDER) or None,
             reflect_llm_api_key=os.getenv(ENV_REFLECT_LLM_API_KEY) or None,
             reflect_llm_model=os.getenv(ENV_REFLECT_LLM_MODEL)
@@ -2561,6 +2713,7 @@ class HindsightConfig:
             if os.getenv(ENV_REFLECT_LLM_TIMEOUT)
             else None,
             reflect_llm_litellmrouter_config=_parse_llm_router_config(ENV_REFLECT_LLM_LITELLMROUTER_CONFIG),
+            reflect_llm_reasoning_effort=os.getenv(ENV_REFLECT_LLM_REASONING_EFFORT) or None,
             consolidation_llm_provider=os.getenv(ENV_CONSOLIDATION_LLM_PROVIDER) or None,
             consolidation_llm_api_key=os.getenv(ENV_CONSOLIDATION_LLM_API_KEY) or None,
             consolidation_llm_model=os.getenv(ENV_CONSOLIDATION_LLM_MODEL)
@@ -2586,6 +2739,7 @@ class HindsightConfig:
             if os.getenv(ENV_CONSOLIDATION_LLM_TIMEOUT)
             else None,
             consolidation_llm_litellmrouter_config=_parse_llm_router_config(ENV_CONSOLIDATION_LLM_LITELLMROUTER_CONFIG),
+            consolidation_llm_reasoning_effort=os.getenv(ENV_CONSOLIDATION_LLM_REASONING_EFFORT) or None,
             # Multi-LLM chains (indexed members + routing strategy)
             llm_members=_parse_llm_members(""),
             llm_strategy=_parse_llm_strategy(os.getenv(ENV_LLM_STRATEGY)),
@@ -2602,6 +2756,10 @@ class HindsightConfig:
                 ENV_EMBEDDINGS_LOCAL_FORCE_CPU, str(DEFAULT_EMBEDDINGS_LOCAL_FORCE_CPU)
             ).lower()
             in ("true", "1"),
+            embeddings_local_allow_mps=os.getenv(
+                ENV_EMBEDDINGS_LOCAL_ALLOW_MPS, str(DEFAULT_EMBEDDINGS_LOCAL_ALLOW_MPS)
+            ).lower()
+            in ("true", "1"),
             embeddings_local_trust_remote_code=os.getenv(
                 ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE, str(DEFAULT_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE)
             ).lower()
@@ -2745,6 +2903,10 @@ class HindsightConfig:
                 ENV_RERANKER_LOCAL_FORCE_CPU, str(DEFAULT_RERANKER_LOCAL_FORCE_CPU)
             ).lower()
             in ("true", "1"),
+            reranker_local_allow_mps=os.getenv(
+                ENV_RERANKER_LOCAL_ALLOW_MPS, str(DEFAULT_RERANKER_LOCAL_ALLOW_MPS)
+            ).lower()
+            in ("true", "1"),
             reranker_local_max_concurrent=int(
                 os.getenv(ENV_RERANKER_LOCAL_MAX_CONCURRENT, str(DEFAULT_RERANKER_LOCAL_MAX_CONCURRENT))
             ),
@@ -2771,6 +2933,15 @@ class HindsightConfig:
             ),
             reranker_max_candidates=int(os.getenv(ENV_RERANKER_MAX_CANDIDATES, str(DEFAULT_RERANKER_MAX_CANDIDATES))),
             semantic_min_similarity=float(os.getenv(ENV_SEMANTIC_MIN_SIMILARITY, str(DEFAULT_SEMANTIC_MIN_SIMILARITY))),
+            graph_seed_min_similarity=float(
+                os.getenv(ENV_GRAPH_SEED_MIN_SIMILARITY, str(DEFAULT_GRAPH_SEED_MIN_SIMILARITY))
+            ),
+            temporal_semantic_min_similarity=float(
+                os.getenv(ENV_TEMPORAL_SEMANTIC_MIN_SIMILARITY, str(DEFAULT_TEMPORAL_SEMANTIC_MIN_SIMILARITY))
+            ),
+            semantic_link_min_similarity=float(
+                os.getenv(ENV_SEMANTIC_LINK_MIN_SIMILARITY, str(DEFAULT_SEMANTIC_LINK_MIN_SIMILARITY))
+            ),
             bm25_min_score=float(os.getenv(ENV_BM25_MIN_SCORE, str(DEFAULT_BM25_MIN_SCORE))),
             bm25_max_query_terms=_parse_non_negative_int(
                 ENV_BM25_MAX_QUERY_TERMS,
@@ -3089,11 +3260,7 @@ class HindsightConfig:
             ),
             worker_http_port=int(os.getenv(ENV_WORKER_HTTP_PORT, str(DEFAULT_WORKER_HTTP_PORT))),
             worker_max_slots=int(os.getenv(ENV_WORKER_MAX_SLOTS, str(DEFAULT_WORKER_MAX_SLOTS))),
-            worker_slot_reservations={
-                op_type: int(os.getenv(env_var, str(default)))
-                for op_type, (env_var, default) in WORKER_SLOT_RESERVATION_TYPES.items()
-                if int(os.getenv(env_var, str(default))) > 0
-            },
+            worker_slot_reservations=worker_slot_reservations,
             worker_consolidation_bank_priority=_parse_bank_priority(
                 os.getenv(ENV_WORKER_CONSOLIDATION_BANK_PRIORITY, "")
             ),
@@ -3108,6 +3275,7 @@ class HindsightConfig:
                 DEFAULT_OPERATION_CLEANUP_BATCH_SIZE,
             ),
             retain_max_concurrent=int(os.getenv(ENV_RETAIN_MAX_CONCURRENT, str(DEFAULT_RETAIN_MAX_CONCURRENT))),
+            retain_wall_timeout=int(os.getenv(ENV_RETAIN_WALL_TIMEOUT, str(DEFAULT_RETAIN_WALL_TIMEOUT))),
             # Reflect agent settings
             reflect_max_iterations=int(os.getenv(ENV_REFLECT_MAX_ITERATIONS, str(DEFAULT_REFLECT_MAX_ITERATIONS))),
             reflect_prompt_cache_enabled=os.getenv(
@@ -3168,6 +3336,18 @@ class HindsightConfig:
             in ("true", "1", "yes"),
             metrics_backlog_enabled=os.getenv(ENV_METRICS_BACKLOG_ENABLED, str(DEFAULT_METRICS_BACKLOG_ENABLED)).lower()
             in ("true", "1", "yes"),
+            # Runtime-stall observability (static, server-level only)
+            loop_watchdog_enabled=os.getenv(ENV_LOOP_WATCHDOG_ENABLED, str(DEFAULT_LOOP_WATCHDOG_ENABLED)).lower()
+            in ("true", "1", "yes"),
+            loop_watchdog_stall_threshold_ms=int(
+                os.getenv(ENV_LOOP_WATCHDOG_STALL_THRESHOLD_MS, str(DEFAULT_LOOP_WATCHDOG_STALL_THRESHOLD_MS))
+            ),
+            loop_watchdog_poll_interval_ms=int(
+                os.getenv(ENV_LOOP_WATCHDOG_POLL_INTERVAL_MS, str(DEFAULT_LOOP_WATCHDOG_POLL_INTERVAL_MS))
+            ),
+            db_acquire_warn_threshold_ms=int(
+                os.getenv(ENV_DB_ACQUIRE_WARN_THRESHOLD_MS, str(DEFAULT_DB_ACQUIRE_WARN_THRESHOLD_MS))
+            ),
             # Audit log configuration (static, server-level only)
             audit_log_enabled=os.getenv(ENV_AUDIT_LOG_ENABLED, str(DEFAULT_AUDIT_LOG_ENABLED)).lower() == "true",
             audit_log_actions=[

hindsight-api-slim/hindsight_api/config_resolver.py

@@ -331,11 +331,17 @@ class ConfigResolver:
             logger.error(f"Failed to bulk-load bank configs: {e}")
         return result
 
-    async def update_bank_config(
-        self, bank_id: str, updates: dict[str, Any], context: RequestContext | None = None
-    ) -> None:
+    async def validate_bank_config_updates(
+        self,
+        bank_id: str,
+        updates: dict[str, Any],
+        context: RequestContext | None = None,
+        *,
+        projected_bank_overrides: dict[str, Any] | None = None,
+        check_permissions: bool = True,
+    ) -> dict[str, Any]:
         """
-        Update bank configuration overrides (with permission checking).
+        Normalize and validate bank configuration overrides.
 
         Args:
             bank_id: Bank identifier
@@ -344,9 +350,16 @@ class ConfigResolver:
                     or Python field format (llm_provider).
                     Only configurable fields are allowed.
             context: Request context for permission checking
+            projected_bank_overrides: Bank overrides to use as the validation
+                base instead of loading the current bank row.
+            check_permissions: Whether client field permissions apply to these
+                updates. Server-owned projected values set this to false.
+
+        Returns:
+            Normalized updates ready to persist.
 
         Raises:
-            ValueError: If attempting to override invalid/disallowed fields
+            ValueError: If attempting to override invalid/disallowed fields.
         """
         # Normalize keys
         normalized_updates = normalize_config_dict(updates)
@@ -378,7 +391,7 @@ class ConfigResolver:
                 )
 
         # PERMISSIONS: Check tenant/bank permissions
-        if self.tenant_extension and context:
+        if check_permissions and self.tenant_extension and context:
             try:
                 allowed_fields = await self.tenant_extension.get_allowed_config_fields(context, bank_id)
                 if allowed_fields is not None:  # None means "allow all"
@@ -427,7 +440,11 @@ class ConfigResolver:
         )
         if chunking_fields_updated:
             config_dict = await self._resolve_parent_config_dict(bank_id, context)
-            active_bank_overrides = await self._load_bank_config(bank_id)
+            active_bank_overrides = (
+                await self._load_bank_config(bank_id)
+                if projected_bank_overrides is None
+                else dict(projected_bank_overrides)
+            )
             for key, value in normalized_updates.items():
                 if key not in self._configurable_fields:
                     continue
@@ -443,17 +460,26 @@ class ConfigResolver:
             )
             _validate_retain_strategy_chunking(base_config, base_config.retain_strategies)
 
-        # Persist the override. Banks are created lazily (on first retain), so a
-        # PATCH that precedes any ingestion would otherwise UPDATE zero rows and
-        # silently no-op while returning 200. Ensure the bank row exists first
-        # (this also creates its per-bank vector indexes), then merge defensively:
-        # COALESCE guards against a NULL config column (NULL || jsonb is NULL),
-        # which would drop the override even when a row is updated.
-        from .engine.retain.fact_storage import ensure_bank_exists
+        return normalized_updates
 
+    async def update_bank_config(
+        self, bank_id: str, updates: dict[str, Any], context: RequestContext | None = None
+    ) -> None:
+        """Validate and persist bank configuration overrides for an existing bank.
+
+        Bank creation belongs to ``MemoryEngine``; this raises ``ValueError`` if
+        the bank does not exist rather than silently discarding the overrides.
+        """
+        normalized_updates = await self.validate_bank_config_updates(bank_id, updates, context)
+        await self._persist_bank_config(bank_id, normalized_updates)
+
+    async def _persist_bank_config(self, bank_id: str, normalized_updates: dict[str, Any]) -> None:
+        """Persist already-validated overrides without changing bank lifecycle state."""
+        # Bank lifecycle belongs to MemoryEngine. Callers must create the row
+        # before reaching this persistence step. COALESCE guards against a NULL
+        # config column (NULL || jsonb is NULL), which would drop the override.
         async with self._backend.acquire() as conn:
-            await ensure_bank_exists(conn, bank_id, ops=self._backend.ops)
-            await conn.execute(
+            result = await conn.execute(
                 f"""
                 UPDATE {fq_table("banks")}
                 SET config = COALESCE(config, '{{}}'::jsonb) || $1::jsonb,
@@ -464,6 +490,14 @@ class ConfigResolver:
                 bank_id,
             )
 
+        # A missing bank row matches zero rows, which would otherwise persist
+        # nothing while reporting success. Fail loudly instead: reaching here
+        # without the row means a caller skipped the engine's provisioning step.
+        # (The Oracle wrapper reshapes rowcount into the same "UPDATE <n>" form.)
+        updated = int(result.split()[-1]) if isinstance(result, str) and result.startswith("UPDATE") else 0
+        if updated == 0:
+            raise ValueError(f"Cannot update config for bank '{bank_id}': the bank does not exist")
+
         logger.info(f"Updated bank config for {bank_id}: {list(normalized_updates.keys())}")
 
     async def reset_bank_config(self, bank_id: str) -> None:

hindsight-api-slim/hindsight_api/daemon.py

@@ -14,10 +14,7 @@ import subprocess
 import sys
 import time
 from pathlib import Path
-from typing import TYPE_CHECKING
-
-if TYPE_CHECKING:
-    from typing import IO
+from typing import IO
 
 logger = logging.getLogger(__name__)
 
@@ -42,37 +39,28 @@ class IdleTimeoutMiddleware:
         self.app = app
         self.idle_timeout = idle_timeout
         self.last_activity = time.time()
-        self._checker_task = None
 
     async def __call__(self, scope, receive, send):
-        # Update activity timestamp on each request
         self.last_activity = time.time()
         await self.app(scope, receive, send)
 
-    def start_idle_checker(self):
-        """Start the background task that checks for idle timeout."""
-        self._checker_task = asyncio.create_task(self._check_idle())
-
     async def _check_idle(self):
-        """Background task that exits the process after idle timeout."""
-        # If idle_timeout is 0, don't auto-exit
+        """Exit the daemon after the configured period without requests."""
         if self.idle_timeout <= 0:
             return
 
         while True:
-            await asyncio.sleep(30)  # Check every 30 seconds
+            await asyncio.sleep(30)
             idle_time = time.time() - self.last_activity
             if idle_time > self.idle_timeout:
                 logger.info(f"Idle timeout reached ({self.idle_timeout}s), shutting down daemon")
-                # Give a moment for any in-flight requests
                 await asyncio.sleep(1)
-                # Send SIGTERM to ourselves to trigger graceful shutdown
                 import signal
 
                 os.kill(os.getpid(), signal.SIGTERM)
 
 
-def _detach_popen_kwargs(log_handle: "IO[bytes]") -> dict:
+def _detach_popen_kwargs(log_handle: IO[bytes]) -> dict:
     """Cross-platform kwargs to spawn a subprocess detached from the caller.
 
     On POSIX, ``start_new_session=True`` calls ``setsid(2)`` so the child
@@ -169,17 +157,3 @@ def daemonize():
         subprocess.Popen(cmd, env=env, **_detach_popen_kwargs(log_handle))
 
     sys.exit(0)
-
-
-def check_daemon_running(port: int = DEFAULT_DAEMON_PORT) -> bool:
-    """Check if a daemon is running and responsive on the given port."""
-    import socket
-
-    try:
-        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        sock.settimeout(1)
-        result = sock.connect_ex(("127.0.0.1", port))
-        sock.close()
-        return result == 0
-    except Exception:
-        return False

hindsight-api-slim/hindsight_api/engine/__init__.py

@@ -3,7 +3,7 @@ Memory Engine - Core implementation of the memory system.
 
 This package contains all the implementation details of the memory engine:
 - MemoryEngine: Main class for memory operations
-- Utility modules: embedding_utils, link_utils, think_utils, bank_utils
+- Utility modules: embedding_utils, link_utils, bank_utils
 - Supporting modules: embeddings, cross_encoder, entity_resolver, etc.
 """
 

hindsight-api-slim/hindsight_api/engine/audit.py

@@ -20,6 +20,7 @@ from pydantic import BaseModel, Field
 
 from ..engine.db_utils import acquire_with_retry
 from ..models import RequestContext
+from .schema import fq_table_explicit
 
 logger = logging.getLogger(__name__)
 
@@ -188,8 +189,12 @@ class AuditLogger:
             logger.debug("Audit log skipped: pool not available")
             return
         try:
-            schema = self._schema_getter()
-            table = f"{schema}.audit_log"
+            # fq_table_explicit qualifies per dialect: "schema".audit_log on
+            # PostgreSQL, bare audit_log on Oracle (where the schema is set at the
+            # session level). A raw f"{schema}.audit_log" produced public.audit_log
+            # on Oracle, where "public" is a reserved word — every write failed
+            # with ORA-00903 even though the table exists.
+            table = fq_table_explicit("audit_log", self._schema_getter())
             async with acquire_with_retry(pool, max_retries=1) as conn:
                 await conn.execute(
                     f"""

hindsight-api-slim/hindsight_api/engine/causal_links.py

@@ -5,9 +5,66 @@ preserves historical relationship types so existing banks keep their graph
 semantics without allowing new retain output to create those types.
 """
 
+from dataclasses import dataclass
+from typing import Any
+
 CANONICAL_CAUSAL_LINK_TYPE = "caused_by"
 LEGACY_CAUSAL_LINK_TYPE_NAMES = ("causes", "enables", "prevents")
 
 CANONICAL_CAUSAL_LINK_TYPES = frozenset({CANONICAL_CAUSAL_LINK_TYPE})
 LEGACY_CAUSAL_LINK_TYPES = frozenset(LEGACY_CAUSAL_LINK_TYPE_NAMES)
 CAUSAL_LINK_TYPES = (CANONICAL_CAUSAL_LINK_TYPE, *LEGACY_CAUSAL_LINK_TYPE_NAMES)
+
+DEFAULT_CAUSAL_LINK_WEIGHT = 1.0
+
+
+@dataclass(frozen=True)
+class CausalLinkDescriptor:
+    """One causal edge, parked on the curation archive while an endpoint is invalidated.
+
+    Invalidation moves a fact out of ``memory_units``, so the FK cascade deletes
+    its ``memory_links`` rows — and nothing could recreate a causal edge, which
+    is extraction output rather than derived data. The descriptor is what the
+    archive row stores so revert can rematerialize the edge (#2864).
+    """
+
+    from_unit_id: str
+    to_unit_id: str
+    link_type: str
+    weight: float = DEFAULT_CAUSAL_LINK_WEIGHT
+
+    def as_json_dict(self) -> dict[str, Any]:
+        """Serializable form written to ``invalidated_memory_units.causal_links``.
+
+        The key names double as the column list of the ``jsonb_to_recordset``
+        read in ``snapshot_causal_links`` — keep them in sync.
+        """
+        return {
+            "from_unit_id": self.from_unit_id,
+            "to_unit_id": self.to_unit_id,
+            "link_type": self.link_type,
+            "weight": self.weight,
+        }
+
+    @classmethod
+    def from_json_dict(cls, raw: Any) -> "CausalLinkDescriptor | None":
+        """Parse one stored descriptor, or None when it isn't a usable causal edge.
+
+        The archive column is plain JSON with no schema enforcement (a restore
+        from an older backup, or a hand-edited row, can put anything there), and
+        ``memory_links`` has a ``link_type`` CHECK constraint — so an unusable
+        entry is skipped rather than allowed to abort the whole revert.
+        """
+        if not isinstance(raw, dict):
+            return None
+        from_unit_id = raw.get("from_unit_id")
+        to_unit_id = raw.get("to_unit_id")
+        link_type = raw.get("link_type")
+        if not from_unit_id or not to_unit_id or link_type not in CAUSAL_LINK_TYPES:
+            return None
+        return cls(
+            from_unit_id=str(from_unit_id),
+            to_unit_id=str(to_unit_id),
+            link_type=str(link_type),
+            weight=float(raw.get("weight") or DEFAULT_CAUSAL_LINK_WEIGHT),
+        )

hindsight-api-slim/hindsight_api/engine/consolidation/consolidator.py

@@ -59,6 +59,25 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
+def _native_search_vector_update(config, param: str) -> str:
+    """UPDATE-clause fragment that repopulates ``search_vector`` inline, or ''
+    when the backend does not maintain a native tsvector column that way.
+
+    ``to_tsvector(...)::regconfig`` is PostgreSQL-only. On Oracle ``search_vector``
+    is a CLOB maintained by Oracle's own text index rather than an inline
+    tsvector, so emit nothing there (mirrors the insert path, which gates
+    ``search_vector`` on the PG-only ``pg_search_vector_expr``). Without this
+    guard the PG expression reaches Oracle and fails with DPY-4010 (the
+    ``::regconfig`` cast becomes an unbound ``:REGCONFIG`` placeholder).
+    """
+    from ..schema import _is_oracle  # noqa: PLC0415
+
+    if config.text_search_extension != "native" or _is_oracle():
+        return ""
+    lang = config.text_search_extension_native_language
+    return f",\n            search_vector = to_tsvector('{lang}'::regconfig, COALESCE({param}, ''))"
+
+
 def _norm_obs_text(text: str) -> str:
     """Whitespace-normalised observation text for exact-duplicate matching.
 
@@ -209,7 +228,7 @@ async def _dedup_adjudicate(
     grouped = await retrieve_semantic_bm25_combined(
         conn, anchor_emb_str, anchor_text, bank_id, ["observation"], _DEDUP_TOP_K, tags=tags, tags_match=tags_match
     )
-    results = grouped.get("observation", ([], []))[0]
+    results = grouped["observation"].semantic
     best_id: str | None = None
     best_text = ""
     best_sim = threshold  # only candidates at/above the threshold are considered
@@ -262,11 +281,7 @@ async def _dedup_reconcile_create(
     # Fold the new source facts into the twin and persist the merged text. We keep the twin's
     # existing embedding: the merged text is >= threshold similar, so the stored vector stays
     # representative and we avoid a re-embed + a dialect-specific vector UPDATE.
-    search_vector_clause = (
-        f",\n            search_vector = to_tsvector('{config.text_search_extension_native_language}'::regconfig, COALESCE($1, ''))"
-        if config.text_search_extension == "native"
-        else ""
-    )
+    search_vector_clause = _native_search_vector_update(config, "$1")
     await conn.execute(
         f"""
         UPDATE {fq_table("memory_units")}
@@ -322,11 +337,7 @@ async def _dedup_reconcile_update(
     # the create path) then delete the now-redundant updated row. The all_strict/any tag match
     # guarantees twin and updated share scope, so dropping the updated row's tags loses no
     # visibility. Temporal fields follow the surviving twin (minimal scope; matches create).
-    search_vector_clause = (
-        f",\n            search_vector = to_tsvector('{config.text_search_extension_native_language}'::regconfig, COALESCE($1, ''))"
-        if config.text_search_extension == "native"
-        else ""
-    )
+    search_vector_clause = _native_search_vector_update(config, "$1")
     await conn.execute(
         f"""
         UPDATE {fq_table("memory_units")} t
@@ -679,9 +690,19 @@ def _effective_scope_limit(config: Any, fact_tags: list[str]) -> int:
     return config.max_observations_per_scope
 
 
-def _build_response_model(max_creates: int | None = None) -> type[_ConsolidationBatchResponse]:
-    """Build a response model, optionally constraining max creates via JSON schema."""
-    if max_creates is None or max_creates < 0:
+def _build_response_model(
+    max_creates: int | None = None,
+    *,
+    supports_max_items: bool = True,
+) -> type[_ConsolidationBatchResponse]:
+    """Build a response model, optionally constraining creates via JSON schema.
+
+    Some structured-output backends (notably Bedrock Converse) reject the JSON
+    Schema ``maxItems`` keyword emitted by Pydantic's list ``max_length``. Operators
+    can disable the schema hint for those backends; the prompt capacity note and
+    post-response truncation still enforce the observation cap.
+    """
+    if not supports_max_items or max_creates is None or max_creates < 0:
         return _ConsolidationBatchResponse
 
     from pydantic import Field as PydanticField
@@ -1900,11 +1921,7 @@ async def _execute_update_action(
 
     config = get_config()
 
-    search_vector_clause = (
-        f",\n            search_vector = to_tsvector('{config.text_search_extension_native_language}'::regconfig, COALESCE($1, ''))"
-        if config.text_search_extension == "native"
-        else ""
-    )
+    search_vector_clause = _native_search_vector_update(config, "$1")
 
     t0 = time.time()
     await conn.execute(
@@ -2020,30 +2037,6 @@ async def _execute_delete_action(
     logger.debug(f"Deleted observation {observation_id}")
 
 
-async def _create_memory_links(
-    conn: "Connection",
-    memory_id: uuid.UUID,
-    observation_id: uuid.UUID,
-) -> None:
-    """
-    Placeholder for observation link creation.
-
-    Observations do NOT get any memory_links copied from their source facts.
-    Instead, retrieval uses source_memory_ids to traverse:
-    - Entity connections: observation → source_memory_ids → unit_entities
-    - Semantic similarity: observations have their own embeddings
-    - Temporal proximity: observations have their own temporal fields
-
-    This avoids data duplication and ensures observations are always
-    connected via their source facts' relationships.
-
-    The memory_id and observation_id parameters are kept for interface
-    compatibility but no links are created.
-    """
-    # No links are created - observations rely on source_memory_ids for traversal
-    pass
-
-
 async def _find_related_observations(
     memory_engine: "MemoryEngine",
     bank_id: str,
@@ -2275,7 +2268,10 @@ async def _consolidate_batch_with_llm(
             cached_prefix_name = None
 
     # Use a constrained response model when observation limit is active
-    response_model = _build_response_model(max_creates=remaining_observation_slots)
+    response_model = _build_response_model(
+        max_creates=remaining_observation_slots,
+        supports_max_items=config.llm_supports_max_items,
+    )
 
     max_attempts = config.consolidation_max_attempts
     inner_max_retries = config.consolidation_llm_max_retries
@@ -2387,6 +2383,8 @@ async def _create_observation_directly(
     observation_id = uuid.uuid4()
 
     # Query varies based on text search backend
+    from ..schema import _is_oracle  # noqa: PLC0415
+
     config = get_config()
     if config.text_search_extension == "vchord":
         # VectorChord: manually tokenize and insert search_vector
@@ -2399,10 +2397,12 @@ async def _create_observation_directly(
                     tokenize($3, 'llmlingua2')::bm25_catalog.bm25vector)
             RETURNING id
         """
-    elif config.text_search_extension == "native":
-        # Native: search_vector is populated with to_tsvector() using the
-        # configured native language dictionary, matching the batch insert
-        # path in ops_postgresql.insert_facts_batch.
+    elif config.text_search_extension == "native" and not _is_oracle():
+        # Native (PostgreSQL): search_vector is populated with to_tsvector()
+        # using the configured native language dictionary, matching the batch
+        # insert path in ops_postgresql.insert_facts_batch. On Oracle this falls
+        # through to the no-search_vector branch below (Oracle maintains its text
+        # index separately; to_tsvector/::regconfig is PG-only).
         query = f"""
             INSERT INTO {fq_table("memory_units")} (
                 id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids,

hindsight-api-slim/hindsight_api/engine/consolidation/prompts.py

@@ -37,19 +37,36 @@ _PROCESSING_RULES = """## PROCESSING RULES
 
 9. KEEP DISTINCT TOPICS DISTINCT: do not merge observations about different people, entities, or unrelated topics. Merging is for the same canonical fact recurring — not for related-but-distinct claims."""
 
+# Field-by-field definitions of the input shape used by the cached system
+# prefix. The call site runs .format(), so these strings must contain no braces.
+_FACT_FIELDS = """One per line, formatted as `[uuid] fact text (temporal fields)`:
+- `[uuid]`: the fact's identifier — copy it verbatim into `source_fact_ids`
+- `occurred_start` / `occurred_end`: when the described event happened. This can be long before the fact was stated — a fact recorded today may describe a 2019 event.
+- `mentioned_at`: when the source material that states this fact was written. This is the fact's recency: how up to date the statement is, NOT when it was added to memory. A fact taken from an old document keeps its old `mentioned_at` even if it was only just processed."""
+
+_OBSERVATION_FIELDS = """- `id`: unique identifier — copy this exactly when issuing an UPDATE or DELETE
+- `text`: the observation content
+- `proof_count`: how many source facts this observation has already merged
+- `occurred_start` / `occurred_end`: the span of the events behind the observation — earliest start and latest end across its source facts
+- `mentioned_at`: the latest of the `mentioned_at` values of its source facts — the most recent point at which this observation was stated
+- `source_memories`: the supporting facts behind this observation. May be partial or absent for large observations — the count above remains the true total. Each entry carries the same `text` and temporal fields as a new fact, plus:
+  - `context`: optional surrounding context for that fact"""
+
 # Stable description of the input shape. For the cached split path this lives in
 # the system prefix (build_consolidation_system_prompt) so it is not re-sent on
 # every batch; the per-batch user message then carries only the actual data.
-_INPUT_FORMAT_NOTE = """## INPUT FORMAT
+_INPUT_FORMAT_NOTE = f"""## INPUT FORMAT
 
-Each request provides new facts and existing observations:
-- New facts: one per line, each prefixed with its `[uuid]`, followed by the fact text and optional temporal fields.
-- Existing observations: a JSON array pooled from recalls across the new facts. Each entry has:
-  - `id`: unique identifier — copy this exactly when issuing an UPDATE or DELETE
-  - `text`: the observation content
-  - `proof_count`: number of supporting memories
-  - `occurred_start` / `occurred_end`: temporal range of source facts
-  - `source_memories`: array of supporting facts with their text and dates"""
+Each request provides new facts and existing observations. Every temporal field is optional and is omitted when unknown.
+
+### New facts
+
+{_FACT_FIELDS}
+
+### Existing observations
+
+A JSON array pooled from recalls across the new facts. Each entry has:
+{_OBSERVATION_FIELDS}"""
 
 # Per-batch data section for the cached split path — the stable format
 # explanation above is omitted here (it lives in the cached prefix); only the
@@ -64,24 +81,6 @@ _SPLIT_INPUT_SECTION = """## INPUT
 
 {observations_text}"""
 
-# Data section — format placeholders {facts_text} and {observations_text} are substituted at call time
-_INPUT_SECTION = """## INPUT
-
-### New facts
-
-{facts_text}
-
-### Existing observations
-
-JSON array, pooled from recalls across all new facts above. Each entry has:
-- `id`: unique identifier — copy this exactly when issuing an UPDATE or DELETE
-- `text`: the observation content
-- `proof_count`: number of supporting memories
-- `occurred_start` / `occurred_end`: temporal range of source facts
-- `source_memories`: array of supporting facts with their text and dates
-
-{observations_text}"""
-
 _DECISION_GUIDE = """## DECISION GUIDE
 
 - **Same canonical event, decision, claim, or facet as an existing observation → UPDATE** (use `observation_id` + new `source_fact_ids`).
@@ -142,39 +141,6 @@ Expected output (UPDATE for the state change; CREATE for the unrelated work-hour
 - Return `{{"creates": [], "updates": [], "deletes": []}}` if nothing durable is found."""
 
 
-def build_batch_consolidation_prompt(
-    observations_mission: str | None = None,
-    observation_capacity_note: str | None = None,
-    llm_output_language: str | None = None,
-) -> str:
-    """
-    Build the consolidation prompt for batch mode (multiple facts per LLM call).
-
-    The mission defines *what* to track (customisable per bank) and takes
-    priority over the built-in processing rules when the two conflict.
-    Processing rules, decision guide, and output format are always present.
-    When ``llm_output_language`` is set, observations are emitted in that
-    language.
-    """
-    mission = escape_for_prompt(observations_mission or _DEFAULT_MISSION)
-
-    capacity_section = ""
-    if observation_capacity_note:
-        capacity_section = f"\n\n## CAPACITY CONSTRAINT\n\n{escape_for_prompt(observation_capacity_note)}"
-
-    return (
-        "You are a memory consolidation system. Synthesize new facts into "
-        "observations, merging with existing observations when appropriate.\n\n"
-        f"## MISSION\n\n{mission}\n\n"
-        f"{_MISSION_PRIORITY_NOTE}"
-        f"{capacity_section}\n\n"
-        f"{_PROCESSING_RULES}\n\n"
-        f"{_INPUT_SECTION}\n\n"
-        f"{_DECISION_GUIDE}\n\n"
-        f"{_OUTPUT_SECTION}" + output_language_directive(llm_output_language)
-    )
-
-
 def build_consolidation_system_prompt(
     llm_output_language: str | None = None,
 ) -> str:

hindsight-api-slim/hindsight_api/engine/cross_encoder.py

@@ -7,7 +7,6 @@ Configuration via environment variables - see hindsight_api.config for all env v
 """
 
 import asyncio
-import gc
 import logging
 import os
 import warnings
@@ -48,53 +47,16 @@ from ..config import (
     ENV_RERANKER_ZEROENTROPY_API_KEY,
 )
 from .bank_attribution import reranker_bank_attribution_headers
+from .local_device import (
+    release_local_inference_memory,
+    resolve_model_device_type,
+    select_local_device,
+)
+from .tei_retry import tei_retry_delay
 
 logger = logging.getLogger(__name__)
 
 
-def _resolve_malloc_trim():
-    """Return a callable that asks glibc to release freed heap pages to the OS.
-
-    Local CPU rerankers (FlashRank/ONNX, SentenceTransformers/torch) allocate
-    large transient numpy/tensor buffers per call. On Linux glibc, those pages
-    are freed at the Python level but kept by the allocator as a high-water
-    mark — RSS grows monotonically across many recalls (see issue #1717).
-    Calling `malloc_trim(0)` after each batch returns those pages to the OS.
-
-    Resolved once at import; returns a no-op on non-glibc platforms (macOS,
-    musl, Windows) where the call is unavailable or unnecessary.
-    """
-    import sys
-
-    if sys.platform != "linux":
-        return lambda: None
-
-    import ctypes
-    import ctypes.util
-
-    libc_path = ctypes.util.find_library("c")
-    if libc_path is None:
-        return lambda: None
-    try:
-        libc = ctypes.CDLL(libc_path)
-        trim = libc.malloc_trim
-    except (OSError, AttributeError):
-        # Not glibc (musl has no malloc_trim) or libc lookup failed.
-        return lambda: None
-    trim.argtypes = [ctypes.c_size_t]
-    trim.restype = ctypes.c_int
-    return lambda: trim(0)
-
-
-_malloc_trim = _resolve_malloc_trim()
-
-
-def _release_rerank_heap() -> None:
-    """Release transient Python and native heap memory after local reranking."""
-    gc.collect()
-    _malloc_trim()
-
-
 class CrossEncoderModel(ABC):
     """
     Abstract base class for cross-encoder reranking.
@@ -159,6 +121,7 @@ class LocalSTCrossEncoder(CrossEncoderModel):
         fp16: bool = False,
         bucket_batching: bool = False,
         batch_size: int = DEFAULT_RERANKER_LOCAL_BATCH_SIZE,
+        allow_mps: bool = False,
     ):
         """
         Initialize local SentenceTransformers cross-encoder.
@@ -180,6 +143,9 @@ class LocalSTCrossEncoder(CrossEncoderModel):
                             Default: False (opt-in via env var).
             batch_size: Batch size for predict() calls. Optimal values vary by
                        hardware and model (MPS: 32, CUDA: 128+). Default: 32.
+            allow_mps: Opt in to the Apple Silicon MPS GPU. Disabled by default
+                      because MPS leaks memory under variable-length workloads
+                      (see engine/local_device.py). Default: False
         """
         self.model_name = model_name or DEFAULT_RERANKER_LOCAL_MODEL
         self.force_cpu = force_cpu
@@ -187,7 +153,9 @@ class LocalSTCrossEncoder(CrossEncoderModel):
         self.fp16 = fp16
         self.bucket_batching = bucket_batching
         self.batch_size = batch_size
+        self.allow_mps = allow_mps
         self._model = None
+        self._device_type: str = "cpu"
         LocalSTCrossEncoder._max_concurrent = max_concurrent
 
     @property
@@ -209,33 +177,13 @@ class LocalSTCrossEncoder(CrossEncoderModel):
 
         logger.info(f"Reranker: initializing local provider with model {self.model_name}")
 
-        # Determine device based on hardware availability.
-        # We always set low_cpu_mem_usage=False to prevent lazy loading (meta tensors)
-        # which can cause issues when accelerate is installed but no GPU is available.
+        # Determine device based on hardware availability. We always set
+        # low_cpu_mem_usage=False to prevent lazy loading (meta tensors) which can
+        # cause issues when accelerate is installed but no GPU is available.
         # Note: We do NOT use device_map because CrossEncoder internally calls .to(device)
         # after loading, which conflicts with accelerate's device_map handling.
-        import torch
-
-        # Force CPU mode if configured (used in daemon mode to avoid MPS/XPC issues on macOS)
-        if self.force_cpu:
-            device = "cpu"
-            logger.info("Reranker: forcing CPU mode (HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1)")
-        else:
-            # Check for GPU (CUDA), Apple Silicon (MPS), or Intel XPU
-            # Wrap in try-except to gracefully handle any device detection issues
-            # (e.g., in CI environments or when PyTorch is built without GPU support)
-            device = "cpu"  # Default to CPU
-            try:
-                has_gpu = torch.cuda.is_available() or (
-                    hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
-                )
-                # Intel Arc XPU support — torch.xpu is available when the XPU build is loaded
-                if not has_gpu and hasattr(torch, "xpu"):
-                    has_gpu = torch.xpu.is_available()
-                if has_gpu:
-                    device = None  # Let sentence-transformers auto-detect GPU/MPS/XPU
-            except Exception as e:
-                logger.warning(f"Failed to detect GPU/MPS/XPU, falling back to CPU: {e}")
+        # MPS is opt-in (allow_mps) — see engine/local_device.py for why.
+        device = select_local_device(self.force_cpu, self.allow_mps)
 
         # Patch transformers 5.x compatibility for models using XLM-RoBERTa
         # (e.g., jina-reranker-v2-base-multilingual). transformers 5.x removed
@@ -279,9 +227,11 @@ class LocalSTCrossEncoder(CrossEncoderModel):
                 # Restore original logging level
                 transformers_logger.setLevel(original_level)
 
+        self._device_type = resolve_model_device_type(self._model)
+
         # FP16 inference: convert model weights to half precision.
         # Empirically validated: 27-36% faster on MPS, quality-identical (20/20 overlap).
-        if self.fp16 and device != "cpu":
+        if self.fp16 and self._device_type != "cpu":
             self._model.model.half()
             logger.info("Reranker: FP16 inference enabled")
 
@@ -324,7 +274,7 @@ class LocalSTCrossEncoder(CrossEncoderModel):
             scores = self._model.predict(pairs, batch_size=self.batch_size, show_progress_bar=False)
             return scores.tolist() if hasattr(scores, "tolist") else list(scores)
         finally:
-            _release_rerank_heap()
+            release_local_inference_memory(self._device_type)
 
     async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
         """
@@ -440,14 +390,20 @@ class RemoteTEICrossEncoder(CrossEncoderModel):
                         await asyncio.sleep(delay)
                         delay *= 2  # Exponential backoff
                 except httpx.HTTPStatusError as e:
-                    # Retry on 5xx server errors
-                    if e.response.status_code >= 500 and attempt < self.max_retries:
+                    # TEI uses 429 as normal overload backpressure. Retry it with
+                    # the same bounded budget as transient server errors.
+                    if (e.response.status_code == 429 or e.response.status_code >= 500) and attempt < self.max_retries:
                         last_error = e
-                        logger.warning(
-                            f"TEI server error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. "
-                            f"Retrying in {delay}s..."
+                        sleep_delay = tei_retry_delay(
+                            e.response,
+                            delay,
+                            request_timeout=self.timeout,
                         )
-                        await asyncio.sleep(delay)
+                        logger.warning(
+                            f"TEI transient error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. "
+                            f"Retrying in {sleep_delay:.2f}s..."
+                        )
+                        await asyncio.sleep(sleep_delay)
                         delay *= 2
                     else:
                         raise
@@ -933,6 +889,7 @@ class FlashRankCrossEncoder(CrossEncoderModel):
         self.max_length = max_length
         self.cpu_mem_arena = cpu_mem_arena
         self._ranker = None
+        self._device_type: str = "cpu"  # FlashRank runs on CPU via ONNX Runtime
         FlashRankCrossEncoder._max_concurrent = max_concurrent
 
     @property
@@ -1037,7 +994,7 @@ class FlashRankCrossEncoder(CrossEncoderModel):
 
             return all_scores
         finally:
-            _release_rerank_heap()
+            release_local_inference_memory(self._device_type)
 
     async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
         """
@@ -1651,6 +1608,7 @@ def create_cross_encoder_from_env() -> CrossEncoderModel:
             fp16=config.reranker_local_fp16,
             bucket_batching=config.reranker_local_bucket_batching,
             batch_size=config.reranker_local_batch_size,
+            allow_mps=config.reranker_local_allow_mps,
         )
     elif provider == "cohere":
         api_key = config.reranker_cohere_api_key

hindsight-api-slim/hindsight_api/engine/db/ops_oracle.py

@@ -282,22 +282,29 @@ class OracleOps(DataAccessOps):
     ) -> None:
         if not unit_ids:
             return
-        # Oracle doesn't support ON CONFLICT; rely on the PK and the
-        # IGNORE_ROW_ON_DUPKEY_INDEX hint to skip duplicates server-side.
-        # The hint name must match the PK constraint exactly.
+        # Locking upsert (#3034), the Oracle analogue of the PG
+        # ``ON CONFLICT DO UPDATE``. The old IGNORE_ROW_ON_DUPKEY_INDEX insert
+        # skipped duplicates WITHOUT locking the existing row, so a mutation
+        # re-enqueueing an already-queued unit could not block a worker from
+        # concurrently claiming (deleting) that row and processing the unit's
+        # pre-mutation state — the re-enqueue signal was silently lost. MERGE
+        # WHEN MATCHED takes an exclusive row lock on the existing queue row
+        # (the SET is a deliberate no-op that preserves enqueued_at); WHEN NOT
+        # MATCHED inserts a fresh row. That serialises the mutation against the
+        # worker's claim for the same (bank_id, unit_id).
         #
-        # Sort to enforce a global lock-acquisition order on the
-        # (bank_id, unit_id) PK. Without this, two concurrent
-        # transactions inserting overlapping unit_id sets in different
-        # orders can deadlock on the unique-check row locks. Sorting
-        # gives every concurrent caller the same lock order, so
-        # conflicting inserts queue cleanly instead of cycling.
+        # Sort to enforce a global (bank_id, unit_id) lock-acquisition order,
+        # matching claim_graph_maintenance_batch's delete order, so overlapping
+        # mutation/worker sets acquire the shared row locks ascending and cannot
+        # cycle.
         sorted_unit_ids = sorted(unit_ids)
         await conn.executemany(
             f"""
-            INSERT /*+ IGNORE_ROW_ON_DUPKEY_INDEX({table}, pk_graph_maintenance_queue) */
-            INTO {table} (bank_id, unit_id)
-            VALUES ($1, $2)
+            MERGE INTO {table} q
+            USING (SELECT $1 AS bank_id, $2 AS unit_id FROM dual) s
+            ON (q.bank_id = s.bank_id AND q.unit_id = s.unit_id)
+            WHEN MATCHED THEN UPDATE SET q.enqueued_at = q.enqueued_at
+            WHEN NOT MATCHED THEN INSERT (bank_id, unit_id) VALUES (s.bank_id, s.unit_id)
             """,
             [(bank_id, uid) for uid in sorted_unit_ids],
         )
@@ -322,7 +329,15 @@ class OracleOps(DataAccessOps):
             bank_id,
             limit,
         )
-        claimed = [str(row["unit_id"]) for row in rows]
+        # Ordered locking (#3034): the per-row DELETE takes the queue rows'
+        # exclusive locks in executemany array order. Sort the claimed keys by
+        # unit_id so those locks are acquired in the same (bank_id, unit_id)
+        # order the enqueue MERGE uses — overlapping mutation/worker sets then
+        # lock the shared rows ascending and cannot cycle. (The batch is still
+        # *chosen* oldest-first by enqueued_at above; only the lock/delete order
+        # is normalised.) The Pass 1 retry wrap in run_graph_maintenance_job is
+        # the ORA-00060 backstop for any residual interleaving.
+        claimed = sorted(str(row["unit_id"]) for row in rows)
         if claimed:
             await conn.executemany(
                 f"DELETE FROM {table} WHERE bank_id = $1 AND unit_id = $2",

hindsight-api-slim/hindsight_api/engine/db/ops_postgresql.py

@@ -253,11 +253,20 @@ class PostgreSQLOps(DataAccessOps):
         entity_names: list[str],
         entity_dates: list,
     ) -> dict[str, str]:
+        # ORDER BY LOWER(name) so every concurrent batch inserts in the same order
+        # as the conflict target (bank_id, LOWER(canonical_name)). ON CONFLICT DO
+        # NOTHING takes a ShareLock on the inserting transaction of any speculative
+        # row it collides with, so two batches with overlapping names inserting in
+        # different orders deadlock. The caller already sorts by Python's
+        # ``str.lower()``, which agrees with the index for ASCII but not for every
+        # locale (see the Turkish-İ note in entity_resolver) — ordering in SQL makes
+        # the database's own collation the single arbiter for all writers.
         inserted_rows = await conn.fetch(
             f"""
             INSERT INTO {table} (bank_id, canonical_name, first_seen, last_seen, mention_count)
             SELECT $1, name, COALESCE(event_date, now()), COALESCE(event_date, now()), 0
             FROM unnest($2::text[], $3::timestamptz[]) AS t(name, event_date)
+            ORDER BY LOWER(name)
             ON CONFLICT (bank_id, LOWER(canonical_name))
             DO NOTHING
             RETURNING id, LOWER(canonical_name) AS name_lower
@@ -360,11 +369,24 @@ class PostgreSQLOps(DataAccessOps):
         # concurrent caller the same lock order, so conflicting inserts
         # queue cleanly instead of cycling.
         sorted_unit_ids = sorted(unit_ids)
+        # DO UPDATE (not DO NOTHING) on a duplicate enqueue — #3034. The SET is a
+        # deliberate no-op that preserves enqueued_at; its only purpose is to take
+        # the existing row's lock. DO NOTHING does NOT lock the conflicting row, so
+        # a mutation that re-enqueues an already-queued unit could not block a
+        # worker from concurrently claiming (deleting) that row and processing the
+        # unit's pre-mutation state; the re-enqueue signal was then silently lost
+        # and the unit's derived links stayed stale with an empty queue. Locking
+        # the row serialises the mutation against the worker's claim for that
+        # (bank_id, unit_id): the worker either waits for the committed post-mutation
+        # state, or (if it claimed first) this INSERT lands a fresh row after the
+        # worker's delete commits. Row locks are acquired in sorted unit_id order,
+        # matching claim_graph_maintenance_batch, so the two never cycle.
         await conn.execute(
             f"""
             INSERT INTO {table} (bank_id, unit_id)
             SELECT $1, v FROM unnest($2::uuid[]) AS t(v)
-            ON CONFLICT (bank_id, unit_id) DO NOTHING
+            ON CONFLICT (bank_id, unit_id)
+                DO UPDATE SET enqueued_at = {table}.enqueued_at
             """,
             bank_id,
             sorted_unit_ids,
@@ -377,16 +399,35 @@ class PostgreSQLOps(DataAccessOps):
         bank_id: str,
         limit: int,
     ) -> list[str]:
+        # Ordered locking (#3034). Choose the oldest batch by enqueued_at, but
+        # acquire the row locks in (bank_id, unit_id) order — the same order the
+        # enqueue upsert takes them — so a foreground mutation re-enqueueing an
+        # overlapping unit set can never cycle against a worker draining it. The
+        # `chosen` CTE is MATERIALIZED so the enqueued_at pick is fenced from the
+        # locking clause; `FOR UPDATE OF q ... ORDER BY q.unit_id` then puts
+        # LockRows above the Sort, so locks are taken ascending by unit_id (same
+        # idiom as prune_stale_cooccurrences' #2529 ordered lock). A concurrent
+        # enqueue holding one of these rows blocks this claim until it commits, at
+        # which point the worker deletes and processes the committed state.
         rows = await conn.fetch(
             f"""
-            DELETE FROM {table}
-            WHERE (bank_id, unit_id) IN (
+            WITH chosen AS MATERIALIZED (
                 SELECT bank_id, unit_id FROM {table}
                 WHERE bank_id = $1
                 ORDER BY enqueued_at
                 LIMIT $2
+            ),
+            locked AS (
+                SELECT q.bank_id, q.unit_id
+                FROM {table} q
+                JOIN chosen c ON c.bank_id = q.bank_id AND c.unit_id = q.unit_id
+                ORDER BY q.unit_id
+                FOR UPDATE OF q
             )
-            RETURNING unit_id
+            DELETE FROM {table} q
+            USING locked l
+            WHERE q.bank_id = l.bank_id AND q.unit_id = l.unit_id
+            RETURNING q.unit_id
             """,
             bank_id,
             limit,
@@ -481,11 +522,21 @@ class PostgreSQLOps(DataAccessOps):
         mu_table: str,
         unit_ids: list[str],
     ) -> list[ResultRow]:
+        # Cast only canonical UUID text inputs, never the indexed column. The old
+        # ``id::text`` predicate silently ignored malformed, uppercase, braced,
+        # and unhyphenated inputs; filtering before the cast preserves that
+        # behavior while allowing the primary-key index to serve the lookup.
         return await conn.fetch(
             f"""
             SELECT id, event_date, fact_type
             FROM {mu_table}
-            WHERE id::text = ANY($1)
+            WHERE id = ANY(
+                ARRAY(
+                    SELECT input.unit_id::uuid
+                    FROM unnest($1::text[]) AS input(unit_id)
+                    WHERE input.unit_id ~ '^[0-9a-f]{{8}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{12}}$'
+                )
+            )
             """,
             unit_ids,
         )
@@ -800,10 +851,16 @@ class PostgreSQLOps(DataAccessOps):
         internal_id: str,
         fact_types: dict[str, str],
     ) -> None:
+        # CONCURRENTLY so the drop takes ShareUpdateExclusive, not ACCESS
+        # EXCLUSIVE, on the shared memory_units table. A plain DROP INDEX blocks
+        # (and deadlocks with) every other bank's concurrent reads/writes on the
+        # table; CONCURRENTLY does not conflict with DML. The caller
+        # (delete_bank) runs this on an autocommit connection after its delete
+        # transaction has committed — CONCURRENTLY cannot run inside a tx.
         for ft, suffix in fact_types.items():
             uid = str(internal_id).replace("-", "")[:16]
             idx = f"idx_mu_emb_{suffix}_{uid}"
-            await conn.execute(f"DROP INDEX IF EXISTS {schema}.{idx}")
+            await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}.{idx}")
 
     def get_entity_resolution_strategy(self) -> str:
         return "trigram"

hindsight-api-slim/hindsight_api/engine/db/oracle.py

@@ -23,6 +23,8 @@ from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 from typing import Any, NamedTuple
 
+from .pool_instrumentation import PoolStats, acquire_conn
+
 
 class _OracleJSONEncoder(json.JSONEncoder):
     """JSON encoder that handles datetime and UUID objects."""
@@ -146,6 +148,7 @@ _JSON_COL_NAMES = {
     "config",
     "observation_scopes",
     "source_memory_ids",
+    "causal_links",
     "trigger",
     "http_config",
     "event_types",
@@ -444,9 +447,6 @@ def _rewrite_pg_to_oracle(query: str) -> RewriteResult:
     if has_for_update:
         # FOR UPDATE path: use ROWNUM instead of FETCH FIRST.
         # Extract and remove LIMIT clause, inject ROWNUM into WHERE.
-        def _limit_to_rownum(m):
-            return ""  # Remove the LIMIT clause; we'll add ROWNUM below
-
         limit_val = None
         limit_match = re.search(r"\bLIMIT\s+(\d+|:\w+)\b", query, re.IGNORECASE)
         if limit_match:
@@ -1246,6 +1246,7 @@ class OracleBackend(DatabaseBackend):
         # SESSION_USER so default-schema acquisitions can explicitly reset a
         # connection that was previously used for a tenant schema.
         self._default_schema: str | None = None
+        self._acquire_warn_threshold_s: float = 1.0
 
     async def initialize(
         self,
@@ -1261,6 +1262,10 @@ class OracleBackend(DatabaseBackend):
         oracledb = _import_oracledb()
         self._oracledb = oracledb
 
+        from ...config import get_config
+
+        self._acquire_warn_threshold_s = get_config().db_acquire_warn_threshold_ms / 1000.0
+
         # Parse URL-format DSN (oracle://user:pass@host:port/service)
         from urllib.parse import urlparse
 
@@ -1322,10 +1327,23 @@ class OracleBackend(DatabaseBackend):
             # expression" and aborts every acquire().
             cursor.close()
 
+    def _pool_stats(self) -> PoolStats | None:
+        """Snapshot for slow-acquire logs, from oracledb pool attributes."""
+        pool = self._pool
+        if pool is None:
+            return None
+        try:
+            busy = pool.busy
+            return PoolStats(in_use=busy, max=pool.max, idle=pool.opened - busy)
+        except Exception:
+            return None
+
     @asynccontextmanager
     async def acquire(self) -> AsyncIterator[OracleConnection]:
         pool = self._ensure_pool()
-        conn = await pool.acquire()
+        conn = await acquire_conn(
+            pool.acquire(), pool_stats=self._pool_stats, warn_threshold_s=self._acquire_warn_threshold_s
+        )
         try:
             await self._set_session_schema(conn)
             yield OracleConnection(conn)
@@ -1341,7 +1359,9 @@ class OracleBackend(DatabaseBackend):
     @asynccontextmanager
     async def transaction(self) -> AsyncIterator[OracleConnection]:
         pool = self._ensure_pool()
-        conn = await pool.acquire()
+        conn = await acquire_conn(
+            pool.acquire(), pool_stats=self._pool_stats, warn_threshold_s=self._acquire_warn_threshold_s
+        )
         try:
             await self._set_session_schema(conn)
             yield OracleConnection(conn)

hindsight-api-slim/hindsight_api/engine/db/pool_instrumentation.py

@@ -0,0 +1,137 @@
+"""Instrumentation for database connection-pool acquisition.
+
+asyncpg exposes pool *size* and *idle* counts, but not how many callers are
+currently **queued waiting** for a connection — and that queue depth is the
+signal that actually distinguishes a saturated pool from a healthy one. When the
+pool is exhausted, ``/health`` (which itself acquires a connection to run
+``SELECT 1``) blocks in ``pool.acquire()`` until a connection frees or the acquire
+times out, so a liveness probe can fail **with the event loop completely idle**.
+
+This module tracks the process-wide count of in-flight acquisitions that have not
+yet obtained a connection, and times each acquire so a slow one logs with full
+pool stats. It is the DB-side counterpart to ``loop_watchdog`` (which covers loop
+stalls); together, a stuck ``/health`` can be attributed to either a blocked loop
+or pool exhaustion from the logs alone.
+
+The counter is a plain int mutated only from the event-loop thread (asyncpg
+acquisitions are awaited on the loop), so no lock is needed.
+"""
+
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import AsyncIterator, Callable
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
+from typing import Any
+
+logger = logging.getLogger("hindsight.db.pool")
+
+_waiting = 0  # callers currently blocked in pool.acquire(), process-wide
+
+
+@dataclass(frozen=True, slots=True)
+class PoolStats:
+    """Point-in-time connection-pool utilization snapshot."""
+
+    in_use: int
+    max: int
+    idle: int
+
+
+def waiting_count() -> int:
+    """Number of callers currently blocked waiting to acquire a pooled connection."""
+    return _waiting
+
+
+@asynccontextmanager
+async def instrument_acquire(
+    acquire_cm: Any,
+    *,
+    pool_stats: Callable[[], PoolStats | None] | None = None,
+    warn_threshold_s: float,
+) -> AsyncIterator[Any]:
+    """Wrap a pool's ``acquire()`` context manager with wait tracking + slow-acquire logging.
+
+    Args:
+        acquire_cm: an async context manager yielding a connection (e.g. the object
+            returned by ``asyncpg.Pool.acquire()``).
+        pool_stats: optional zero-arg callable returning a ``PoolStats`` snapshot for
+            the slow-acquire log line.
+        warn_threshold_s: log a warning when the acquire itself takes at least this long.
+
+    Yields:
+        The acquired connection.
+    """
+    global _waiting
+    _waiting += 1
+    start = time.monotonic()
+    acquired = False
+    try:
+        async with acquire_cm as conn:
+            acquired = True
+            _waiting -= 1
+            _record_acquire_wait(time.monotonic() - start, pool_stats, warn_threshold_s)
+            yield conn
+    finally:
+        # If __aenter__ raised (acquire timeout / cancellation), we never
+        # decremented above — do it here so the waiter count can't leak.
+        if not acquired:
+            _waiting -= 1
+
+
+async def acquire_conn(
+    acquire_awaitable: Any,
+    *,
+    pool_stats: Callable[[], PoolStats | None] | None = None,
+    warn_threshold_s: float,
+) -> Any:
+    """Await a pool acquire that returns a connection, with wait tracking + slow log.
+
+    For pools whose acquire is ``conn = await pool.acquire()`` (oracledb) rather than
+    an async context manager (asyncpg — use ``instrument_acquire`` for those). The
+    caller is responsible for releasing the returned connection.
+    """
+    global _waiting
+    _waiting += 1
+    start = time.monotonic()
+    try:
+        conn = await acquire_awaitable
+    finally:
+        _waiting -= 1
+    _record_acquire_wait(time.monotonic() - start, pool_stats, warn_threshold_s)
+    return conn
+
+
+def _record_acquire_wait(
+    wait_s: float,
+    pool_stats: Callable[[], PoolStats | None] | None,
+    warn_threshold_s: float,
+) -> None:
+    try:
+        from ...metrics import get_metrics_collector
+
+        get_metrics_collector().record_db_acquire_wait(wait_s)
+    except Exception:
+        pass
+
+    if wait_s < warn_threshold_s:
+        return
+
+    stats: PoolStats | None = None
+    if pool_stats is not None:
+        try:
+            stats = pool_stats()
+        except Exception:
+            stats = None
+    logger.warning(
+        "slow DB pool acquire: waited %.3fs for a connection "
+        "(in_use=%s max=%s idle=%s waiting=%s). The pool is likely saturated; "
+        "/health can stall on connection acquisition while the event loop is free.",
+        wait_s,
+        stats.in_use if stats else None,
+        stats.max if stats else None,
+        stats.idle if stats else None,
+        _waiting,
+    )

hindsight-api-slim/hindsight_api/engine/db/postgresql.py

@@ -15,6 +15,7 @@ from typing import Any
 import asyncpg  # noqa: F401
 
 from .base import DatabaseBackend, DatabaseConnection
+from .pool_instrumentation import PoolStats, instrument_acquire
 
 logger = logging.getLogger(__name__)
 
@@ -76,6 +77,8 @@ class PostgreSQLBackend(DatabaseBackend):
 
     def __init__(self) -> None:
         self._pool: asyncpg.Pool | None = None
+        self._acquire_warn_threshold_s: float = 1.0
+        self._acquire_timeout_s: float | None = None
 
     async def initialize(
         self,
@@ -88,6 +91,16 @@ class PostgreSQLBackend(DatabaseBackend):
         statement_cache_size: int = 0,
         init_callback: Any | None = None,
     ) -> None:
+        from ...config import get_config
+
+        self._acquire_warn_threshold_s = get_config().db_acquire_warn_threshold_ms / 1000.0
+        # Kept for acquire() below: asyncpg's ``timeout`` create_pool kwarg is a
+        # *connect* kwarg (how long establishing a new connection may take), and
+        # ``Pool.acquire()`` defaults to waiting for a free connection forever.
+        # Passing it here alone made HINDSIGHT_API_DB_ACQUIRE_TIMEOUT a no-op for
+        # the wait it names: a pool-exhaustion stall never surfaced as an error,
+        # it just hung (#3002). 0 restores the unbounded behaviour.
+        self._acquire_timeout_s = acquire_timeout if acquire_timeout > 0 else None
         self._pool = await asyncpg.create_pool(
             dsn,
             min_size=min_size,
@@ -121,16 +134,32 @@ class PostgreSQLBackend(DatabaseBackend):
     def is_ready(self) -> bool:
         return self._pool is not None
 
+    def _pool_stats(self) -> PoolStats | None:
+        """Snapshot for slow-acquire logs. in_use = live connections minus idle ones."""
+        pool = self._pool
+        if pool is None:
+            return None
+        idle = pool.get_idle_size()
+        return PoolStats(in_use=pool.get_size() - idle, max=pool.get_max_size(), idle=idle)
+
     @asynccontextmanager
     async def acquire(self) -> AsyncIterator[PostgresConnection]:
         pool = self._ensure_pool()
-        async with pool.acquire() as conn:
+        async with instrument_acquire(
+            pool.acquire(timeout=self._acquire_timeout_s),
+            pool_stats=self._pool_stats,
+            warn_threshold_s=self._acquire_warn_threshold_s,
+        ) as conn:
             yield PostgresConnection(conn)
 
     @asynccontextmanager
     async def transaction(self) -> AsyncIterator[PostgresConnection]:
         pool = self._ensure_pool()
-        async with pool.acquire() as conn:
+        async with instrument_acquire(
+            pool.acquire(timeout=self._acquire_timeout_s),
+            pool_stats=self._pool_stats,
+            warn_threshold_s=self._acquire_warn_threshold_s,
+        ) as conn:
             async with conn.transaction():
                 yield PostgresConnection(conn)
 

hindsight-api-slim/hindsight_api/engine/db_budget.py

@@ -164,35 +164,6 @@ class BudgetedOperation:
         """
         return BudgetedPool(pool, self)
 
-    async def acquire_many(
-        self,
-        pool: Any,
-        count: int,
-    ) -> AsyncIterator[list[Any]]:
-        """
-        Acquire multiple connections within the budget.
-
-        Note: This acquires connections sequentially to respect the budget.
-        For parallel acquisition, use multiple acquire() calls with asyncio.gather().
-        This method is intended for use with raw asyncpg pools only, not DatabaseBackend.
-
-        Args:
-            pool: asyncpg connection pool (raw pool only)
-            count: Number of connections to acquire
-
-        Yields:
-            List of database connections
-        """
-        connections = []
-        try:
-            for _ in range(count):
-                conn = await pool.acquire()
-                connections.append(conn)
-            yield connections
-        finally:
-            for conn in connections:
-                await pool.release(conn)
-
 
 # Global default manager instance
 _default_manager: ConnectionBudgetManager | None = None

hindsight-api-slim/hindsight_api/engine/embeddings.py

@@ -48,6 +48,12 @@ from ..config import (
     ENV_LLM_API_KEY,
 )
 from .bank_attribution import apply_bank_attribution
+from .local_device import (
+    release_local_inference_memory,
+    resolve_model_device_type,
+    select_local_device,
+)
+from .tei_retry import tei_retry_delay
 
 logger = logging.getLogger(__name__)
 
@@ -155,7 +161,13 @@ class LocalSTEmbeddings(Embeddings):
     The embedding dimension is auto-detected from the model.
     """
 
-    def __init__(self, model_name: str | None = None, force_cpu: bool = False, trust_remote_code: bool = False):
+    def __init__(
+        self,
+        model_name: str | None = None,
+        force_cpu: bool = False,
+        trust_remote_code: bool = False,
+        allow_mps: bool = False,
+    ):
         """
         Initialize local SentenceTransformers embeddings.
 
@@ -167,12 +179,17 @@ class LocalSTEmbeddings(Embeddings):
             trust_remote_code: Allow loading models with custom code (security risk).
                               Required for some models with custom architectures.
                               Default: False (disabled for security)
+            allow_mps: Opt in to the Apple Silicon MPS GPU. Disabled by default
+                      because MPS leaks memory under variable-length workloads
+                      (see engine/local_device.py). Default: False
         """
         self.model_name = model_name or DEFAULT_EMBEDDINGS_LOCAL_MODEL
         self.force_cpu = force_cpu
         self.trust_remote_code = trust_remote_code
+        self.allow_mps = allow_mps
         self._model = None
         self._dimension: int | None = None
+        self._device_type: str = "cpu"
 
     @property
     def provider_name(self) -> str:
@@ -199,31 +216,11 @@ class LocalSTEmbeddings(Embeddings):
 
         logger.info(f"Embeddings: initializing local provider with model {self.model_name}")
 
-        # Determine device based on hardware availability.
-        # We always set low_cpu_mem_usage=False to prevent lazy loading (meta tensors)
-        # which can cause issues when accelerate is installed but no GPU is available.
-        import torch
-
-        # Force CPU mode if configured (used in daemon mode to avoid MPS/XPC issues on macOS)
-        if self.force_cpu:
-            device = "cpu"
-            logger.info("Embeddings: forcing CPU mode")
-        else:
-            # Check for GPU (CUDA), Apple Silicon (MPS), or Intel XPU
-            # Wrap in try-except to gracefully handle any device detection issues
-            # (e.g., in CI environments or when PyTorch is built without GPU support)
-            device = "cpu"  # Default to CPU
-            try:
-                has_gpu = torch.cuda.is_available() or (
-                    hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
-                )
-                # Intel Arc XPU support — torch.xpu is available when the XPU build is loaded
-                if not has_gpu and hasattr(torch, "xpu"):
-                    has_gpu = torch.xpu.is_available()
-                if has_gpu:
-                    device = None  # Let sentence-transformers auto-detect GPU/MPS/XPU
-            except Exception as e:
-                logger.warning(f"Failed to detect GPU/MPS/XPU, falling back to CPU: {e}")
+        # Determine device based on hardware availability. We always set
+        # low_cpu_mem_usage=False to prevent lazy loading (meta tensors) which can
+        # cause issues when accelerate is installed but no GPU is available.
+        # MPS is opt-in (allow_mps) — see engine/local_device.py for why.
+        device = select_local_device(self.force_cpu, self.allow_mps)
 
         # Suppress verbose transformers warnings during model loading
         # This suppresses the "UNEXPECTED" warnings from BertModel which are harmless
@@ -250,7 +247,8 @@ class LocalSTEmbeddings(Embeddings):
                 transformers_logger.setLevel(original_level)
 
         self._dimension = self._model.get_sentence_embedding_dimension()
-        logger.info(f"Embeddings: local provider initialized (dim: {self._dimension})")
+        self._device_type = resolve_model_device_type(self._model)
+        logger.info(f"Embeddings: local provider initialized (dim: {self._dimension}, device: {self._device_type})")
 
     def encode(self, texts: list[str]) -> list[list[float]]:
         """
@@ -265,8 +263,19 @@ class LocalSTEmbeddings(Embeddings):
         if self._model is None:
             raise RuntimeError("Embeddings not initialized. Call initialize() first.")
 
-        embeddings = self._model.encode(texts, convert_to_numpy=True, show_progress_bar=False)
-        return [emb.tolist() for emb in embeddings]
+        try:
+            embeddings = self._model.encode(texts, convert_to_numpy=True, show_progress_bar=False)
+            return [emb.tolist() for emb in embeddings]
+        finally:
+            # Only reclaim the GPU allocator pool here, and only when actually on a
+            # GPU (opt-in MPS/CUDA/XPU). encode() runs in tight retain loops, so a
+            # gc.collect()/malloc_trim on every call is too costly on the CPU default
+            # — and unnecessary: refcounting frees the small transient buffers
+            # immediately and the allocator reuses them for the next batch. (The
+            # reranker keeps its per-batch heap trim for the #1717 CPU case; it runs
+            # on the lighter recall path.) See engine/local_device.py.
+            if self._device_type != "cpu":
+                release_local_inference_memory(self._device_type)
 
 
 class OnnxEmbeddings(Embeddings):
@@ -506,13 +515,20 @@ class RemoteTEIEmbeddings(Embeddings):
                     time.sleep(delay)
                     delay *= 2  # Exponential backoff
             except httpx.HTTPStatusError as e:
-                # Retry on 5xx server errors
-                if e.response.status_code >= 500 and attempt < self.max_retries:
+                # TEI uses 429 as normal overload backpressure. Retry it with
+                # the same bounded budget as transient server errors.
+                if (e.response.status_code == 429 or e.response.status_code >= 500) and attempt < self.max_retries:
                     last_error = e
-                    logger.warning(
-                        f"TEI server error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. Retrying in {delay}s..."
+                    sleep_delay = tei_retry_delay(
+                        e.response,
+                        delay,
+                        request_timeout=self.timeout,
                     )
-                    time.sleep(delay)
+                    logger.warning(
+                        f"TEI transient error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. "
+                        f"Retrying in {sleep_delay:.2f}s..."
+                    )
+                    time.sleep(sleep_delay)
                     delay *= 2
                 else:
                     raise
@@ -1637,6 +1653,7 @@ def create_embeddings_from_env() -> Embeddings:
             model_name=config.embeddings_local_model,
             force_cpu=config.embeddings_local_force_cpu,
             trust_remote_code=config.embeddings_local_trust_remote_code,
+            allow_mps=config.embeddings_local_allow_mps,
         )
     elif provider == "onnx":
         return OnnxEmbeddings(

hindsight-api-slim/hindsight_api/engine/entity_resolver.py

@@ -948,58 +948,3 @@ class EntityResolver:
                 _CooccurrencePair(entity_id_1=e1, entity_id_2=e2, event_date=ed)
                 for (e1, e2), ed in cooccurrence_pairs.items()
             )
-
-    async def get_units_by_entity(self, entity_id: str, limit: int = 100) -> list[str]:
-        """
-        Get all units that mention an entity.
-
-        Args:
-            entity_id: Entity ID
-            limit: Max results
-
-        Returns:
-            List of unit IDs
-        """
-        async with acquire_with_retry(self.pool) as conn:
-            rows = await conn.fetch(
-                f"""
-                SELECT unit_id
-                FROM {fq_table("unit_entities")}
-                WHERE entity_id = $1
-                ORDER BY unit_id
-                LIMIT $2
-                """,
-                entity_id,
-                limit,
-            )
-            return [row["unit_id"] for row in rows]
-
-    async def get_entity_by_text(
-        self,
-        bank_id: str,
-        entity_text: str,
-    ) -> str | None:
-        """
-        Find an entity by text (for query resolution).
-
-        Args:
-            bank_id: bank ID
-            entity_text: Entity text to search for
-
-        Returns:
-            Entity ID if found, None otherwise
-        """
-        async with acquire_with_retry(self.pool) as conn:
-            row = await conn.fetchrow(
-                f"""
-                SELECT id FROM {fq_table("entities")}
-                WHERE bank_id = $1
-                  AND canonical_name ILIKE $2
-                ORDER BY mention_count DESC
-                LIMIT 1
-                """,
-                bank_id,
-                entity_text,
-            )
-
-            return row["id"] if row else None

hindsight-api-slim/hindsight_api/engine/graph_maintenance.py

@@ -39,6 +39,7 @@ import uuid as uuid_module
 from dataclasses import dataclass
 from typing import TYPE_CHECKING, Any
 
+from ..config import get_config
 from ..models import RequestContext
 from .db.base import DatabaseConnection
 from .retain.link_utils import (
@@ -81,6 +82,20 @@ class _SweepCounts:
     stale_cooccurrences_pruned: int
 
 
+@dataclass
+class _BatchOutcome:
+    """Result of one relink claim+top-up batch (avoids a bare tuple return).
+
+    Returned by value from the retried batch helper so the caller only folds it
+    into ``JobResult`` after the batch's transaction has actually committed — a
+    deadlock/timeout retry rolls the batch back, so accumulating inside the
+    retried body would double-count.
+    """
+
+    units_claimed: int
+    links_added: int
+
+
 @dataclass
 class JobResult:
     """Counters surfaced to the worker dispatcher and operation result."""
@@ -102,35 +117,48 @@ class JobResult:
 async def enqueue_relink_victims(
     conn: DatabaseConnection,
     bank_id: str,
-    deleted_unit_ids: list[str],
+    affected_unit_ids: list[str],
     ops: Any,
+    include_affected_units: bool = False,
 ) -> int:
     """Enqueue surviving units whose outgoing temporal/semantic links pointed at
-    ``deleted_unit_ids`` for later link top-up.
+    ``affected_unit_ids`` for later link top-up.
 
-    Must run inside the same transaction that deletes the units, *before* the
-    cascade fires — once the rows are gone, the join that finds the victims
-    returns nothing.
+    Must run inside the same transaction that drops those links, *before* the
+    delete (or cascade) fires — once the rows are gone, the join that finds the
+    victims returns nothing.
+
+    ``include_affected_units`` covers the case where the affected units are NOT
+    being removed: an edit deletes every link incident to the edited unit but
+    leaves it live, so the unit needs its own outgoing adjacency rebuilt too.
+    Passing it for a unit that will be gone at commit is harmless but pointless
+    — the drain skips queue rows with no live unit — so callers should only set
+    it when the unit survives the transaction.
 
     Args:
-        conn: Database connection inside the active delete transaction.
-        bank_id: Bank owning the deleted units.
-        deleted_unit_ids: Memory_unit IDs about to be (or being) deleted.
+        conn: Database connection inside the active transaction.
+        bank_id: Bank owning the affected units.
+        affected_unit_ids: Memory_unit IDs whose incident temporal/semantic
+            links are about to be (or are being) removed.
         ops: ``DataAccessOps`` instance, supplies the dialect-specific
             bulk-insert path.
+        include_affected_units: Also enqueue ``affected_unit_ids`` themselves,
+            for callers that leave them live. One combined insert (rather than a
+            second call) keeps the queue's sorted lock ordering intact: two
+            transactions editing mutually linked units would otherwise take the
+            ``(bank_id, unit_id)`` keys in opposite orders and deadlock.
 
     Returns:
-        Number of distinct victim units enqueued (after dedup against rows
-        already in the queue).
+        Number of distinct units passed to the queue insert.
     """
-    if not deleted_unit_ids:
+    if not affected_unit_ids:
         return 0
 
-    deleted_uuids = [uuid_module.UUID(uid) if isinstance(uid, str) else uid for uid in deleted_unit_ids]
-    deleted_str_set = {str(uid) for uid in deleted_uuids}
+    affected_uuids = [uuid_module.UUID(uid) if isinstance(uid, str) else uid for uid in affected_unit_ids]
+    affected_str_set = {str(uid) for uid in affected_uuids}
 
-    # Find units (other than the ones being deleted) that have an outgoing
-    # temporal/semantic link pointing at a doomed unit. Entity links are
+    # Find units (other than the affected ones) that have an outgoing
+    # temporal/semantic link pointing at an affected unit. Entity links are
     # intentionally excluded — they're scheduled for removal and would only
     # add noise to the recompute job.
     victim_rows = await conn.fetch(
@@ -141,27 +169,29 @@ async def enqueue_relink_victims(
           AND bank_id = $2
           AND link_type IN ('temporal', 'semantic')
         """,
-        deleted_uuids,
+        affected_uuids,
         bank_id,
     )
 
-    victim_ids = [row["from_unit_id"] for row in victim_rows if str(row["from_unit_id"]) not in deleted_str_set]
+    relink_ids = {row["from_unit_id"] for row in victim_rows if str(row["from_unit_id"]) not in affected_str_set}
+    if include_affected_units:
+        relink_ids.update(affected_uuids)
 
-    if not victim_ids:
+    if not relink_ids:
         return 0
 
     await ops.enqueue_graph_maintenance(
         conn,
         fq_table("graph_maintenance_queue"),
         bank_id,
-        victim_ids,
+        list(relink_ids),
     )
 
     logger.debug(
-        f"[GRAPH_MAINT] Enqueued {len(victim_ids)} relink victims in "
-        f"bank={bank_id} (deleted {len(deleted_unit_ids)} units)"
+        f"[GRAPH_MAINT] Enqueued {len(relink_ids)} units for relinking in "
+        f"bank={bank_id} ({len(affected_unit_ids)} units affected)"
     )
-    return len(victim_ids)
+    return len(relink_ids)
 
 
 async def run_graph_maintenance_job(
@@ -182,15 +212,26 @@ async def run_graph_maintenance_job(
 
     result = JobResult()
     job_start = time.time()
+    semantic_link_min_similarity = get_config().semantic_link_min_similarity
 
     # --- Pass 1: relink ---
     # Per-iteration loop: claim → top up → commit. We rely on submit-time
     # dedup to keep at most one job per bank running, so no need for
     # SKIP LOCKED.
-    iterations = 0
-    while True:
-        from .memory_engine import acquire_with_retry
+    #
+    # The claim now takes the queue rows FOR UPDATE in (bank_id, unit_id) order
+    # (#3034) so it serialises against a concurrent mutation re-enqueueing the
+    # same units instead of racing it. On PG the matching enqueue/claim lock
+    # order prevents a cycle outright; on Oracle the ordered locks are a strong
+    # mitigation but the exact interleaving is harder to guarantee, so each
+    # batch runs inside retry_with_backoff — which already treats ORA-00060 and
+    # Postgres DeadlockDetectedError as retryable. The batch is idempotent: a
+    # rolled-back claim leaves its rows queued, and _relink_batch only tops up
+    # missing links, so re-running re-claims and re-tops-up cleanly.
+    from .db_utils import retry_with_backoff
+    from .memory_engine import acquire_with_retry
 
+    async def _drain_one_batch() -> _BatchOutcome:
         async with acquire_with_retry(backend) as conn:
             async with conn.transaction():
                 unit_ids = await ops.claim_graph_maintenance_batch(
@@ -200,11 +241,28 @@ async def run_graph_maintenance_job(
                     _DRAIN_BATCH_SIZE,
                 )
                 if not unit_ids:
-                    break
+                    return _BatchOutcome(units_claimed=0, links_added=0)
 
-                result.relink_links_added += await _relink_batch(conn, bank_id, unit_ids, ops, backend)
+                links_added = await _relink_batch(
+                    conn,
+                    bank_id,
+                    unit_ids,
+                    ops,
+                    backend,
+                    semantic_link_min_similarity,
+                )
+                return _BatchOutcome(units_claimed=len(unit_ids), links_added=links_added)
 
-        result.relink_units_processed += len(unit_ids)
+    iterations = 0
+    while True:
+        # Fold counters in only after the batch commits — retry_with_backoff may
+        # roll back and re-run the body, and accumulating inside it would double-count.
+        outcome = await retry_with_backoff(_drain_one_batch)
+        if outcome.units_claimed == 0:
+            break
+
+        result.relink_links_added += outcome.links_added
+        result.relink_units_processed += outcome.units_claimed
         iterations += 1
 
         if iterations > 10000:
@@ -228,8 +286,7 @@ async def run_graph_maintenance_job(
     # DeadlockDetectedError. Both prunes are idempotent bank-wide sweeps —
     # rerunning only deletes what's still stale — so retrying the whole
     # transaction on deadlock is safe.
-    from .db_utils import retry_with_backoff
-    from .memory_engine import acquire_with_retry
+    # (retry_with_backoff / acquire_with_retry already imported for Pass 1 above.)
 
     async def _run_sweep() -> _SweepCounts:
         async with acquire_with_retry(backend) as conn:
@@ -276,6 +333,7 @@ async def _relink_batch(
     victim_ids: list[str],
     ops: Any,
     backend: Any,
+    semantic_link_min_similarity: float,
 ) -> int:
     """Top up temporal/semantic links for a batch of victim units. Returns rows inserted."""
     # Load each victim's metadata. Victims whose units were deleted between
@@ -372,6 +430,7 @@ async def _relink_batch(
                     seed_ids,
                     seed_embs,
                     fact_types=seed_ftypes,
+                    threshold=semantic_link_min_similarity,
                 )
                 # Strip self-links (rare but possible because the ANN probe
                 # has no exclude list — see the comment in compute_semantic_links_ann).

hindsight-api-slim/hindsight_api/engine/interface.py

@@ -6,6 +6,7 @@ authentication when a TenantExtension is configured.
 """
 
 from abc import ABC, abstractmethod
+from dataclasses import dataclass
 from datetime import datetime
 from typing import TYPE_CHECKING, Any
 
@@ -13,9 +14,26 @@ if TYPE_CHECKING:
     from hindsight_api.engine.memory_engine import BankLlmHealthInfo, Budget
     from hindsight_api.engine.response_models import RecallResult, ReflectResult
     from hindsight_api.engine.search.tags import TagsMatch
+    from hindsight_api.extensions import BankWriteOperation
     from hindsight_api.models import RequestContext
 
 
+@dataclass(frozen=True)
+class BankConfigState:
+    """Resolved bank configuration and its bank-level overrides."""
+
+    config: dict[str, Any]
+    overrides: dict[str, Any]
+
+
+@dataclass(frozen=True)
+class BankTemplateImportWrite:
+    """One bank-write decision reserved for a specific imported resource."""
+
+    operation: "BankWriteOperation"
+    target: str | None = None
+
+
 class MemoryEngineInterface(ABC):
     """
     Abstract interface for the Memory Engine.
@@ -180,6 +198,37 @@ class MemoryEngineInterface(ABC):
         """
         ...
 
+    @abstractmethod
+    async def get_bank_config(
+        self,
+        bank_id: str,
+        *,
+        request_context: "RequestContext",
+    ) -> BankConfigState:
+        """Return resolved configuration after authenticating and authorizing the read."""
+        ...
+
+    @abstractmethod
+    async def update_bank_config(
+        self,
+        bank_id: str,
+        updates: dict[str, Any],
+        *,
+        request_context: "RequestContext",
+    ) -> BankConfigState:
+        """Create a bank if needed and persist validated configuration overrides."""
+        ...
+
+    @abstractmethod
+    async def reset_bank_config(
+        self,
+        bank_id: str,
+        *,
+        request_context: "RequestContext",
+    ) -> BankConfigState:
+        """Remove all bank configuration overrides after authorization."""
+        ...
+
     @abstractmethod
     async def update_bank_disposition(
         self,
@@ -275,6 +324,8 @@ class MemoryEngineInterface(ABC):
         *,
         fact_type: str | None = None,
         search_query: str | None = None,
+        entity_id: str | None = None,
+        created_before: datetime | None = None,
         limit: int = 100,
         offset: int = 0,
         request_context: "RequestContext",
@@ -286,6 +337,8 @@ class MemoryEngineInterface(ABC):
             bank_id: The memory bank ID.
             fact_type: Filter by fact type.
             search_query: Full-text search query.
+            entity_id: Filter to memory units linked to this entity ID.
+            created_before: Keep units with ``created_at`` before this instant.
             limit: Maximum results.
             offset: Pagination offset.
             request_context: Request context for authentication.
@@ -596,6 +649,8 @@ class MemoryEngineInterface(ABC):
         *,
         name: str | None = None,
         mission: str | None = None,
+        config_updates: dict[str, Any] | None = None,
+        create_if_missing: bool = True,
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
@@ -605,6 +660,9 @@ class MemoryEngineInterface(ABC):
             bank_id: The memory bank ID.
             name: New bank name (optional).
             mission: New mission text (optional, replaces existing).
+            config_updates: Bank configuration overrides to apply with the profile update.
+            create_if_missing: Create a missing bank when True; otherwise raise
+                a 404 operation error.
             request_context: Request context for authentication.
 
         Returns:

hindsight-api-slim/hindsight_api/engine/llm_trace.py

@@ -36,6 +36,21 @@ from .db_utils import acquire_with_retry
 logger = logging.getLogger(__name__)
 
 
+def _llm_requests_persistable() -> bool:
+    """Whether the ``llm_requests`` table exists on the active backend.
+
+    ``llm_requests`` is PostgreSQL-only: its migration is ``run_for_dialect(pg=...)``
+    with the Oracle slot intentionally absent, and MaintenanceLoop skips its
+    retention sweep on Oracle for the same reason. On Oracle the table does not
+    exist, so best-effort trace writes must be skipped rather than attempted —
+    otherwise every LLM call fires an INSERT that fails with ORA-00903 and spams
+    the error log. Mirrors the ``_is_oracle()`` gate in MaintenanceLoop.start.
+    """
+    from .schema import _is_oracle
+
+    return not _is_oracle()
+
+
 # ── bank/operation attribution (carried across the async call chain) ──────────
 
 
@@ -400,6 +415,8 @@ class LLMTraceRecorder:
         """Whether tracing is active for the given call scope."""
         if not self._enabled:
             return False
+        if not _llm_requests_persistable():
+            return False
         if self._allowed_scopes is not None:
             return scope in self._allowed_scopes
         return True
@@ -566,7 +583,7 @@ class LLMTraceRecorder:
         ids are snapshotted synchronously here because the caller may reset the
         context immediately after.
         """
-        if not self._enabled or trace_ctx is None or not trace_ctx.trace_id:
+        if not self._enabled or not _llm_requests_persistable() or trace_ctx is None or not trace_ctx.trace_id:
             return
         created_ids = list(dict.fromkeys([*(created or []), *trace_ctx.created_memory_ids]))
         source_ids = list(dict.fromkeys([*(source or []), *trace_ctx.source_memory_ids]))

hindsight-api-slim/hindsight_api/engine/llm_wrapper.py

@@ -888,7 +888,13 @@ class LLMProvider:
         from ..worker.stage import set_stage
 
         structured = "+structured" if response_format is not None else ""
-        set_stage(f"llm.{self.provider}.{scope}{structured}")
+        # `.queued` until the concurrency permits are in hand — see the acquire
+        # below. Without it, a call waiting on a saturated semaphore is
+        # indistinguishable from one the provider is actively running, and the
+        # label points at the provider (#3002: an operator lost an hour to
+        # "llm.bedrock.*" for tasks that had never reached Bedrock).
+        base_stage = f"llm.{self.provider}.{scope}{structured}"
+        set_stage(f"{base_stage}.queued")
 
         # Resolve the retry policy: explicit per-call arg wins, else the provider's
         # configured per-operation/global default, else this method's own fallback.
@@ -948,6 +954,7 @@ class LLMProvider:
             async with AsyncExitStack() as stack:
                 for sem in _semaphores_for_scope(scope):
                     await stack.enter_async_context(sem)
+                set_stage(base_stage)
 
                 # cached_prefix is only set for providers that returned a handle
                 # from get_or_create_cached_prefix() (e.g. Gemini); it's None for
@@ -1039,7 +1046,9 @@ class LLMProvider:
         """
         from ..worker.stage import set_stage
 
-        set_stage(f"llm.{self.provider}.{scope}+tools")
+        # `.queued` until the permits are held — see the structured path above.
+        base_stage = f"llm.{self.provider}.{scope}+tools"
+        set_stage(f"{base_stage}.queued")
 
         # Resolve the retry policy: explicit per-call arg wins, else the provider's
         # configured per-operation/global default, else this method's own fallback.
@@ -1081,6 +1090,7 @@ class LLMProvider:
             async with AsyncExitStack() as stack:
                 for sem in _semaphores_for_scope(scope):
                     await stack.enter_async_context(sem)
+                set_stage(base_stage)
 
                 # cached_prefix is only set for providers that returned a handle
                 # from get_or_create_cached_prefix() / create_incremental_cache();

hindsight-api-slim/hindsight_api/engine/local_device.py

@@ -0,0 +1,172 @@
+"""Device selection and post-inference memory release for local (in-process)
+SentenceTransformer / CrossEncoder models.
+
+Two concerns live here, both about keeping a local API instance's memory flat:
+
+**1. Device selection — MPS is opt-in.**
+On Apple Silicon the PyTorch **MPS** (Metal) backend caches a distinct compiled
+kernel graph *and* allocator pool per unique input tensor shape, and never
+releases them. Under the variable-length, high-volume recall/rerank/embed traffic
+the engine generates (documents and candidate sets of every size), that per-shape
+cache grows without bound. A local instance was observed idling at ~20 GB — ~9.4 GB
+of Metal graphics memory plus ~8 GB of native heap, essentially all of it stale
+per-shape MPS cache. CPU inference has no per-shape cache: the same workload holds
+flat at a few hundred MB, with negligible latency cost for the small default
+models (and MPS actually *slows down* over time as it recompiles graphs for new
+shapes). So MPS is excluded from auto-detection and must be opted into explicitly;
+CUDA and Intel XPU still auto-select.
+
+This is a confirmed, still-open PyTorch bug in the MPSGraph compilation cache
+(keyed on tensor shape, no eviction path). We are tracking it upstream:
+  - https://github.com/pytorch/pytorch/issues/181213
+    ([MPS] unbounded RSS growth with varying-shape inference — our exact case)
+  - https://github.com/pytorch/pytorch/issues/164299 (graphCache identified as
+    the primary leak culprit)
+  - https://github.com/pytorch/pytorch/issues/182815 (proposes, but has not yet
+    shipped, a torch.mps.invalidate_graph_cache() API / PYTORCH_MPS_DISABLE_GRAPH_CACHE
+    env var that would let us keep MPS)
+No released mitigation exists today: empty_cache(), synchronize(),
+PYTORCH_MPS_HIGH_WATERMARK_RATIO, and autorelease pools were all confirmed
+ineffective upstream. Revisit MPS-as-default once one of those knobs lands.
+
+**2. Memory release after each batch.**
+Local CPU inference allocates large transient numpy/tensor buffers per call. The
+allocator keeps those freed pages as a high-water mark, so RSS grows monotonically
+across many calls (issue #1717). We return them to the OS after each batch —
+``malloc_trim`` on glibc/Linux, ``malloc_zone_pressure_relief`` on macOS (the
+original #1717 fix covered only Linux). When the model ran on a GPU we also empty
+that backend's allocator pool via ``torch.<backend>.empty_cache()``.
+"""
+
+from __future__ import annotations
+
+import ctypes
+import ctypes.util
+import gc
+import logging
+import sys
+
+logger = logging.getLogger(__name__)
+
+
+def select_local_device(force_cpu: bool, allow_mps: bool) -> str | None:
+    """Choose the device for a local SentenceTransformer / CrossEncoder.
+
+    Returns a value suitable to pass as the model's ``device`` argument:
+
+    - ``"cpu"``  — forced CPU, or the only accelerator is MPS and it is not allowed.
+    - ``None``   — let sentence-transformers auto-detect (picks CUDA / XPU,
+                   handling multi-GPU correctly).
+    - ``"mps"``  — Apple Silicon GPU, only when ``allow_mps`` is set.
+
+    MPS is never auto-selected because its per-shape cache leaks unbounded memory
+    under the engine's variable-length workload (see the module docstring). Set the
+    matching ``*_ALLOW_MPS`` config flag to opt back in.
+    """
+    if force_cpu:
+        return "cpu"
+
+    try:
+        import torch
+
+        if torch.cuda.is_available():
+            return None  # auto-detect CUDA
+        if hasattr(torch, "xpu") and torch.xpu.is_available():
+            return None  # auto-detect Intel XPU
+
+        mps_available = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
+        if mps_available:
+            if allow_mps:
+                return "mps"
+            logger.info(
+                "Local model: MPS (Apple Silicon GPU) is available but disabled by "
+                "default because its per-shape cache leaks memory under variable-length "
+                "workloads; running on CPU. Set the *_ALLOW_MPS flag to opt in."
+            )
+            return "cpu"
+        return "cpu"
+    except Exception as e:  # pragma: no cover - defensive
+        logger.warning("Local device detection failed, falling back to CPU: %s", e)
+        return "cpu"
+
+
+def resolve_model_device_type(model: object) -> str:
+    """Best-effort device *type* ("cpu" / "cuda" / "mps" / "xpu") of a loaded model.
+
+    Used to decide which GPU allocator pool to empty after inference. Falls back to
+    ``"cpu"`` (the safe no-op choice for release) if the device can't be read.
+    """
+    device = getattr(model, "device", None)
+    if device is None:
+        inner = getattr(model, "model", None)  # CrossEncoder wraps the HF model
+        device = getattr(inner, "device", None)
+    try:
+        return device.type if device is not None else "cpu"
+    except Exception:  # pragma: no cover - defensive
+        return "cpu"
+
+
+def _resolve_heap_trim():
+    """Return a callable that asks the C allocator to release freed pages to the OS.
+
+    glibc (Linux) exposes ``malloc_trim``; macOS exposes
+    ``malloc_zone_pressure_relief``. Resolved once at import; returns a no-op on
+    platforms where neither is available (musl, Windows).
+    """
+    if sys.platform == "linux":
+        libc_path = ctypes.util.find_library("c")
+        if libc_path is None:
+            return lambda: None
+        try:
+            libc = ctypes.CDLL(libc_path)
+            trim = libc.malloc_trim
+        except (OSError, AttributeError):
+            # Not glibc (musl has no malloc_trim) or libc lookup failed.
+            return lambda: None
+        trim.argtypes = [ctypes.c_size_t]
+        trim.restype = ctypes.c_int
+        return lambda: trim(0)
+
+    if sys.platform == "darwin":
+        try:
+            libc = ctypes.CDLL("/usr/lib/libSystem.dylib")
+            default_zone = libc.malloc_default_zone
+            default_zone.restype = ctypes.c_void_p
+            relief = libc.malloc_zone_pressure_relief
+            relief.argtypes = [ctypes.c_void_p, ctypes.c_size_t]
+            relief.restype = ctypes.c_size_t
+        except (OSError, AttributeError):
+            return lambda: None
+        # pressure_relief(zone, goal=0) reclaims as much as possible.
+        return lambda: relief(default_zone(), 0)
+
+    return lambda: None
+
+
+_heap_trim = _resolve_heap_trim()
+
+
+def _empty_gpu_cache(device_type: str | None) -> None:
+    """Empty the allocator pool of the GPU backend the model ran on, if any."""
+    if not device_type or device_type == "cpu":
+        return
+    try:
+        import torch
+
+        backend = getattr(torch, device_type, None)  # torch.cuda / torch.mps / torch.xpu
+        if backend is not None and hasattr(backend, "empty_cache"):
+            backend.empty_cache()
+    except Exception:  # pragma: no cover - defensive
+        pass
+
+
+def release_local_inference_memory(device_type: str | None = None) -> None:
+    """Release transient heap (and GPU allocator) memory after a local inference batch.
+
+    Frees Python objects, returns freed native pages to the OS, and empties the GPU
+    allocator pool when the model ran on a GPU. Safe to call on every platform and
+    device; the pieces that don't apply are cheap no-ops.
+    """
+    gc.collect()
+    _heap_trim()
+    _empty_gpu_cache(device_type)

hindsight-api-slim/hindsight_api/engine/providers/claude_code_llm.py

@@ -238,6 +238,13 @@ class ClaudeCodeLLM(LLMInterface):
             max_turns=1,  # Single-turn for API-style interactions
             tools=[],  # Disable built-in tools so nothing forces a ToolSearch deferral
             allowed_tools=[],  # Disable tools for standard LLM calls
+            # Pin the configured model (issue #2881). Without this the spawned CLI
+            # runs its own default model — an Opus-class model on Pro/Max OAuth —
+            # regardless of HINDSIGHT_API_*_LLM_MODEL, while metrics/logs still print
+            # self.model, so the mismatch is invisible. The isolated CLAUDE_CONFIG_DIR
+            # (fresh temp dir) means a host settings.json can't reach the CLI either,
+            # so passing it through here is the only channel.
+            model=self.model or None,
             env=_get_isolated_claude_env(),
         )
 
@@ -317,7 +324,7 @@ class ClaudeCodeLLM(LLMInterface):
 
                 # Record trace span
                 try:
-                    from hindsight_api.tracing import get_span_recorder
+                    from hindsight_api.tracing import _serialize_for_span, get_span_recorder
 
                     span_recorder = get_span_recorder()
                     span_recorder.record_llm_call(
@@ -325,15 +332,17 @@ class ClaudeCodeLLM(LLMInterface):
                         model=self.model,
                         scope=scope,
                         messages=messages,
-                        response_content=result if isinstance(result, str) else result.model_dump_json(),
+                        response_content=_serialize_for_span(result),
                         input_tokens=estimated_input,
                         output_tokens=estimated_output,
                         duration=duration,
                         finish_reason=None,
                         error=None,
                     )
-                except Exception:
-                    pass  # logging failure must never affect the operation
+                except Exception as span_error:
+                    # Tracing must remain best-effort, but expose instrumentation
+                    # bugs that would otherwise silently erase spans (#3025).
+                    logger.debug("Claude Code span recording failed: %s", span_error, exc_info=True)
 
                 # Log slow calls
                 if duration > 10.0:
@@ -532,16 +541,33 @@ class ClaudeCodeLLM(LLMInterface):
         # else: tool_choice == "auto" or unspecified - use default behavior (no changes needed)
 
         # Configure SDK options with MCP server
+        #
         # tools=[] disables built-in CLI tools (Read, Write, Bash, ToolSearch, etc.)
         # Without this, Claude Code CLI defers MCP tools when too many built-in tools
-        # are loaded, forcing Claude to use ToolSearch first — which wastes the max_turns
+        # are loaded, forcing Claude to use ToolSearch first — which wastes the turn
         # budget and prevents direct MCP tool calls.
+        #
+        # max_turns=1 is critical (issue #2966). call_with_tools() is one *round* of
+        # an agentic loop the caller drives: the model proposes tool calls, we return
+        # them, and the orchestrator (reflect/agent.py) executes the REAL tools and
+        # feeds the results back on the next call. The SDK, however, runs its own
+        # in-process loop: it invokes our SDK MCP handlers — which are deliberate
+        # placeholders returning "[Tool <name> called successfully]" (no real data) —
+        # and lets the model react. With max_turns >= 2 the model calls recall, sees
+        # the empty placeholder, re-queries with reworded searches, exhausts the turn
+        # budget, and the run ends in error_max_turns with its tool calls discarded —
+        # exactly the "0 tool calls / no information" failure in #2966. Capping at a
+        # single turn stops the SDK from acting on the placeholder results: the model
+        # emits its first tool call (or a text answer) and we return that to the caller
+        # unchanged, matching how every other provider's call_with_tools() behaves.
         options = ClaudeAgentOptions(
             system_prompt=system_prompt if system_prompt else None,
             tools=[],  # Disable built-in tools so MCP tools load eagerly
-            max_turns=2,  # Allow tool call + tool result round-trip
+            max_turns=1,  # One round: propose tool calls (or answer); caller drives the loop
             mcp_servers=mcp_servers_config,
             allowed_tools=allowed_tool_names,
+            # Pin the configured model (issue #2881) — see the call() options block.
+            model=self.model or None,
             env=_get_isolated_claude_env(),
         )
 
@@ -560,9 +586,6 @@ class ClaudeCodeLLM(LLMInterface):
 
                     # Receive response
                     async for message in client.receive_response():
-                        if isinstance(message, ResultMessage) and message.is_error:
-                            # Surface the CLI's actual error text (issue #2702).
-                            raise RuntimeError(_result_error_detail(message))
                         if isinstance(message, AssistantMessage):
                             for block in message.content:
                                 if isinstance(block, TextBlock):
@@ -581,6 +604,21 @@ class ClaudeCodeLLM(LLMInterface):
                                             arguments=block.input,
                                         )
                                     )
+                            if tool_calls:
+                                # This round proposed tool call(s). Stop consuming the
+                                # stream so the SDK does not run another turn against our
+                                # placeholder handlers (issue #2966) — the caller executes
+                                # the real tools and calls us again with the results.
+                                break
+                        elif isinstance(message, ResultMessage) and message.is_error:
+                            # With max_turns=1 the CLI reports error_max_turns whenever
+                            # the model spent its single turn issuing a tool call (there
+                            # was no follow-up turn to emit final text). That is expected
+                            # here and not a failure: we already captured the tool call
+                            # above and break before reaching this branch. Only a genuine
+                            # error with nothing to return should surface (issue #2702).
+                            if not tool_calls:
+                                raise RuntimeError(_result_error_detail(message))
 
                 # Record metrics
                 duration = time.time() - start_time

hindsight-api-slim/hindsight_api/engine/providers/codex_llm.py

@@ -172,8 +172,8 @@ class CodexLLM(LLMInterface):
         if self.model.startswith("openai/"):
             self.model = self.model[len("openai/") :]
 
-        # Map reasoning effort to Codex reasoning summary format
-        # Codex supports: "auto", "concise", "detailed"
+        # Reasoning summary controls presentation separately from the backend's
+        # reasoning effort, which is sent unchanged in each request payload.
         self.reasoning_summary = self._map_reasoning_effort(reasoning_effort)
 
         # HTTP client for SSE streaming
@@ -448,7 +448,7 @@ class CodexLLM(LLMInterface):
             "tools": [],
             "tool_choice": "auto",
             "parallel_tool_calls": True,
-            "reasoning": {"summary": reasoning_summary},
+            "reasoning": {"effort": self.reasoning_effort, "summary": reasoning_summary},
             "store": False,  # Codex uses stateless mode
             "stream": True,  # SSE streaming
             "include": ["reasoning.encrypted_content"],
@@ -573,7 +573,7 @@ class CodexLLM(LLMInterface):
 
                 # Record trace span
                 try:
-                    from hindsight_api.tracing import get_span_recorder
+                    from hindsight_api.tracing import _serialize_for_span, get_span_recorder
 
                     # Estimate tokens for tracing
                     estimated_input = sum(len(m.get("content", "")) for m in messages) // 4
@@ -584,15 +584,17 @@ class CodexLLM(LLMInterface):
                         model=self.model,
                         scope=scope,
                         messages=messages,
-                        response_content=result if isinstance(result, str) else result.model_dump_json(),
+                        response_content=_serialize_for_span(result),
                         input_tokens=estimated_input,
                         output_tokens=estimated_output,
                         duration=duration,
                         finish_reason=None,
                         error=None,
                     )
-                except Exception:
-                    pass  # logging failure must never affect the operation
+                except Exception as span_error:
+                    # Tracing must remain best-effort, but expose instrumentation
+                    # bugs that would otherwise silently erase spans (#3025).
+                    logger.debug("Codex span recording failed: %s", span_error, exc_info=True)
 
                 if return_usage:
                     # Codex doesn't provide token counts, estimate based on content
@@ -831,7 +833,7 @@ class CodexLLM(LLMInterface):
                 else tool_choice.mode.value
             ),
             "parallel_tool_calls": True,
-            "reasoning": {"summary": reasoning_summary},
+            "reasoning": {"effort": self.reasoning_effort, "summary": reasoning_summary},
             "store": False,
             "stream": True,
             "include": ["reasoning.encrypted_content"],

hindsight-api-slim/hindsight_api/engine/providers/gemini_llm.py

@@ -424,8 +424,7 @@ class GeminiLLM(LLMInterface):
         last_exception = None
 
         for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.gemini.{scope}.attempt={attempt + 1}/{max_retries + 1}")
+            set_stage(f"llm.gemini.{scope}.attempt={attempt + 1}/{max_retries + 1}")
             try:
                 response = await asyncio.wait_for(
                     self._client.aio.models.generate_content(
@@ -769,8 +768,7 @@ class GeminiLLM(LLMInterface):
 
         last_exception = None
         for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.gemini.tools.attempt={attempt + 1}/{max_retries + 1}")
+            set_stage(f"llm.gemini.tools.attempt={attempt + 1}/{max_retries + 1}")
             try:
                 # With the cache active, send only the un-cached tail (delta);
                 # on the uncached fallback path send the full conversation so the

hindsight-api-slim/hindsight_api/engine/providers/litellm_llm.py

@@ -255,8 +255,7 @@ class LiteLLMLLM(LLMInterface):
         last_exception = None
 
         for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.{self._stage_label}.{scope}.attempt={attempt + 1}/{max_retries + 1}")
+            set_stage(f"llm.{self._stage_label}.{scope}.attempt={attempt + 1}/{max_retries + 1}")
             try:
                 response = await asyncio.wait_for(
                     self._acompletion(**call_kwargs),
@@ -447,8 +446,7 @@ class LiteLLMLLM(LLMInterface):
 
         last_exception = None
         for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.{self._stage_label}.tools.attempt={attempt + 1}/{max_retries + 1}")
+            set_stage(f"llm.{self._stage_label}.tools.attempt={attempt + 1}/{max_retries + 1}")
             try:
                 response = await asyncio.wait_for(
                     self._acompletion(**call_kwargs),

hindsight-api-slim/hindsight_api/engine/providers/mock_llm.py

@@ -9,7 +9,7 @@ import logging
 from collections.abc import Callable
 from typing import Any
 
-from ..llm_interface import LLM_TOOL_CHOICE_AUTO, LLMInterface, LLMToolChoice
+from ..llm_interface import LLM_TOOL_CHOICE_AUTO, LLMInterface, LLMToolChoice, LLMToolChoiceMode
 from ..response_models import LLMToolCall, LLMToolCallResult, TokenUsage
 
 logger = logging.getLogger(__name__)
@@ -266,7 +266,7 @@ class MockLLM(LLMInterface):
             else:
                 result = LLMToolCallResult(content="mock response", finish_reason="stop")
         else:
-            result = LLMToolCallResult(content="mock response", finish_reason="stop")
+            result = self._compliant_tool_call(tools, tool_choice, messages)
 
         # Set mock token usage on result if not already set
         if result.input_tokens == 0:
@@ -297,6 +297,61 @@ class MockLLM(LLMInterface):
 
         return result
 
+    @staticmethod
+    def _compliant_tool_call(
+        tools: list[dict[str, Any]],
+        tool_choice: LLMToolChoice,
+        messages: list[dict[str, Any]],
+    ) -> LLMToolCallResult:
+        """Default tool response: simulate a compliant tool-calling model.
+
+        Real providers drive the reflect loop entirely through tool calls -- they
+        honor a forced tool choice, then finish via ``done`` -- and the reflect
+        agent now rejects a turn that yields no tool call at all (a transport that
+        can't tool-call raises ReflectToolCallError). So the mock must behave like a
+        working provider here rather than returning bare "mock response" prose,
+        which used to be salvaged as the answer. Only this default path is affected;
+        tests that script turns via ``_response_callback`` / ``_mock_response`` are not.
+        """
+        tool_names = {t.get("function", {}).get("name") for t in tools}
+
+        def _mock_query() -> str:
+            for message in reversed(messages):
+                content = message.get("content")
+                if message.get("role") == "user" and isinstance(content, str) and content.strip():
+                    return content[:200]
+            return "mock query"
+
+        # Honor a forced retrieval tool so the loop actually runs recall/search and
+        # gathers evidence (populates based_on for tests that assert on it).
+        if tool_choice.mode is LLMToolChoiceMode.NAMED and tool_choice.function_name in {
+            "search_mental_models",
+            "search_observations",
+            "recall",
+        }:
+            return LLMToolCallResult(
+                tool_calls=[
+                    LLMToolCall(
+                        id="mock_forced",
+                        name=tool_choice.function_name,
+                        arguments={"reason": "mock", "query": _mock_query()},
+                    )
+                ],
+                finish_reason="tool_calls",
+            )
+
+        # Auto turn: finish via the done tool, mirroring a model that has gathered
+        # enough. The reflect evidence guardrail handles the empty-bank case (no
+        # evidence -> forced text synthesis on the final iteration).
+        if "done" in tool_names:
+            return LLMToolCallResult(
+                tool_calls=[LLMToolCall(id="mock_done", name="done", arguments={"answer": "mock response"})],
+                finish_reason="tool_calls",
+            )
+
+        # No done tool offered (non-reflect tool call): fall back to plain text.
+        return LLMToolCallResult(content="mock response", finish_reason="stop")
+
     @staticmethod
     def _build_mock_facts(messages: list[dict]) -> dict:
         """Build a canned fact extraction response from the user message text.

hindsight-api-slim/hindsight_api/engine/providers/openai_compatible_llm.py

@@ -83,6 +83,30 @@ def _validate_ollama_num_ctx(value: Any) -> int | None:
 # intentionally excluded (#1179).
 _TOOL_CHOICE_REQUIRED_UNSUPPORTED_PROVIDERS = frozenset({"lmstudio", "ollama"})
 
+# Local providers whose OpenAI-compatible surface always lives under a `/v1`
+# path (LM Studio: http://localhost:1234/v1, Ollama: http://localhost:11434/v1).
+# For these we know the exact endpoint shape, so a bare host base URL can be
+# normalized safely. Cloud/proxy endpoints are left untouched — their path is
+# provider-specific and must be supplied verbatim.
+_V1_PATH_LOCAL_PROVIDERS = frozenset({"lmstudio", "ollama"})
+
+
+def _ensure_v1_base_url(base_url: str) -> str:
+    """Append the OpenAI-compatible ``/v1`` prefix to a bare local base URL.
+
+    LM Studio's server UI advertises its address as ``http://localhost:1234``,
+    so users commonly set ``HINDSIGHT_API_LLM_BASE_URL`` to that bare host. The
+    OpenAI SDK then POSTs to ``<host>/chat/completions`` and LM Studio rejects it
+    with ``Unexpected endpoint or method`` — its OpenAI-compatible routes live
+    under ``/v1``. Only a base URL with no meaningful path (bare host or a lone
+    trailing slash) is rewritten; anything with an explicit path (e.g. a reverse
+    proxy mount or an already-correct ``/v1``) is returned unchanged. See #2922.
+    """
+    parsed = urlparse(base_url)
+    if parsed.path.strip("/"):
+        return base_url
+    return urlunparse(parsed._replace(path="/v1"))
+
 
 class ProviderResponseError(RuntimeError):
     """Raised when a provider returns a success response without usable content."""
@@ -517,7 +541,10 @@ class OpenAICompatibleLLM(LLMInterface):
             api_key: API key (optional for ollama/lmstudio).
             base_url: Base URL for the API (uses defaults for groq/ollama/lmstudio if empty).
             model: Model name.
-            reasoning_effort: Reasoning effort level for supported models ("low", "medium", "high").
+            reasoning_effort: Reasoning effort level for supported models
+                ("none", "low", "medium", "high"). "none" is required when calling
+                function tools on some reasoning models, which reject every other
+                value — including omitting the parameter entirely.
             timeout: Request timeout in seconds (uses env var or 120s default).
             groq_service_tier: Groq service tier ("on_demand", "flex", "auto").
             extra_body: Extra body params merged into every API call.
@@ -577,6 +604,11 @@ class OpenAICompatibleLLM(LLMInterface):
                 # lives on a separate control-plane host — see FireworksLLM.
                 self.base_url = "https://api.fireworks.ai/inference/v1"
 
+        # Normalize bare local base URLs (e.g. a user pasting the address shown
+        # in the LM Studio UI) so the OpenAI SDK targets the `/v1` routes. See #2922.
+        if self.provider in _V1_PATH_LOCAL_PROVIDERS and self.base_url:
+            self.base_url = _ensure_v1_base_url(self.base_url)
+
         # For ollama/lmstudio, use dummy key if not provided
         if self.provider in ("ollama", "lmstudio") and not self.api_key:
             self.api_key = "local"
@@ -870,8 +902,7 @@ class OpenAICompatibleLLM(LLMInterface):
             # Surface attempt count in worker stage so JSON-schema retry loops
             # are visible from logs (small models on strict structured output
             # often loop here). Cheap no-op outside worker context.
-            if attempt > 0:
-                set_stage(f"llm.{self.provider}.{scope}.attempt={attempt + 1}/{max_retries + 1}")
+            set_stage(f"llm.{self.provider}.{scope}.attempt={attempt + 1}/{max_retries + 1}")
             try:
                 if response_format is not None:
                     response = await self._client.chat.completions.create(**call_params)
@@ -1227,6 +1258,13 @@ class OpenAICompatibleLLM(LLMInterface):
                 temperature = max(0.01, min(temperature, 1.0))
             call_params["temperature"] = temperature
 
+        # Set reasoning_effort for reasoning models, matching call(). Omitting it
+        # here is not a neutral default: OpenAI rejects function tools on a
+        # reasoning model unless reasoning_effort is present and set to "none",
+        # so leaving it out fails exactly like sending an unsupported value.
+        if self._supports_reasoning_model():
+            call_params["reasoning_effort"] = self.reasoning_effort
+
         # Provider-specific parameters
         extra_body: dict[str, Any] = {**self._config_extra_body}
         self._apply_provider_extra_body_defaults(extra_body)
@@ -1240,8 +1278,7 @@ class OpenAICompatibleLLM(LLMInterface):
         last_exception = None
 
         for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.{self.provider}.tools.attempt={attempt + 1}/{max_retries + 1}")
+            set_stage(f"llm.{self.provider}.tools.attempt={attempt + 1}/{max_retries + 1}")
             try:
                 response = await self._client.chat.completions.create(**call_params)
 
@@ -1445,8 +1482,7 @@ class OpenAICompatibleLLM(LLMInterface):
 
         async with httpx.AsyncClient(timeout=300.0) as client:
             for attempt in range(max_retries + 1):
-                if attempt > 0:
-                    set_stage(f"llm.ollama_native.{scope}.attempt={attempt + 1}/{max_retries + 1}")
+                set_stage(f"llm.ollama_native.{scope}.attempt={attempt + 1}/{max_retries + 1}")
                 try:
                     response = await client.post(native_url, json=payload, headers=headers)
                     response.raise_for_status()

hindsight-api-slim/hindsight_api/engine/reflect/__init__.py

@@ -7,12 +7,13 @@ The reflect agent uses an iterative loop with tools to:
 3. Expand memories (get chunk/document context)
 """
 
-from .agent import ReflectAgentResult, run_reflect_agent
+from .agent import ReflectAgentResult, ReflectToolCallError, run_reflect_agent
 from .models import ReflectAction, ReflectActionBatch
 
 __all__ = [
     "run_reflect_agent",
     "ReflectAgentResult",
+    "ReflectToolCallError",
     "ReflectAction",
     "ReflectActionBatch",
 ]

hindsight-api-slim/hindsight_api/engine/reflect/agent.py

@@ -10,7 +10,6 @@ Uses hierarchical retrieval:
 import asyncio
 import json
 import logging
-import re
 import time
 from typing import TYPE_CHECKING, Any, Awaitable, Callable
 
@@ -56,6 +55,20 @@ DEFAULT_MAX_ITERATIONS = 10
 NO_ANSWER_TEXT = "No answer provided."
 
 
+class ReflectToolCallError(RuntimeError):
+    """The model never produced a tool call reflect could understand.
+
+    Reflect is driven entirely by structured tool calls (``recall``, ``expand``,
+    ``done`` ...). Some provider transports do not actually support function
+    calling and silently drop the tool definitions from the request (e.g. litellm's
+    Vertex AI gpt-oss MaaS path strips ``tools``/``tool_choice`` when the model is
+    flagged as not supporting them). The model then answers in free text that may
+    mimic a ``done`` payload. Rather than salvage that untooled text -- and risk
+    surfacing raw tool-call JSON as the answer -- we fail loudly so the caller can
+    switch to a tool-calling-capable model/transport.
+    """
+
+
 def _normalize_tool_name(name: str) -> str:
     """Normalize tool name from various LLM output formats.
 
@@ -88,143 +101,6 @@ def _is_done_tool(name: str) -> bool:
     return _normalize_tool_name(name) == "done"
 
 
-# Pattern to match done() call as text - handles done({...}) with nested JSON
-_DONE_CALL_PATTERN = re.compile(r"done\s*\(\s*\{.*$", re.DOTALL)
-
-# Patterns for leaked structured output in the answer field
-_LEAKED_JSON_SUFFIX = re.compile(
-    r'\s*```(?:json)?\s*\{[^}]*(?:"(?:observation_ids|memory_ids|mental_model_ids)"|\})\s*```\s*$',
-    re.DOTALL | re.IGNORECASE,
-)
-_TRAILING_IDS_PATTERN = re.compile(
-    r"\s*(?:observation_ids|memory_ids|mental_model_ids)\s*[=:]\s*\[.*?\]\s*$", re.DOTALL | re.IGNORECASE
-)
-_JSON_CODE_FENCE_PATTERN = re.compile(r"^\s*```(?:json)?\s*(\{.*\})\s*```\s*$", re.DOTALL | re.IGNORECASE)
-
-_DONE_ARGUMENT_KEYS = frozenset(
-    {
-        "answer",
-        "directive_compliance",
-        "memory_ids",
-        "mental_model_ids",
-        "observation_ids",
-        "model_ids",
-    }
-)
-_DONE_ARGUMENT_MARKER_KEYS = _DONE_ARGUMENT_KEYS - {"answer"}
-_LEAKED_JSON_ID_KEYS = frozenset({"memory_ids", "mental_model_ids", "observation_ids", "model_ids"})
-
-
-def _unwrap_leaked_done_arguments(text: str) -> str | None:
-    """Return the answer when a done tool call was rendered as JSON text.
-
-    Some providers leak the done tool's argument object instead of surfacing it
-    as a native tool call, e.g. {"answer": "...", "memory_ids": [...]}. Only
-    unwrap objects that match the done argument shape so normal JSON answers
-    stay intact.
-    """
-    candidate = text.strip()
-    if not candidate:
-        return None
-
-    fenced = _JSON_CODE_FENCE_PATTERN.match(candidate)
-    if fenced:
-        candidate = fenced.group(1).strip()
-
-    try:
-        payload = json.loads(candidate)
-    except json.JSONDecodeError:
-        return None
-
-    if not isinstance(payload, dict):
-        return None
-    answer = payload.get("answer")
-    if not isinstance(answer, str) or not answer.strip():
-        return None
-
-    keys = set(payload)
-    if not keys.intersection(_DONE_ARGUMENT_MARKER_KEYS):
-        return None
-    if not keys.issubset(_DONE_ARGUMENT_KEYS):
-        return None
-
-    for key in ("memory_ids", "mental_model_ids", "observation_ids", "model_ids"):
-        value = payload.get(key)
-        if value is not None and not isinstance(value, list):
-            return None
-
-    return answer.strip()
-
-
-def _strip_trailing_id_json_object(text: str) -> str:
-    stripped = text.rstrip()
-    if not stripped.endswith("}"):
-        return text.strip()
-
-    start = stripped.rfind("{")
-    if start < 0:
-        return text.strip()
-
-    try:
-        payload = json.loads(stripped[start:])
-    except json.JSONDecodeError:
-        return text.strip()
-
-    if not isinstance(payload, dict) or not payload:
-        return text.strip()
-    keys = set(payload)
-    if not keys.issubset(_LEAKED_JSON_ID_KEYS):
-        return text.strip()
-
-    return stripped[:start].strip()
-
-
-def _clean_answer_text(text: str) -> str:
-    """Clean up answer text by removing any done() tool call syntax.
-
-    Some LLMs output the done() call as text instead of a proper tool call.
-    This strips out patterns like: done({"answer": "...", ...})
-    """
-    unwrapped = _unwrap_leaked_done_arguments(text)
-    if unwrapped is not None:
-        return unwrapped
-
-    # Remove done() call pattern from the end of the text
-    cleaned = _DONE_CALL_PATTERN.sub("", text).strip()
-    return cleaned if cleaned else text
-
-
-def _clean_done_answer(text: str) -> str:
-    """Clean up the answer field from a done() tool call.
-
-    Some LLMs leak structured output patterns into the answer text, such as:
-    - JSON code blocks with observation_ids/memory_ids at the end
-    - Raw JSON objects with these fields
-    - Plain text like "observation_ids: [...]"
-
-    This cleans those patterns while preserving the actual answer content.
-    """
-    if not text:
-        return text
-
-    unwrapped = _unwrap_leaked_done_arguments(text)
-    if unwrapped is not None:
-        return unwrapped
-
-    cleaned = text
-
-    # Remove leaked JSON in code blocks at the end
-    cleaned = _LEAKED_JSON_SUFFIX.sub("", cleaned).strip()
-
-    # Remove leaked raw JSON objects at the end
-    cleaned = _strip_trailing_id_json_object(cleaned)
-
-    # Remove trailing ID patterns
-    cleaned = _TRAILING_IDS_PATTERN.sub("", cleaned).strip()
-
-    return cleaned if cleaned else text
-
-
 async def _generate_structured_output(
     answer: str,
     response_schema: dict,
@@ -544,6 +420,7 @@ async def _run_reflect_agent_inner(
     max_context_tokens: int = 100_000,
     llm_output_language: str | None = None,
     cancel_check: Callable[[], None] | None = None,
+    store_document_text: bool = True,
     *,
     reflect_id: str,
     provider_impl: Any,
@@ -587,8 +464,8 @@ async def _run_reflect_agent_inner(
 
     # Get tools for this agent (with directive compliance field if directives exist).
     # The expand tool only reads back raw source text (chunks/documents), so it is
-    # useless and excluded when document text storage is disabled.
-    include_expand = get_config().store_document_text
+    # useless and excluded when document text storage is disabled (per bank).
+    include_expand = store_document_text
     tools = get_reflect_tools(
         directive_rules=directive_rules,
         include_mental_models=has_mental_models,
@@ -678,6 +555,10 @@ async def _run_reflect_agent_inner(
 
     # Tracking
     total_tools_called = 0
+    # Whether the model has ever produced a tool call reflect could understand.
+    # Stays False when a transport silently strips tool support (the model then
+    # only ever returns free text) -- that case fails via ReflectToolCallError.
+    saw_tool_call = False
     tool_trace: list[ToolCall] = []
     tool_trace_summary: list[dict[str, Any]] = []
     llm_trace: list[dict[str, Any]] = []
@@ -791,7 +672,7 @@ async def _run_reflect_agent_inner(
                     "output_tokens": usage.output_tokens,
                 }
             )
-            answer = _clean_answer_text(response.strip())
+            answer = response.strip()
 
             # Generate structured output if schema provided
             structured_output = None
@@ -856,7 +737,7 @@ async def _run_reflect_agent_inner(
                     "output_tokens": usage.output_tokens,
                 }
             )
-            answer = _clean_answer_text(response.strip())
+            answer = response.strip()
 
             structured_output = None
             if response_schema and answer:
@@ -999,7 +880,7 @@ async def _run_reflect_agent_inner(
                     "output_tokens": usage.output_tokens,
                 }
             )
-            answer = _clean_answer_text(response.strip())
+            answer = response.strip()
 
             # Generate structured output if schema provided
             structured_output = None
@@ -1023,85 +904,29 @@ async def _run_reflect_agent_inner(
                 directives_applied=directives_applied,
             )
 
-        # No tool calls - LLM wants to respond with text
+        # No tool calls this turn.
         if not result.tool_calls:
-            # When directives are present but no evidence has been gathered,
-            # the LLM tends to echo directive content verbatim as its answer.
-            # Fall through to the final-prompt path which doesn't include
-            # directives and handles "no data" gracefully.
-            has_gathered_evidence = (
-                bool(available_memory_ids) or bool(available_mental_model_ids) or bool(available_observation_ids)
-            )
-            directive_leak_risk = directives and not has_gathered_evidence
-            if result.content and not directive_leak_risk:
-                answer = _clean_answer_text(result.content.strip())
-
-                # The call_with_tools call above is intentionally uncapped so the
-                # LLM has headroom to emit tool-call JSON plus any intermediate
-                # reasoning. But when the LLM short-circuits and returns text
-                # directly, that text becomes the user-visible final answer and
-                # must respect max_tokens like the forced-final paths do. If it
-                # overshoots, run one extra capped call to rewrite it within
-                # the cap.
-                if max_tokens is not None and count_cl100k_tokens(answer) > max_tokens:
-                    rewrite_start = time.time()
-                    rewritten, rewrite_usage = await llm_config.call(
-                        messages=[
-                            {
-                                "role": "system",
-                                "content": (
-                                    "Rewrite the user's text so it fits within the requested token "
-                                    "budget. Preserve the key facts and structure; drop lower-priority "
-                                    "detail. Respond with the rewritten text only, no preamble."
-                                ),
-                            },
-                            {
-                                "role": "user",
-                                "content": f"Target budget: {max_tokens} tokens.\n\nText to rewrite:\n{answer}",
-                            },
-                        ],
-                        scope="reflect",
-                        max_completion_tokens=max_tokens,
-                        return_usage=True,
-                    )
-                    total_input_tokens += rewrite_usage.input_tokens
-                    total_output_tokens += rewrite_usage.output_tokens
-                    total_cached_tokens += getattr(rewrite_usage, "cached_tokens", 0) or 0
-                    total_thoughts_tokens += getattr(rewrite_usage, "thoughts_tokens", 0) or 0
-                    llm_trace.append(
-                        {
-                            "scope": "final_rewrite",
-                            "duration_ms": int((time.time() - rewrite_start) * 1000),
-                            "input_tokens": rewrite_usage.input_tokens,
-                            "output_tokens": rewrite_usage.output_tokens,
-                        }
-                    )
-                    answer = _clean_answer_text(rewritten.strip())
-
-                # Generate structured output if schema provided
-                structured_output = None
-                if response_schema and answer:
-                    struct = await _generate_structured_output(
-                        answer, response_schema, llm_config, reflect_id, max_tokens
-                    )
-                    structured_output = struct.structured_output
-                    total_input_tokens += struct.input_tokens
-                    total_output_tokens += struct.output_tokens
-                    total_cached_tokens += struct.cached_tokens
-                    total_thoughts_tokens += struct.thoughts_tokens
-
-                _log_completion(answer, iteration + 1)
-                return ReflectAgentResult(
-                    text=answer,
-                    structured_output=structured_output,
-                    iterations=iteration + 1,
-                    tools_called=total_tools_called,
-                    tool_trace=tool_trace,
-                    llm_trace=_get_llm_trace(),
-                    usage=_get_usage(),
-                    directives_applied=directives_applied,
+            # Reflect is driven by structured tool calls. A turn with no tool call
+            # means one of two things:
+            #   * the model already gathered evidence via earlier tool calls and is
+            #     now stopping -- fine, synthesize a clean final answer below;
+            #   * the transport can't produce tool calls at all, so it only ever
+            #     returns free text (e.g. litellm strips tools on the Vertex gpt-oss
+            #     MaaS path). In that case ``saw_tool_call`` is still False.
+            # We no longer salvage that free text as the answer -- it can be a raw
+            # done()-payload with sibling id fields leaking into user-visible text.
+            # Fail loudly instead so the caller picks a tool-calling-capable model.
+            if not saw_tool_call:
+                snippet = (result.content or "").strip()
+                if len(snippet) > 500:
+                    snippet = snippet[:500] + "..."
+                detail = f" Response: {snippet!r}" if snippet else " The model returned no content."
+                raise ReflectToolCallError(
+                    f"Reflect requires a tool-calling model, but {llm_config.provider}/{llm_config.model} "
+                    f"produced no usable tool call (the transport may not support function calling)." + detail
                 )
-            # Empty response, force final
+            # Model tool-called earlier and is now stopping: fall through to a clean
+            # forced final synthesis (tools disabled, prose expected).
             prompt = build_final_prompt(
                 query, context_history, bank_profile, context, max_context_tokens=max_context_tokens
             )
@@ -1133,7 +958,7 @@ async def _run_reflect_agent_inner(
                     "output_tokens": usage.output_tokens,
                 }
             )
-            answer = _clean_answer_text(response.strip())
+            answer = response.strip()
 
             # Generate structured output if schema provided
             structured_output = None
@@ -1157,6 +982,11 @@ async def _run_reflect_agent_inner(
                 directives_applied=directives_applied,
             )
 
+        # The model produced at least one tool call reflect could parse: it can
+        # drive the loop, so a later text-only turn is a legitimate stop, not a
+        # broken transport.
+        saw_tool_call = True
+
         # Check for done tool call (handle various LLM output formats)
         done_call = next((tc for tc in result.tool_calls if _is_done_tool(tc.name)), None)
         if done_call:
@@ -1432,9 +1262,10 @@ async def _process_done_tool(
     """Process the done tool call and return the result."""
     args = done_call.arguments
 
-    # Extract and clean the answer - some LLMs leak structured output into the answer text
-    raw_answer = args.get("answer", "").strip()
-    answer = _clean_done_answer(raw_answer) if raw_answer else ""
+    # ``done`` is a structured tool call: trust its ``answer`` field verbatim.
+    # Sibling id fields (memory_ids, ...) live in their own arguments and are
+    # validated separately below -- they can't bleed into a parsed answer string.
+    answer = args.get("answer", "").strip()
     if not answer:
         answer = NO_ANSWER_TEXT
 
@@ -1460,7 +1291,7 @@ async def _process_done_tool(
             max_completion_tokens=max_tokens,
             return_usage=True,
         )
-        answer = _clean_answer_text(rewritten.strip())
+        answer = rewritten.strip()
         final_usage = TokenUsageSummary(
             input_tokens=usage.input_tokens + rewrite_usage.input_tokens,
             output_tokens=usage.output_tokens + rewrite_usage.output_tokens,

hindsight-api-slim/hindsight_api/engine/reflect/prompts.py

@@ -421,76 +421,6 @@ def build_system_prompt_for_tools(
     return "\n".join(parts)
 
 
-def build_agent_prompt(
-    query: str,
-    context_history: list[dict],
-    bank_profile: dict,
-    additional_context: str | None = None,
-) -> str:
-    """Build the user prompt for the reflect agent."""
-    parts = []
-
-    # Bank identity
-    name = bank_profile.get("name", "Assistant")
-    mission = bank_profile.get("mission", "")
-
-    parts.append(f"## Memory Bank Context\nName: {name}")
-    if mission:
-        parts.append(f"Mission: {mission}")
-
-    # Disposition traits if present
-    disposition = bank_profile.get("disposition", {})
-    if disposition:
-        traits = []
-        if "skepticism" in disposition:
-            traits.append(f"skepticism={disposition['skepticism']}")
-        if "literalism" in disposition:
-            traits.append(f"literalism={disposition['literalism']}")
-        if "empathy" in disposition:
-            traits.append(f"empathy={disposition['empathy']}")
-        if traits:
-            parts.append(f"Disposition: {', '.join(traits)}")
-
-    # Additional context from caller
-    if additional_context:
-        parts.append(f"\n## Additional Context\n{additional_context}")
-
-    # Tool call history
-    if context_history:
-        parts.append("\n## Tool Results (synthesize and reason from this data)")
-        for i, entry in enumerate(context_history, 1):
-            tool = entry["tool"]
-            output = entry["output"]
-            # Format as proper JSON for LLM readability
-            try:
-                output_str = json.dumps(output, indent=2, default=str, ensure_ascii=False)
-            except (TypeError, ValueError):
-                output_str = str(output)
-            parts.append(f"\n### Call {i}: {tool}\n```json\n{output_str}\n```")
-
-    # The question
-    parts.append(f"\n## Question\n{query}")
-
-    # Instructions
-    if context_history:
-        parts.append(
-            "\n## Instructions\n"
-            "Based on the tool results above, either call more tools or provide your final answer. "
-            "Synthesize and reason from the data - make reasonable inferences when helpful. "
-            "If you have related information, use it to give the best possible answer."
-        )
-    else:
-        parts.append(
-            "\n## Instructions\n"
-            "Start by searching for relevant information using the hierarchical retrieval strategy:\n"
-            "1. Try search_mental_models() first for curated summaries\n"
-            "2. Try search_observations() for consolidated knowledge\n"
-            "3. Use recall() for specific details or to verify stale data"
-        )
-
-    return "\n".join(parts)
-
-
 def build_final_prompt(
     query: str,
     context_history: list[dict],
@@ -890,38 +820,3 @@ ABSOLUTE RULES:
 OUTPUT FORMAT:
 - Output ONLY the updated markdown document. No preamble, no explanation, no diff markers, no commentary.
 - Do not wrap the output in code fences unless the CURRENT DOCUMENT itself was entirely a code fence."""
-
-
-def build_delta_prompt(
-    *,
-    current_content: str,
-    candidate_content: str,
-    supporting_facts: list[dict[str, Any]],
-    source_query: str,
-) -> str:
-    """Build the user prompt for a delta-mode mental model refresh.
-
-    Args:
-        current_content: The existing mental model content (to preserve as much as possible).
-        candidate_content: Fresh synthesis from the reflect agent reflecting new reality.
-        supporting_facts: Flat list of fact dicts (id, text, type) supporting the candidate.
-        source_query: The mental model's source query, for topical framing.
-    """
-    fact_lines: list[str] = []
-    for f in supporting_facts:
-        fid = f.get("id", "")
-        text = (f.get("text") or "").strip().replace("\n", " ")
-        ftype = f.get("type", "")
-        fact_lines.append(f"- [{ftype}:{fid}] {text}")
-    facts_block = "\n".join(fact_lines) if fact_lines else "(no supporting facts retrieved)"
-
-    return (
-        f"## Topic\n{source_query}\n\n"
-        f"## CURRENT DOCUMENT\n```markdown\n{current_content}\n```\n\n"
-        f"## CANDIDATE UPDATE\n```markdown\n{candidate_content}\n```\n\n"
-        f"## SUPPORTING FACTS\n{facts_block}\n\n"
-        "## Task\n"
-        "Produce the updated mental model document by applying the minimum necessary changes "
-        "to CURRENT DOCUMENT so that it reflects CANDIDATE UPDATE and SUPPORTING FACTS. "
-        "Preserve unchanged content byte-for-byte. Output only the final markdown."
-    )

hindsight-api-slim/hindsight_api/engine/reflect/tools.py

@@ -95,8 +95,10 @@ async def tool_search_mental_models(
     params: list[Any] = [bank_id, str(query_embedding), max_results]
     next_param = 4
 
-    # Use the centralized tag filtering logic
-    if tags:
+    # Exact matching treats absent or empty tags as the global scope. Do not
+    # skip the filter, or mental models would see every scope while the other
+    # reflect retrieval tools correctly see only untagged data.
+    if tags or tags_match == "exact":
         tag_clause, tag_params, next_param = build_tags_where_clause(tags, param_offset=next_param, match=tags_match)
         filters += f" {tag_clause}"
         params.extend(tag_params)

hindsight-api-slim/hindsight_api/engine/retain/bank_utils.py

@@ -12,7 +12,7 @@ from pydantic import BaseModel, Field
 
 from ..._vector_index import index_using_clause, uses_per_bank_vector_indexes
 from ...config import get_config
-from ..db_utils import acquire_with_retry
+from ..db_utils import acquire_with_retry, retry_with_backoff
 from ..memory_engine import fq_table, get_current_schema
 from ..response_models import DispositionTraits
 
@@ -188,9 +188,19 @@ async def get_or_create_bank_profile(pool, bank_id: str) -> BankProfileResult:
     or rolls back atomically with the caller's write), use
     ``get_or_create_bank_profile_on_conn`` instead.
     """
-    async with acquire_with_retry(pool) as conn:
-        async with conn.transaction():
-            return await get_or_create_bank_profile_on_conn(conn, bank_id, ops=pool.ops)
+
+    # A fresh bank builds its per-(bank, fact_type) partial vector indexes with
+    # a plain CREATE INDEX (it must — this runs inside the bank-create tx, and
+    # CONCURRENTLY cannot). That CREATE takes a ShareLock on the shared
+    # memory_units table, which can deadlock with concurrent writers. The build
+    # is idempotent (INSERT ... ON CONFLICT + CREATE INDEX IF NOT EXISTS), so a
+    # transient deadlock (40P01 / ORA-00060) is safe to retry as a whole tx.
+    async def _create() -> BankProfileResult:
+        async with acquire_with_retry(pool) as conn:
+            async with conn.transaction():
+                return await get_or_create_bank_profile_on_conn(conn, bank_id, ops=pool.ops)
+
+    return await retry_with_backoff(_create)
 
 
 async def get_or_create_bank_profile_on_conn(conn, bank_id: str, *, ops) -> BankProfileResult:

hindsight-api-slim/hindsight_api/engine/retain/chunk_storage.py

@@ -8,7 +8,7 @@ import hashlib
 import logging
 from dataclasses import dataclass
 
-from ...config import get_config
+from ...config import _get_raw_config
 from ..memory_engine import fq_table
 from .types import ChunkMetadata
 
@@ -116,7 +116,12 @@ async def delete_chunks_by_ids(conn, chunk_ids: list[str]) -> None:
 
 
 async def store_chunks_batch(
-    conn, bank_id: str, document_id: str, chunks: list[ChunkMetadata], ops=None
+    conn,
+    bank_id: str,
+    document_id: str,
+    chunks: list[ChunkMetadata],
+    ops=None,
+    store_document_text: bool | None = None,
 ) -> dict[int, str]:
     """
     Store document chunks in the database.
@@ -127,6 +132,9 @@ async def store_chunks_batch(
         document_id: Document identifier
         chunks: List of ChunkMetadata objects
         ops: DataAccessOps instance (from backend.ops)
+        store_document_text: Whether to persist raw chunk text. When ``None``,
+            falls back to the server-level default; callers on the retain path
+            pass the per-bank resolved value.
 
     Returns:
         Dictionary mapping global chunk index to chunk_id
@@ -137,7 +145,9 @@ async def store_chunks_batch(
     # When document text storage is disabled, persist empty chunk_text (the
     # column is NOT NULL) while still computing content_hash from the real text
     # so delta-retain dedup is unaffected.
-    store_text = get_config().store_document_text
+    # Fallback to the raw global default (not get_config(), which guards
+    # bank-configurable fields); the retain path always passes the resolved value.
+    store_text = store_document_text if store_document_text is not None else _get_raw_config().store_document_text
 
     # Prepare chunk data for batch insert
     chunk_ids = []
@@ -169,21 +179,3 @@ async def store_chunks_batch(
     )
 
     return chunk_id_map
-
-
-def map_facts_to_chunks(facts_chunk_indices: list[int], chunk_id_map: dict[int, str]) -> list[str | None]:
-    """
-    Map fact chunk indices to chunk IDs.
-
-    Args:
-        facts_chunk_indices: List of chunk indices for each fact
-        chunk_id_map: Dictionary mapping chunk index to chunk_id
-
-    Returns:
-        List of chunk_ids (same length as facts_chunk_indices)
-    """
-    chunk_ids = []
-    for chunk_idx in facts_chunk_indices:
-        chunk_id = chunk_id_map.get(chunk_idx)
-        chunk_ids.append(chunk_id)
-    return chunk_ids

hindsight-api-slim/hindsight_api/engine/retain/embedding_utils.py

@@ -33,36 +33,6 @@ def _validate_embedding_vector(vector: list[float], *, index: int, expected_dime
     return vector
 
 
-def generate_embedding(
-    embeddings_backend: EmbeddingsBackend, text: str, input_type: EmbeddingInputType = "document"
-) -> list[float]:
-    """
-    Generate embedding for text using the provided embeddings backend.
-
-    Args:
-        embeddings_backend: Embeddings instance to use for encoding
-        text: Text to embed
-        input_type: Whether text is retained document text or recall/search query text.
-
-    Returns:
-        Embedding vector (dimension depends on embeddings backend)
-    """
-    try:
-        embeddings = _encode_with_input_type(embeddings_backend, [text], input_type)
-    except Exception as e:
-        raise Exception(f"Failed to generate embedding: {str(e)}")
-
-    if len(embeddings) != 1:
-        raise RuntimeError(
-            f"Embeddings backend returned {len(embeddings)} vectors for 1 input text; expected exact 1:1 alignment"
-        )
-    return _validate_embedding_vector(
-        embeddings[0],
-        index=0,
-        expected_dimension=embeddings_backend.dimension,
-    )
-
-
 def _encode_with_input_type(
     embeddings_backend: EmbeddingsBackend, texts: list[str], input_type: EmbeddingInputType
 ) -> list[list[float]]:

hindsight-api-slim/hindsight_api/engine/retain/fact_extraction.py

@@ -229,23 +229,6 @@ class ExtractedFact(BaseModel):
     def ensure_entities_list(cls, v):
         return _coerce_entity_strings(v)
 
-    def build_fact_text(self) -> str:
-        """Combine all dimensions into a single comprehensive fact string."""
-        parts = [self.what]
-
-        # Add 'who' if not N/A
-        if self.who and self.who.upper() != "N/A":
-            parts.append(f"Involving: {self.who}")
-
-        # Add 'why' if not N/A
-        if self.why and self.why.upper() != "N/A":
-            parts.append(self.why)
-
-        if len(parts) == 1:
-            return parts[0]
-
-        return " | ".join(parts)
-
 
 class FactExtractionResponse(BaseModel):
     """Response containing all extracted facts (causal relations are embedded in each fact)."""
@@ -1484,6 +1467,8 @@ async def _extract_facts_from_chunk(
                 # Fallback to old format if new fields not present
                 if not what:
                     what = get_value("factual_core")
+                if not what:
+                    what = get_value("text")
                 if not what:
                     # In verbatim mode, 'what' is intentionally absent — text is backfilled from chunk
                     if extraction_mode != "verbatim":
@@ -2246,6 +2231,8 @@ async def extract_facts_from_contents_batch_api(
             what = get_value("what")
             if not what:
                 what = get_value("factual_core")
+            if not what:
+                what = get_value("text")
             if not what:
                 continue
 
@@ -2425,6 +2412,7 @@ async def extract_facts_from_contents_batch_api(
 
     for chunk_meta, chunk_facts in facts_by_chunk:
         content = contents[chunk_meta.content_index]
+        extraction_group_start_idx = global_fact_idx
 
         for fact_from_llm in chunk_facts:
             extracted_fact = ExtractedFactType(
@@ -2433,7 +2421,9 @@ async def extract_facts_from_contents_batch_api(
                 entities=list(fact_from_llm.entities or []),
                 occurred_start=_parse_datetime(fact_from_llm.occurred_start) if fact_from_llm.occurred_start else None,
                 occurred_end=_parse_datetime(fact_from_llm.occurred_end) if fact_from_llm.occurred_end else None,
-                causal_relations=_convert_causal_relations(fact_from_llm.causal_relations or [], global_fact_idx),
+                causal_relations=_convert_causal_relations(
+                    fact_from_llm.causal_relations or [], extraction_group_start_idx, len(chunk_facts)
+                ),
                 content_index=chunk_meta.content_index,
                 chunk_index=chunk_meta.chunk_index,
                 context=content.context,
@@ -2558,8 +2548,7 @@ async def extract_facts_from_contents(
     # Step 1: Create parallel fact extraction tasks
     fact_extraction_tasks = []
     for item in contents:
-        # Call extract_facts_from_text directly (defined earlier in this file)
-        # to avoid circular import with utils.extract_facts
+        # Call extract_facts_from_text directly (defined earlier in this file).
         task = extract_facts_from_text(
             text=item.content,
             event_date=item.event_date,
@@ -2617,40 +2606,37 @@ async def extract_facts_from_contents(
         fact_idx_in_content = 0
         for chunk_idx_in_content, (chunk_text, chunk_fact_count) in enumerate(chunks_from_llm):
             chunk_global_idx = chunk_start_idx + chunk_idx_in_content
+            extraction_group_start_idx = global_fact_idx
+            chunk_facts = facts_from_llm[fact_idx_in_content : fact_idx_in_content + chunk_fact_count]
 
-            for _ in range(chunk_fact_count):
-                if fact_idx_in_content < len(facts_from_llm):
-                    fact_from_llm = facts_from_llm[fact_idx_in_content]
+            for fact_from_llm in chunk_facts:
+                # Convert Fact model from LLM to ExtractedFactType dataclass
+                # mentioned_at is always the event_date (when the conversation/document occurred)
+                extracted_fact = ExtractedFactType(
+                    fact_text=fact_from_llm.fact,
+                    fact_type=fact_from_llm.fact_type,
+                    entities=list(fact_from_llm.entities or []),
+                    # occurred_start/end: from LLM only, leave None if not provided
+                    occurred_start=_parse_datetime(fact_from_llm.occurred_start)
+                    if fact_from_llm.occurred_start
+                    else None,
+                    occurred_end=_parse_datetime(fact_from_llm.occurred_end) if fact_from_llm.occurred_end else None,
+                    causal_relations=_convert_causal_relations(
+                        fact_from_llm.causal_relations or [], extraction_group_start_idx, len(chunk_facts)
+                    ),
+                    content_index=content_index,
+                    chunk_index=chunk_global_idx,
+                    context=content.context,
+                    # mentioned_at: always the event_date (when the conversation/document occurred)
+                    mentioned_at=content.event_date,
+                    metadata=content.metadata,
+                    tags=content.tags,
+                    observation_scopes=content.observation_scopes,
+                )
 
-                    # Convert Fact model from LLM to ExtractedFactType dataclass
-                    # mentioned_at is always the event_date (when the conversation/document occurred)
-                    extracted_fact = ExtractedFactType(
-                        fact_text=fact_from_llm.fact,
-                        fact_type=fact_from_llm.fact_type,
-                        entities=list(fact_from_llm.entities or []),
-                        # occurred_start/end: from LLM only, leave None if not provided
-                        occurred_start=_parse_datetime(fact_from_llm.occurred_start)
-                        if fact_from_llm.occurred_start
-                        else None,
-                        occurred_end=_parse_datetime(fact_from_llm.occurred_end)
-                        if fact_from_llm.occurred_end
-                        else None,
-                        causal_relations=_convert_causal_relations(
-                            fact_from_llm.causal_relations or [], global_fact_idx
-                        ),
-                        content_index=content_index,
-                        chunk_index=chunk_global_idx,
-                        context=content.context,
-                        # mentioned_at: always the event_date (when the conversation/document occurred)
-                        mentioned_at=content.event_date,
-                        metadata=content.metadata,
-                        tags=content.tags,
-                        observation_scopes=content.observation_scopes,
-                    )
-
-                    extracted_facts.append(extracted_fact)
-                    global_fact_idx += 1
-                    fact_idx_in_content += 1
+                extracted_facts.append(extracted_fact)
+                global_fact_idx += 1
+                fact_idx_in_content += 1
 
     # Step 4: For verbatim mode, collapse to one fact per chunk with original text
     if config.retain_extraction_mode == "verbatim":
@@ -2702,7 +2688,9 @@ def _parse_datetime(date_str: str):
         return None
 
 
-def _convert_causal_relations(relations_from_llm, fact_start_idx: int) -> list[CausalRelationType]:
+def _convert_causal_relations(
+    relations_from_llm, extraction_group_start_idx: int, extraction_group_size: int
+) -> list[CausalRelationType]:
     """
     Convert causal relations from LLM format to ExtractedFact format.
 
@@ -2710,9 +2698,16 @@ def _convert_causal_relations(relations_from_llm, fact_start_idx: int) -> list[C
     """
     causal_relations = []
     for rel in relations_from_llm:
+        target_fact_index = rel.target_fact_index
+        if (
+            not isinstance(target_fact_index, int)
+            or isinstance(target_fact_index, bool)
+            or not 0 <= target_fact_index < extraction_group_size
+        ):
+            continue
         causal_relation = CausalRelationType(
             relation_type=rel.relation_type,
-            target_fact_index=fact_start_idx + rel.target_fact_index,
+            target_fact_index=extraction_group_start_idx + target_fact_index,
         )
         causal_relations.append(causal_relation)
     return causal_relations

hindsight-api-slim/hindsight_api/engine/retain/fact_storage.py

@@ -9,7 +9,7 @@ import logging
 import uuid
 from datetime import datetime
 
-from ...config import get_config
+from ...config import _get_raw_config, get_config
 from ..memory_engine import fq_table
 from .bank_utils import DEFAULT_DISPOSITION, create_bank_vector_indexes
 from .fact_extraction import _sanitize_text
@@ -35,6 +35,26 @@ async def get_document_content(
     return row
 
 
+async def count_document_memory_units(
+    conn,
+    bank_id: str,
+    document_id: str,
+) -> int:
+    """Count the memory units a document currently owns.
+
+    This is the number reported as ``memory_unit_count`` by the Documents API and
+    by the ``retain.completed`` webhook. Zero means the document is stored but
+    unreachable through recall/reflect — only memory units carry embeddings, so a
+    document without them cannot be retrieved until it is reprocessed (#3040).
+    """
+    count = await conn.fetchval(
+        f"SELECT COUNT(*) FROM {fq_table('memory_units')} WHERE bank_id = $1 AND document_id = $2",
+        bank_id,
+        document_id,
+    )
+    return int(count or 0)
+
+
 async def insert_facts_batch(
     conn, bank_id: str, facts: list[ProcessedFact], document_id: str | None = None, ops=None
 ) -> list[str]:
@@ -271,6 +291,7 @@ async def handle_document_tracking(
     retain_params: dict | None = None,
     document_tags: list[str] | None = None,
     ops=None,
+    store_document_text: bool | None = None,
 ) -> None:
     """
     Handle document tracking in the database (full-replace mode).
@@ -358,6 +379,7 @@ async def handle_document_tracking(
         retain_params,
         document_tags,
         preserved_created_at=preserved_created_at,
+        store_document_text=store_document_text,
     )
 
 
@@ -368,6 +390,7 @@ async def upsert_document_metadata(
     combined_content: str,
     retain_params: dict | None = None,
     document_tags: list[str] | None = None,
+    store_document_text: bool | None = None,
 ) -> None:
     """
     Update document metadata without deleting existing facts/chunks.
@@ -380,7 +403,16 @@ async def upsert_document_metadata(
     combined_content = _sanitize_text(combined_content) or ""
     content_hash = hashlib.sha256(combined_content.encode()).hexdigest()
 
-    await _upsert_document_row(conn, bank_id, document_id, combined_content, content_hash, retain_params, document_tags)
+    await _upsert_document_row(
+        conn,
+        bank_id,
+        document_id,
+        combined_content,
+        content_hash,
+        retain_params,
+        document_tags,
+        store_document_text=store_document_text,
+    )
 
 
 async def _upsert_document_row(
@@ -392,6 +424,7 @@ async def _upsert_document_row(
     retain_params: dict | None = None,
     document_tags: list[str] | None = None,
     preserved_created_at: datetime | None = None,
+    store_document_text: bool | None = None,
 ) -> None:
     """Insert or update a document row.
 
@@ -403,8 +436,13 @@ async def _upsert_document_row(
     When ``store_document_text`` is disabled, the raw source text
     is dropped and ``original_text`` is stored as NULL. The ``content_hash`` is
     still computed from the real content so delta-retain dedup is unaffected.
+    ``store_document_text`` defaults to the server-level config when ``None``;
+    the retain path passes the per-bank resolved value.
     """
-    original_text = combined_content if get_config().store_document_text else None
+    # Fallback to the raw global default (not get_config(), which guards
+    # bank-configurable fields); the retain path always passes the resolved value.
+    store_text = store_document_text if store_document_text is not None else _get_raw_config().store_document_text
+    original_text = combined_content if store_text else None
     await conn.execute(
         f"""
         INSERT INTO {fq_table("documents")} (id, bank_id, original_text, content_hash, retain_params, tags, created_at, updated_at)

hindsight-api-slim/hindsight_api/engine/retain/link_creation.py

@@ -37,6 +37,7 @@ async def create_semantic_links_batch(
     bank_id: str,
     unit_ids: list[str],
     embeddings: list[list[float]],
+    threshold: float,
     pre_computed_ann_links: list[tuple] | None = None,
     ops=None,
 ) -> int:
@@ -52,6 +53,7 @@ async def create_semantic_links_batch(
         bank_id: Bank identifier
         unit_ids: List of unit IDs to create links for
         embeddings: List of embedding vectors (same length as unit_ids)
+        threshold: Minimum cosine similarity for semantic links
         pre_computed_ann_links: Pre-computed ANN results from Phase 1
 
     Returns:
@@ -64,7 +66,14 @@ async def create_semantic_links_batch(
         raise ValueError(f"Mismatch between unit_ids ({len(unit_ids)}) and embeddings ({len(embeddings)})")
 
     return await link_utils.create_semantic_links_batch(
-        conn, bank_id, unit_ids, embeddings, log_buffer=[], pre_computed_ann_links=pre_computed_ann_links, ops=ops
+        conn,
+        bank_id,
+        unit_ids,
+        embeddings,
+        threshold=threshold,
+        log_buffer=[],
+        pre_computed_ann_links=pre_computed_ann_links,
+        ops=ops,
     )
 
 

hindsight-api-slim/hindsight_api/engine/retain/link_utils.py

@@ -4,10 +4,16 @@ Link creation utilities for temporal, semantic, and entity links.
 
 import logging
 import time
-from datetime import UTC, datetime, timedelta
+from datetime import UTC
 
 from ..._vector_index import ann_search_tuning_settings, configured_vector_extension
-from ..causal_links import CANONICAL_CAUSAL_LINK_TYPES, LEGACY_CAUSAL_LINK_TYPES
+from ..causal_links import (
+    CANONICAL_CAUSAL_LINK_TYPES,
+    CAUSAL_LINK_TYPES,
+    DEFAULT_CAUSAL_LINK_WEIGHT,
+    LEGACY_CAUSAL_LINK_TYPES,
+    CausalLinkDescriptor,
+)
 from ..db.base import DatabaseConnection
 from ..db.ops import DataAccessOps
 from ..memory_engine import fq_table
@@ -113,101 +119,6 @@ def _normalize_datetime(dt):
     return dt
 
 
-def compute_temporal_links(
-    new_units: dict,
-    candidates: list,
-    time_window_hours: int = 24,
-) -> list:
-    """
-    Compute temporal links between new units and candidate neighbors.
-
-    This is a pure function that takes query results and returns link tuples,
-    making it easy to test without database access.
-
-    Args:
-        new_units: Dict mapping unit_id (str) to event_date (datetime)
-        candidates: List of dicts with 'id' and 'event_date' keys (candidate neighbors)
-        time_window_hours: Time window in hours for temporal links
-
-    Returns:
-        List of tuples: (from_unit_id, to_unit_id, 'temporal', weight, None)
-    """
-    if not new_units:
-        return []
-
-    links = []
-    for unit_id, unit_event_date in new_units.items():
-        # Units without event_date can't form temporal links
-        if unit_event_date is None:
-            continue
-        # Normalize unit_event_date for consistent comparison
-        unit_event_date_norm = _normalize_datetime(unit_event_date)
-
-        # Calculate time window bounds with overflow protection
-        try:
-            time_lower = unit_event_date_norm - timedelta(hours=time_window_hours)
-        except OverflowError:
-            time_lower = datetime.min.replace(tzinfo=UTC)
-        try:
-            time_upper = unit_event_date_norm + timedelta(hours=time_window_hours)
-        except OverflowError:
-            time_upper = datetime.max.replace(tzinfo=UTC)
-
-        # Filter candidates within this unit's time window
-        matching_neighbors = [
-            (row["id"], row["event_date"])
-            for row in candidates
-            if time_lower <= _normalize_datetime(row["event_date"]) <= time_upper
-        ][:10]  # Limit to top 10
-
-        for recent_id, recent_event_date in matching_neighbors:
-            # Calculate temporal proximity weight
-            time_diff_hours = abs(
-                (unit_event_date_norm - _normalize_datetime(recent_event_date)).total_seconds() / 3600
-            )
-            weight = max(0.3, 1.0 - (time_diff_hours / time_window_hours))
-            links.append((unit_id, str(recent_id), "temporal", weight, None))
-
-    return _cap_links_per_unit(links)
-
-
-def compute_temporal_query_bounds(
-    new_units: dict,
-    time_window_hours: int = 24,
-) -> tuple:
-    """
-    Compute the min/max date bounds for querying temporal neighbors.
-
-    Args:
-        new_units: Dict mapping unit_id (str) to event_date (datetime)
-        time_window_hours: Time window in hours
-
-    Returns:
-        Tuple of (min_date, max_date) with overflow protection
-    """
-    if not new_units:
-        return None, None
-
-    # Normalize all dates to be timezone-aware to avoid comparison issues
-    # Filter out None values — units without event_date can't form temporal links
-    all_dates = [_normalize_datetime(d) for d in new_units.values() if d is not None]
-
-    if not all_dates:
-        return None, None
-
-    try:
-        min_date = min(all_dates) - timedelta(hours=time_window_hours)
-    except OverflowError:
-        min_date = datetime.min.replace(tzinfo=UTC)
-
-    try:
-        max_date = max(all_dates) + timedelta(hours=time_window_hours)
-    except OverflowError:
-        max_date = datetime.max.replace(tzinfo=UTC)
-
-    return min_date, max_date
-
-
 def _log(log_buffer, message, level="info"):
     """Helper to log to buffer if available, otherwise use logger.
 
@@ -520,7 +431,8 @@ async def compute_semantic_links_ann(
     embeddings: list[list[float]],
     fact_types: list[str] | None = None,
     top_k: int = 50,
-    threshold: float = 0.7,
+    *,
+    threshold: float,
     log_buffer: list[str] = None,
 ) -> list[tuple]:
     """
@@ -661,12 +573,13 @@ def compute_semantic_links_within_batch(
     unit_ids: list[str],
     embeddings: list[list[float]],
     top_k: int = 50,
-    threshold: float = 0.7,
+    *,
+    threshold: float,
 ) -> list[tuple]:
     """
     Compute semantic links between units within the same batch (no DB needed).
 
-    Uses numpy dot product on embeddings already in memory — instant.
+    Uses cosine similarity on embeddings already in memory — instant.
 
     Args:
         unit_ids: Unit IDs (real IDs from insert_facts_batch)
@@ -683,15 +596,25 @@ def compute_semantic_links_within_batch(
     import numpy as np
 
     links = []
-    new_embeddings_matrix = np.array(embeddings)
+    new_embeddings_matrix = np.asarray(embeddings, dtype=float)
+    norms = np.linalg.norm(new_embeddings_matrix, axis=1)
+    valid_embeddings = np.isfinite(new_embeddings_matrix).all(axis=1) & np.isfinite(norms) & (norms > 0)
+    normalized_embeddings = np.zeros_like(new_embeddings_matrix)
+    normalized_embeddings[valid_embeddings] = (
+        new_embeddings_matrix[valid_embeddings] / norms[valid_embeddings, np.newaxis]
+    )
 
     for i, unit_id in enumerate(unit_ids):
+        if not valid_embeddings[i]:
+            continue
+
         other_indices = [j for j in range(len(unit_ids)) if j != i]
         if not other_indices:
             continue
 
-        other_embeddings = new_embeddings_matrix[other_indices]
-        similarities = np.dot(other_embeddings, new_embeddings_matrix[i])
+        other_embeddings = normalized_embeddings[other_indices]
+        similarities = np.dot(other_embeddings, normalized_embeddings[i])
+        similarities[~valid_embeddings[other_indices]] = -np.inf
 
         above_threshold = np.where(similarities >= threshold)[0]
         if len(above_threshold) > 0:
@@ -711,7 +634,8 @@ async def create_semantic_links_batch(
     unit_ids: list[str],
     embeddings: list[list[float]],
     top_k: int = 50,
-    threshold: float = 0.7,
+    *,
+    threshold: float,
     log_buffer: list[str] = None,
     pre_computed_ann_links: list[tuple] | None = None,
     ops=None,
@@ -746,7 +670,12 @@ async def create_semantic_links_batch(
 
         # Within-batch similarities (numpy, no DB)
         batch_start = time_mod.time()
-        within_batch_links = compute_semantic_links_within_batch(unit_ids, embeddings, top_k, threshold)
+        within_batch_links = compute_semantic_links_within_batch(
+            unit_ids,
+            embeddings,
+            top_k,
+            threshold=threshold,
+        )
         all_links.extend(within_batch_links)
         _log(
             log_buffer,
@@ -887,3 +816,108 @@ async def _write_causal_links_batch(
 
         traceback.print_exc()
         raise
+
+
+async def snapshot_causal_links(conn: DatabaseConnection, bank_id: str, unit_id: str) -> list[CausalLinkDescriptor]:
+    """Collect the causal edges that must survive a unit's move to the archive.
+
+    Causal edges are retain-time extraction output: unlike temporal/semantic
+    links they can't be recomputed from dates or embeddings, and nothing
+    rebuilds them (graph maintenance only relinks temporal/semantic, and
+    consolidation regenerates observations, not raw-fact edges). Invalidation
+    removes the live row, so the FK cascade takes every incident edge with it —
+    hence this snapshot, parked on the archive row (#2864).
+
+    The snapshot merges two sources:
+
+    * the unit's currently materialized causal edges, and
+    * descriptors already parked on *archived* peers that name this unit — an
+      edge whose other endpoint was invalidated first is no longer in
+      ``memory_links``, so the peer's snapshot is the only copy left.
+
+    Keeping a copy on every archived endpoint makes revert order irrelevant:
+    whichever endpoint comes back last sees both sides live and rematerializes.
+
+    Returns:
+        The descriptors to store on the archive row (deduplicated across both
+        sources by the UNION).
+    """
+    rows = await conn.fetch(
+        f"""
+        SELECT from_unit_id, to_unit_id, link_type, weight
+        FROM {fq_table("memory_links")}
+        WHERE (from_unit_id = $1 OR to_unit_id = $1)
+          AND bank_id = $2
+          AND link_type = ANY($3::text[])
+        UNION
+        SELECT d.from_unit_id, d.to_unit_id, d.link_type, d.weight
+        FROM {fq_table("invalidated_memory_units")} a
+        CROSS JOIN LATERAL jsonb_to_recordset(a.causal_links)
+            AS d(from_unit_id uuid, to_unit_id uuid, link_type text, weight float8)
+        WHERE a.bank_id = $2
+          AND a.causal_links <> '[]'::jsonb
+          AND (d.from_unit_id = $1 OR d.to_unit_id = $1)
+          -- Same guard as CausalLinkDescriptor.from_json_dict: the column is
+          -- schemaless JSON, and a malformed entry would otherwise be copied
+          -- forward as a NULL-endpoint descriptor.
+          AND d.from_unit_id IS NOT NULL
+          AND d.to_unit_id IS NOT NULL
+          AND d.link_type = ANY($3::text[])
+        """,
+        unit_id,
+        bank_id,
+        list(CAUSAL_LINK_TYPES),
+    )
+    return [
+        CausalLinkDescriptor(
+            from_unit_id=str(row["from_unit_id"]),
+            to_unit_id=str(row["to_unit_id"]),
+            link_type=row["link_type"],
+            weight=float(row["weight"]) if row["weight"] is not None else DEFAULT_CAUSAL_LINK_WEIGHT,
+        )
+        for row in rows
+    ]
+
+
+async def rematerialize_causal_links(
+    conn: DatabaseConnection,
+    bank_id: str,
+    stored_descriptors: list,
+    ops: DataAccessOps | None = None,
+) -> int:
+    """Recreate archived causal edges whose endpoints are both live again.
+
+    Counterpart of :func:`snapshot_causal_links`, called when a fact reverts to
+    ``valid``. Descriptors whose peer is still archived (or was permanently
+    deleted) are silently dropped from this insert: the bulk writer only takes
+    links whose endpoints exist in ``memory_units``. That is the point — a
+    still-archived peer keeps its own copy of the descriptor and materializes
+    the edge when *it* reverts.
+
+    Insertion is ``ON CONFLICT DO NOTHING``, so repeated invalidate/revert
+    cycles never duplicate an edge.
+
+    Args:
+        stored_descriptors: The archive row's ``causal_links`` payload, already
+            decoded from JSON. Entries that don't parse as a causal edge are
+            skipped (see :meth:`CausalLinkDescriptor.from_json_dict`).
+
+    Returns:
+        Number of descriptors submitted (not all of which may materialize).
+    """
+    parsed = [CausalLinkDescriptor.from_json_dict(raw) for raw in stored_descriptors]
+    links = [
+        (
+            descriptor.from_unit_id,
+            descriptor.to_unit_id,
+            descriptor.link_type,
+            descriptor.weight,
+            None,
+        )
+        for descriptor in parsed
+        if descriptor is not None
+    ]
+    if not links:
+        return 0
+    await _bulk_insert_links(conn, links, bank_id=bank_id, ops=ops)
+    return len(links)

hindsight-api-slim/hindsight_api/engine/retain/orchestrator.py

@@ -22,6 +22,7 @@ from ...extensions.memory_defense import (
     apply_redaction,
     parse_policy,
 )
+from ...metrics import get_metrics_collector
 from ...worker.stage import set_stage
 from ..db_utils import acquire_with_retry
 from ..memory_engine import count_tokens, fq_table
@@ -71,6 +72,25 @@ def _redact_document_body(body: str, config: Any) -> str:
     return apply_redaction(body).content
 
 
+def _is_strict_append_of_stored_document(
+    stored_original_text: str | None,
+    document_body_override: str | None,
+    config: Any,
+) -> bool:
+    """Return whether an oversized document body strictly appends stored text.
+
+    ``documents.original_text`` is sanitized and may also be Memory Defense
+    redacted before persistence. Apply those same transformations to the
+    complete incoming body before comparing it with the stored prefix.
+    """
+    if stored_original_text is None or document_body_override is None:
+        return False
+
+    redacted_body = _redact_document_body(document_body_override, config)
+    sanitized_body = fact_extraction._sanitize_text(redacted_body) or ""
+    return len(sanitized_body) > len(stored_original_text) and sanitized_body.startswith(stored_original_text)
+
+
 async def _fire_memory_defense_webhook(
     webhook_manager: Any,
     *,
@@ -275,6 +295,30 @@ class _ProcessedFactBatch:
     retained_index_by_original: list[int | None]
 
 
+async def _record_retain_document_outcome(pool: Any, bank_id: str, document_id: str, units_created: int) -> None:
+    """Emit the per-document retain outcome metric.
+
+    The metric reports the document's unit count *after* this retain, not what
+    this call created: a delta retain that touches one chunk can legitimately
+    create zero units while the document keeps the units of its unchanged chunks,
+    and an oversized document is retained as several sequential sub-batches. Only
+    a document left with zero units is unreachable through recall/reflect and
+    worth alerting on (#3040).
+
+    ``units_created > 0`` already settles the outcome, so the count query only
+    runs on the zero case — which is exactly the cheap path (nothing was written).
+    Best-effort: telemetry must never fail a retain.
+    """
+    try:
+        total = units_created
+        if total == 0:
+            async with acquire_with_retry(pool) as conn:
+                total = await fact_storage.count_document_memory_units(conn, bank_id, document_id)
+        get_metrics_collector().record_retain_document(bank_id=bank_id, memory_unit_count=total)
+    except Exception:
+        logger.debug("Failed to record retain document outcome metric", exc_info=True)
+
+
 def _resolve_narrator(profile_name: str, bank_id: str) -> str | None:
     """Resolve the narrator (memory owner) used to prime fact extraction.
 
@@ -377,7 +421,13 @@ async def _pre_resolve_phase1(
         if not skip_semantic_ann:
             fact_types = [fact.fact_type for fact in processed_facts]
             semantic_ann_links = await compute_semantic_links_ann(
-                resolve_conn, bank_id, placeholder_unit_ids, embeddings, fact_types=fact_types, log_buffer=log_buffer
+                resolve_conn,
+                bank_id,
+                placeholder_unit_ids,
+                embeddings,
+                fact_types=fact_types,
+                threshold=config.semantic_link_min_similarity,
+                log_buffer=log_buffer,
             )
 
     return Phase1Result(
@@ -496,6 +546,7 @@ async def _insert_facts_and_links(
                 bank_id,
                 unit_ids,
                 embeddings_for_links,
+                threshold=config.semantic_link_min_similarity,
                 pre_computed_ann_links=semantic_ann_links,
                 ops=ops,
             )
@@ -1098,6 +1149,8 @@ async def _run_final_semantic_ann(
     pool: Any,
     bank_id: str,
     unit_ids: list[str],
+    *,
+    threshold: float,
     log_buffer: list[str],
 ) -> None:
     """
@@ -1176,6 +1229,7 @@ async def _run_final_semantic_ann(
                     chunk_embs,
                     fact_types=chunk_ftypes,
                     top_k=20,  # Recall uses at most 20 neighbors
+                    threshold=threshold,
                     log_buffer=log_buffer,
                 )
                 if ann_links:
@@ -1384,26 +1438,38 @@ async def _streaming_retain_batch(
 
         tasks: list[asyncio.Task] = []
         skipped_total = 0
-        for i, chunk_text in enumerate(all_pre_chunks):
-            chunk_hash = chunk_storage.compute_chunk_hash(chunk_text)
-            if chunk_hash in existing_chunk_hashes:
-                # Memory: skipped chunks aren't needed either.
-                all_pre_chunks[i] = ""
-                skipped_total += 1
-                continue
-            tasks.append(asyncio.create_task(_extract_one(i, chunk_text)))
+        try:
+            for i, chunk_text in enumerate(all_pre_chunks):
+                chunk_hash = chunk_storage.compute_chunk_hash(chunk_text)
+                if chunk_hash in existing_chunk_hashes:
+                    # Memory: skipped chunks aren't needed either.
+                    all_pre_chunks[i] = ""
+                    skipped_total += 1
+                    continue
+                tasks.append(asyncio.create_task(_extract_one(i, chunk_text)))
 
-        if skipped_total > 0:
-            log_buffer.append(f"[streaming] Producer: skipped {skipped_total}/{total_chunks} already-committed chunks")
+            if skipped_total > 0:
+                log_buffer.append(
+                    f"[streaming] Producer: skipped {skipped_total}/{total_chunks} already-committed chunks"
+                )
 
-        # Wait for all extractions; collect exceptions
-        results = await asyncio.gather(*tasks, return_exceptions=True)
-        for r in results:
-            if isinstance(r, BaseException):
-                producer_error.append(r)
+            # Wait for all extractions; collect exceptions
+            results = await asyncio.gather(*tasks, return_exceptions=True)
+            for r in results:
+                if isinstance(r, BaseException):
+                    producer_error.append(r)
 
-        # Signal the consumer that production is done
-        await chunk_queue.put(None)
+            # Signal the consumer that production is done
+            await chunk_queue.put(None)
+        finally:
+            # Cancellation arriving mid-fan-out (the consumer failed, or the worker's
+            # wall-clock ceiling fired) must not strand extraction tasks. Cancelling
+            # the gather above already propagates to them, but tasks created before
+            # we reach it would otherwise survive and park on `chunk_queue.put()`
+            # for the life of the process.
+            for extraction in tasks:
+                if not extraction.done():
+                    extraction.cancel()
 
     # ---- DB Consumer ----
     # Drains enriched chunks from the queue in batches and runs
@@ -1558,6 +1624,7 @@ async def _streaming_retain_batch(
                                 combined_content,
                                 retain_params,
                                 merged_tags,
+                                store_document_text=getattr(config, "store_document_text", True),
                             )
                         else:
                             await fact_storage.handle_document_tracking(
@@ -1569,6 +1636,7 @@ async def _streaming_retain_batch(
                                 retain_params,
                                 merged_tags,
                                 ops=pool.ops,
+                                store_document_text=getattr(config, "store_document_text", True),
                             )
                         doc_tracking_done[0] = True
                         # Memory: combined_content has been persisted; release
@@ -1660,6 +1728,7 @@ async def _streaming_retain_batch(
                                 combined_content,
                                 retain_params,
                                 merged_tags,
+                                store_document_text=getattr(config, "store_document_text", True),
                             )
                             log_buffer.append(
                                 f"[streaming] Document {effective_doc_id} updated "
@@ -1675,6 +1744,7 @@ async def _streaming_retain_batch(
                                 retain_params,
                                 merged_tags,
                                 ops=pool.ops,
+                                store_document_text=getattr(config, "store_document_text", True),
                             )
                             log_buffer.append(f"[streaming] Document {effective_doc_id} tracked (full content)")
                         doc_tracking_done[0] = True
@@ -1701,7 +1771,12 @@ async def _streaming_retain_batch(
                     chunk_id_map = {}
                     if batch_chunk_meta:
                         chunk_id_map = await chunk_storage.store_chunks_batch(
-                            conn, bank_id, effective_doc_id, batch_chunk_meta, ops=pool.ops
+                            conn,
+                            bank_id,
+                            effective_doc_id,
+                            batch_chunk_meta,
+                            ops=pool.ops,
+                            store_document_text=getattr(config, "store_document_text", True),
                         )
                         log_buffer.append(
                             f"  Store chunks: {len(batch_chunk_meta)} chunks in {time.time() - step_start:.3f}s"
@@ -1743,13 +1818,21 @@ async def _streaming_retain_batch(
                 if is_last and outbox_callback is not None:
                     outbox_fired[0] = True
 
-                # Best-effort: flush entity_cooccurrences and other deferred stats.
-                try:
-                    await entity_resolver.flush_pending_stats()
-                except Exception:
-                    logger.warning(
-                        f"Entity stats flush (consumer batch {consumer_batch_idx + 1}) failed", exc_info=True
-                    )
+            # Best-effort: flush entity_cooccurrences and other deferred stats.
+            #
+            # This MUST run after the `acquire_with_retry` block above has exited,
+            # not inside it: flush_pending_stats() acquires its own connection, and
+            # the write above is only committed when the enclosing acquire() block
+            # exits. On Oracle (oracledb does not autocommit — the backend commits
+            # on clean exit of acquire()) doing this inside the block deadlocks
+            # permanently: connection #2 waits on the row locks the still-open
+            # connection #1 holds on `entities`, while connection #1 cannot commit
+            # until this call returns. Oracle never reports ORA-00060 for it,
+            # because session #1 is blocked in Python rather than on the database.
+            try:
+                await entity_resolver.flush_pending_stats()
+            except Exception:
+                logger.warning(f"Entity stats flush (consumer batch {consumer_batch_idx + 1}) failed", exc_info=True)
 
             logger.info(
                 f"[streaming] Consumer batch {consumer_batch_idx + 1} total "
@@ -1824,8 +1907,27 @@ async def _streaming_retain_batch(
             logger.warning("Failed to check operation recovery state", exc_info=True)
 
     if not facts_already_committed:
-        # Run producer and consumer concurrently
-        await asyncio.gather(_llm_producer(), _db_consumer())
+        # Run producer and consumer concurrently.
+        #
+        # Cancellation is explicit because plain gather() leaks: when the consumer
+        # raises (a deadlock victim, a lock timeout) gather propagates that error
+        # immediately but leaves the producer — and every extraction task under it
+        # — running. Those tasks then block forever on `chunk_queue.put()` into a
+        # queue nobody drains, pinning their chunk payloads and still spending LLM
+        # permits and tokens on an operation that already failed (#3002). The same
+        # applies when the worker's wall-clock ceiling cancels us from above.
+        producer_task = asyncio.create_task(_llm_producer())
+        consumer_task = asyncio.create_task(_db_consumer())
+        try:
+            await asyncio.gather(producer_task, consumer_task)
+        finally:
+            for pipeline_task in (producer_task, consumer_task):
+                if not pipeline_task.done():
+                    pipeline_task.cancel()
+            # Await the cancellations so neither half outlives this call; the
+            # results are already accounted for by the gather above (or by the
+            # exception that is propagating).
+            await asyncio.gather(producer_task, consumer_task, return_exceptions=True)
 
         # Propagate producer errors (e.g. LLM failures)
         if producer_error:
@@ -1858,6 +1960,7 @@ async def _streaming_retain_batch(
                             combined_content,
                             retain_params,
                             merged_tags,
+                            store_document_text=getattr(config, "store_document_text", True),
                         )
                     else:
                         await fact_storage.handle_document_tracking(
@@ -1869,6 +1972,7 @@ async def _streaming_retain_batch(
                             retain_params,
                             merged_tags,
                             ops=pool.ops,
+                            store_document_text=getattr(config, "store_document_text", True),
                         )
                     doc_tracking_done[0] = True
                     # Memory: combined_content has been persisted and won't be
@@ -1944,7 +2048,13 @@ async def _streaming_retain_batch(
     if all_unit_ids and not pipeline_aborted[0]:
         ann_start = time.time()
         try:
-            await _run_final_semantic_ann(pool, bank_id, all_unit_ids, log_buffer)
+            await _run_final_semantic_ann(
+                pool,
+                bank_id,
+                all_unit_ids,
+                threshold=config.semantic_link_min_similarity,
+                log_buffer=log_buffer,
+            )
         except Exception:
             # ANN pass is best-effort. FK violations can occur if a concurrent
             # retain cascade-deleted our units between the batch commit and here.
@@ -1970,6 +2080,9 @@ async def _streaming_retain_batch(
     log_buffer.append(f"{'=' * 60}")
     logger.info("\n" + "\n".join(log_buffer) + "\n")
 
+    if not pipeline_aborted[0]:
+        await _record_retain_document_outcome(pool, bank_id, effective_doc_id, len(all_unit_ids))
+
     # Map all unit_ids back to the original content items.
     # For streaming mode with a single document, all units belong to content 0.
     result_unit_ids = [all_unit_ids] + [[] for _ in contents[1:]]
@@ -2058,12 +2171,26 @@ async def _try_delta_retain(
     # between this read and the write. The write TXN verifies the hash hasn't
     # changed; if it has, we fall back to streaming (which has full protection).
     async with acquire_with_retry(pool) as conn:
+        if document_body_override is not None:
+            doc_row_at_load = await conn.fetchrow(
+                f"SELECT content_hash, original_text FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
+                effective_doc_id,
+                bank_id,
+            )
+            doc_hash_at_load = doc_row_at_load["content_hash"] if doc_row_at_load else None
+            original_text_at_load = doc_row_at_load["original_text"] if doc_row_at_load else None
+        else:
+            doc_hash_at_load = await conn.fetchval(
+                f"SELECT content_hash FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
+                effective_doc_id,
+                bank_id,
+            )
+            original_text_at_load = None
+
+        # Load chunks after the document version. If a concurrent writer commits
+        # between these reads, the hash precondition on metadata-only writes (or
+        # the extraction freshness recheck below) forces a streaming fallback.
         existing_chunks = await chunk_storage.load_existing_chunks(conn, bank_id, effective_doc_id)
-        doc_hash_at_load = await conn.fetchval(
-            f"SELECT content_hash FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
-            effective_doc_id,
-            bank_id,
-        )
 
     if not existing_chunks:
         return None
@@ -2095,6 +2222,30 @@ async def _try_delta_retain(
     )
 
     if not unchanged_indices:
+        if _is_strict_append_of_stored_document(
+            original_text_at_load,
+            document_body_override,
+            config,
+        ):
+            log_buffer.append(
+                "[delta] First oversized slice has no stored chunk match, but "
+                "the complete document strictly appends the stored source — "
+                "preserving historical chunks and advancing document metadata"
+            )
+            return await _delta_metadata_only(
+                pool,
+                bank_id,
+                contents_dicts,
+                contents,
+                effective_doc_id,
+                document_tags,
+                log_buffer,
+                start_time,
+                outbox_callback,
+                document_body_override=document_body_override,
+                config=config,
+                expected_content_hash=doc_hash_at_load,
+            )
         logger.info(f"Delta retain: no unchanged chunks for {effective_doc_id}, falling back to full retain")
         return None
 
@@ -2115,6 +2266,7 @@ async def _try_delta_retain(
             outbox_callback,
             document_body_override=document_body_override,
             config=config,
+            expected_content_hash=doc_hash_at_load,
         )
 
     # Build content items for only the changed/new chunks
@@ -2133,6 +2285,7 @@ async def _try_delta_retain(
             outbox_callback,
             document_body_override=document_body_override,
             config=config,
+            expected_content_hash=doc_hash_at_load,
         )
 
     # Freshness recheck BEFORE the (expensive) LLM extraction.
@@ -2185,6 +2338,7 @@ async def _try_delta_retain(
                 outbox_callback,
                 document_body_override=document_body_override,
                 config=config,
+                expected_content_hash=recheck_hash,
             )
         log_buffer.append(
             f"[delta] Recheck: {len(recheck.changed) + len(recheck.new) + len(recheck.removed)} chunks still differ — "
@@ -2314,7 +2468,12 @@ async def _try_delta_retain(
                         for cm in new_chunk_metadata
                     ]
                     chunk_id_map = await chunk_storage.store_chunks_batch(
-                        conn, bank_id, effective_doc_id, remapped_chunks, ops=pool.ops
+                        conn,
+                        bank_id,
+                        effective_doc_id,
+                        remapped_chunks,
+                        ops=pool.ops,
+                        store_document_text=getattr(config, "store_document_text", True),
                     )
                     for chunk_idx, chunk_id in chunk_id_map.items():
                         chunk_id_map_by_doc[(effective_doc_id, chunk_idx)] = chunk_id
@@ -2351,12 +2510,6 @@ async def _try_delta_retain(
                     ops=pool.ops,
                 )
 
-            # Flush deferred entity_cooccurrences stats (post-transaction, best-effort).
-            try:
-                await entity_resolver.flush_pending_stats()
-            except Exception:
-                logger.warning("Entity stats flush failed — retrieval unaffected", exc_info=True)
-
             total_time = time.time() - start_time
             log_buffer.append(f"{'=' * 60}")
             log_buffer.append(
@@ -2367,11 +2520,20 @@ async def _try_delta_retain(
             log_buffer.append(f"{'=' * 60}")
             logger.info("\n" + "\n".join(log_buffer) + "\n")
 
+        # Flush deferred entity_cooccurrences stats (best-effort). Must run after
+        # the acquire() block above has exited — see the streaming path for why
+        # doing this while still holding the connection deadlocks on Oracle.
+        try:
+            await entity_resolver.flush_pending_stats()
+        except Exception:
+            logger.warning("Entity stats flush failed — retrieval unaffected", exc_info=True)
+
     if db_semaphore is not None:
         async with db_semaphore:
             await _run_delta_db_work()
     else:
         await _run_delta_db_work()
+    await _record_retain_document_outcome(pool, bank_id, effective_doc_id, sum(len(ids) for ids in result_unit_ids))
     # Count content + context tokens that actually went through extraction.
     # ``delta_contents`` holds the per-chunk RetainContent items for the
     # changed/new chunks (see ``_build_delta_contents``) — i.e. exactly what
@@ -2393,16 +2555,22 @@ async def _delta_metadata_only(
     *,
     document_body_override: str | None = None,
     config: Any = None,
-):
+    expected_content_hash: str | None = None,
+) -> tuple[list[list[str]], TokenUsage, int] | None:
     """Handle the case where no chunks changed — just update document metadata and tags."""
     async with acquire_with_retry(pool) as conn:
         async with conn.transaction():
             # Lock the document row to serialize with concurrent retains
-            await conn.fetchval(
+            current_content_hash = await conn.fetchval(
                 f"SELECT content_hash FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2 FOR UPDATE",
                 document_id,
                 bank_id,
             )
+            if expected_content_hash is not None and current_content_hash != expected_content_hash:
+                log_buffer.append(
+                    f"[delta] Document {document_id} changed before metadata update — falling back to full retain"
+                )
+                return None
             # When this sub-batch is a slice of an oversized item, write the
             # full original body (issue #1838) instead of just the slice.
             # Redact the override since it bypassed per-chunk screening.

hindsight-api-slim/hindsight_api/engine/retain/types.py

@@ -185,11 +185,6 @@ class ProcessedFact:
     # Observation scopes for consolidation
     observation_scopes: Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]] | None = None
 
-    @property
-    def is_duplicate(self) -> bool:
-        """Check if this fact was marked as a duplicate."""
-        return self.unit_id is None
-
     @staticmethod
     def _is_degenerate_text(text: str) -> bool:
         """Check if fact text has zero information content.
@@ -346,11 +341,3 @@ class RetainBatch:
 
     # Results (populated after storage)
     unit_ids_by_content: list[list[str]] = field(default_factory=list)
-
-    def get_facts_for_content(self, content_index: int) -> list[ExtractedFact]:
-        """Get all extracted facts for a specific content item."""
-        return [f for f in self.extracted_facts if f.content_index == content_index]
-
-    def get_chunks_for_content(self, content_index: int) -> list[ChunkMetadata]:
-        """Get all chunks for a specific content item."""
-        return [c for c in self.chunks if c.content_index == content_index]

hindsight-api-slim/hindsight_api/engine/search/graph_retrieval.py

@@ -46,6 +46,7 @@ class GraphRetriever(ABC):
         tag_groups: list[TagGroup] | None = None,  # Compound boolean tag filter groups
         created_after: datetime | None = None,  # Only include memory_units created after this time
         created_before: datetime | None = None,  # Only include memory_units created before this time
+        preselected_semantic_seeds: list[RetrievalResult] | None = None,
     ) -> tuple[list[RetrievalResult], GraphRetrievalTimings | None]:
         """
         Retrieve relevant facts via graph traversal.
@@ -59,6 +60,7 @@ class GraphRetriever(ABC):
             query_text: Original query text (optional, for some strategies)
             adjacency: Pre-loaded typed adjacency graph (optional)
             tags: Optional list of tags for visibility filtering (OR matching)
+            preselected_semantic_seeds: Independently thresholded graph entry points already fetched by the caller
 
         Returns:
             Tuple of (List of RetrievalResult with activation scores, optional timing info)

hindsight-api-slim/hindsight_api/engine/search/link_expansion_retrieval.py

@@ -9,8 +9,8 @@ first-class signals stored in memory_links:
                    COUNT(DISTINCT entity_id). Uses a LATERAL per-entity cap
                    (graph_per_entity_limit, default 200) to prevent high-fanout entities
                    from exploding the self-join intermediate rows.
-2. Semantic links — precomputed kNN graph (each new fact linked to its top-5 most
-                    similar existing facts at insert time, similarity >= 0.7). Checked
+2. Semantic links — precomputed kNN graph (each new fact linked to its most
+                    similar existing facts at insert time, subject to the configured threshold). Checked
                     in both directions since the graph is not symmetric. Score = weight.
 3. Causal links  — explicit causal chains (causes/caused_by/enables/prevents).
                    Score = weight + 1.0 (boosted as highest-quality signal).
@@ -31,7 +31,7 @@ import time
 from datetime import datetime
 from typing import Any
 
-from ...config import get_config
+from ...config import DEFAULT_GRAPH_SEED_MIN_SIMILARITY, get_config
 from ..db_utils import acquire_with_retry
 from ..memory_engine import fq_table
 from .graph_retrieval import GraphRetriever
@@ -40,6 +40,8 @@ from .types import GraphRetrievalTimings, RetrievalResult
 
 logger = logging.getLogger(__name__)
 
+GRAPH_SEED_LIMIT = 20
+
 
 async def _find_semantic_seeds(
     conn,
@@ -47,7 +49,7 @@ async def _find_semantic_seeds(
     bank_id: str,
     fact_type: str,
     limit: int = 20,
-    threshold: float = 0.3,
+    threshold: float = DEFAULT_GRAPH_SEED_MIN_SIMILARITY,
     tags: list[str] | None = None,
     tags_match: TagsMatch = "any",
     tag_groups: list[TagGroup] | None = None,
@@ -133,6 +135,7 @@ class LinkExpansionRetriever(GraphRetriever):
         tag_groups: list[TagGroup] | None = None,
         created_after: "datetime | None" = None,
         created_before: "datetime | None" = None,
+        preselected_semantic_seeds: list[RetrievalResult] | None = None,
     ) -> tuple[list[RetrievalResult], GraphRetrievalTimings | None]:
         """
         Retrieve facts by expanding links from seeds.
@@ -146,6 +149,7 @@ class LinkExpansionRetriever(GraphRetriever):
             query_text: Original query text (unused)
             adjacency: Unused, kept for interface compatibility
             tags: Optional list of tags for visibility filtering
+            preselected_semantic_seeds: Graph-specific entry points derived from a shared semantic candidate pool
 
         Returns:
             Tuple of (results, timings)
@@ -154,27 +158,34 @@ class LinkExpansionRetriever(GraphRetriever):
         timings = GraphRetrievalTimings(fact_type=fact_type)
 
         async with acquire_with_retry(pool) as conn:
-            # Graph traversal deliberately chooses its own bounded seeds. The semantic and temporal
-            # retrieval arms have independent candidate limits and thresholds, so reusing their
-            # results would silently change graph-retrieval recall behavior.
-            seeds_start = time.time()
-            all_seeds = await _find_semantic_seeds(
-                conn,
-                query_embedding_str,
-                bank_id,
-                fact_type,
-                limit=20,
-                threshold=0.3,
-                tags=tags,
-                tags_match=tags_match,
-                tag_groups=tag_groups,
-                created_after=created_after,
-                created_before=created_before,
-            )
-            timings.seeds_time = time.time() - seeds_start
+            if preselected_semantic_seeds is None:
+                # A shared semantic pool is reusable only when its SQL threshold
+                # covers the independently configured graph threshold. Otherwise
+                # retain graph retrieval's own query and result semantics.
+                seeds_start = time.time()
+                all_seeds = await _find_semantic_seeds(
+                    conn,
+                    query_embedding_str,
+                    bank_id,
+                    fact_type,
+                    limit=GRAPH_SEED_LIMIT,
+                    threshold=get_config().graph_seed_min_similarity,
+                    tags=tags,
+                    tags_match=tags_match,
+                    tag_groups=tag_groups,
+                    created_after=created_after,
+                    created_before=created_before,
+                )
+                timings.seeds_time = time.time() - seeds_start
+            else:
+                all_seeds = preselected_semantic_seeds
+
             logger.debug(
-                f"[LinkExpansion] Found {len(all_seeds)} semantic seeds for fact_type={fact_type} "
-                f"(tags={tags}, tags_match={tags_match})"
+                "LinkExpansion found %s semantic seeds for fact_type=%s (tags=%s, tags_match=%s)",
+                len(all_seeds),
+                fact_type,
+                tags,
+                tags_match,
             )
 
             if not all_seeds:
@@ -203,7 +214,7 @@ class LinkExpansionRetriever(GraphRetriever):
         #
         # Entity score: tanh(count × 0.5) maps shared-entity count to [0, 1]:
         #   1 entity → 0.46,  2 → 0.76,  3 → 0.91,  4 → 0.96  (saturates naturally)
-        # Semantic score: similarity weight, already ∈ [0.7, 1.0].
+        # Semantic score: similarity weight, already above the configured construction floor.
         # Causal score:   link weight, already ∈ [0, 1].
         #
         # Facts appearing in multiple signals accumulate higher scores, rewarding

hindsight-api-slim/hindsight_api/engine/search/retrieval.py

@@ -15,12 +15,12 @@ from dataclasses import dataclass, field
 from datetime import UTC, datetime
 from typing import TYPE_CHECKING, Any, Optional
 
-from ...config import DEFAULT_BM25_MAX_QUERY_TERMS, get_config
+from ...config import DEFAULT_BM25_MAX_QUERY_TERMS, DEFAULT_TEMPORAL_SEMANTIC_MIN_SIMILARITY, get_config
 from ..db_utils import acquire_with_retry
 from ..memory_engine import fq_table
 from ..sql import create_sql_dialect
 from .graph_retrieval import GraphRetriever
-from .link_expansion_retrieval import LinkExpansionRetriever
+from .link_expansion_retrieval import GRAPH_SEED_LIMIT, LinkExpansionRetriever
 from .tags import TagGroup, TagsMatch, build_tag_groups_where_clause, build_tags_where_clause_simple
 from .types import GraphRetrievalTimings, RetrievalResult
 
@@ -67,6 +67,15 @@ class MultiFactTypeRetrievalResult:
     max_conn_wait: float = 0.0
 
 
+@dataclass
+class SemanticBm25Result:
+    """Per-fact-type candidates returned by the shared semantic/BM25 query."""
+
+    semantic: list[RetrievalResult]
+    bm25: list[RetrievalResult]
+    graph_seeds: list[RetrievalResult] | None
+
+
 # Default graph retriever instance (can be overridden)
 _default_graph_retriever: GraphRetriever | None = None
 
@@ -106,7 +115,8 @@ async def retrieve_semantic_bm25_combined(
     created_before: datetime | None = None,
     min_semantic: float | None = None,
     min_keyword: float | None = None,
-) -> dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]]:
+    graph_seed_min_similarity: float | None = None,
+) -> dict[str, SemanticBm25Result]:
     """
     Combined semantic + BM25 retrieval for multiple fact types in a single query.
 
@@ -138,9 +148,10 @@ async def retrieve_semantic_bm25_combined(
         tags_match: Tag matching mode
 
     Returns:
-        Dict mapping fact_type -> (semantic_results, bm25_results)
+        Candidate groups for each fact type. ``graph_seeds`` is ``None`` when
+        the semantic query's threshold is too strict to cover graph entry points.
     """
-    result_dict: dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]] = {ft: ([], []) for ft in fact_types}
+    result_dict = {ft: SemanticBm25Result(semantic=[], bm25=[], graph_seeds=None) for ft in fact_types}
 
     config = get_config()
     tokens = tokenize_query(query_text)
@@ -307,8 +318,18 @@ async def retrieve_semantic_bm25_combined(
         else:
             raise
 
-    # Group results; trim semantic to limit (over-fetched for HNSW approximation).
-    sem_counts: dict[str, int] = {ft: 0 for ft in fact_types}
+    # Group results. The semantic SQL deliberately over-fetches for HNSW recall;
+    # when that pool also covers the graph threshold, derive graph entry points
+    # from the same ordered rows instead of issuing one duplicate ANN query per
+    # fact type. Convert only the prefix either consumer can observe, not the
+    # entire HNSW over-fetch pool.
+    graph_seed_threshold = (
+        graph_seed_min_similarity
+        if graph_seed_min_similarity is not None and sem_min <= graph_seed_min_similarity
+        else None
+    )
+    semantic_candidate_limit = max(limit, GRAPH_SEED_LIMIT if graph_seed_threshold is not None else 0)
+    semantic_candidates: dict[str, list[RetrievalResult]] = {ft: [] for ft in fact_types}
     for r in rows:
         row = dict(r)
         source = row.pop("source")
@@ -316,11 +337,19 @@ async def retrieve_semantic_bm25_combined(
         if ft not in result_dict:
             continue
         if source == "semantic":
-            if sem_counts[ft] < limit:
-                result_dict[ft][0].append(RetrievalResult.from_db_row(row))
-                sem_counts[ft] += 1
+            if len(semantic_candidates[ft]) < semantic_candidate_limit:
+                semantic_candidates[ft].append(RetrievalResult.from_db_row(row))
         else:
-            result_dict[ft][1].append(RetrievalResult.from_db_row(row))
+            result_dict[ft].bm25.append(RetrievalResult.from_db_row(row))
+
+    for ft, candidates in semantic_candidates.items():
+        result_dict[ft].semantic.extend(candidates[:limit])
+        if graph_seed_threshold is not None:
+            result_dict[ft].graph_seeds = [
+                candidate
+                for candidate in candidates
+                if candidate.similarity is not None and candidate.similarity >= graph_seed_threshold
+            ][:GRAPH_SEED_LIMIT]
 
     return result_dict
 
@@ -393,7 +422,7 @@ async def retrieve_temporal_combined(
     start_date: datetime,
     end_date: datetime,
     budget: int,
-    semantic_threshold: float = 0.1,
+    semantic_threshold: float = DEFAULT_TEMPORAL_SEMANTIC_MIN_SIMILARITY,
     tags: list[str] | None = None,
     tags_match: TagsMatch = "any",
     tag_groups: list[TagGroup] | None = None,
@@ -747,6 +776,7 @@ async def retrieve_all_fact_types_parallel(
     import time
 
     retriever = graph_retriever or get_default_graph_retriever()
+    config = get_config()
     start_time = time.time()
     timings: dict[str, float] = {}
 
@@ -783,6 +813,7 @@ async def retrieve_all_fact_types_parallel(
             created_before=created_before,
             min_semantic=min_semantic,
             min_keyword=min_keyword,
+            graph_seed_min_similarity=config.graph_seed_min_similarity,
         )
         semantic_bm25_time = time.time() - semantic_bm25_start
 
@@ -798,7 +829,7 @@ async def retrieve_all_fact_types_parallel(
                 tc_start,
                 tc_end,
                 budget=thinking_budget,
-                semantic_threshold=0.1,
+                semantic_threshold=config.temporal_semantic_min_similarity,
                 tags=tags,
                 tags_match=tags_match,
                 tag_groups=tag_groups,
@@ -827,6 +858,7 @@ async def retrieve_all_fact_types_parallel(
             tag_groups=tag_groups,
             created_after=created_after,
             created_before=created_before,
+            preselected_semantic_seeds=semantic_bm25_results[ft].graph_seeds,
         )
         return ft, results, time.time() - graph_start, graph_timing
 
@@ -841,7 +873,8 @@ async def retrieve_all_fact_types_parallel(
 
     for ft in fact_types:
         # Get semantic + bm25 results for this fact type
-        semantic_results, bm25_results = semantic_bm25_results.get(ft, ([], []))
+        semantic_results = semantic_bm25_results[ft].semantic
+        bm25_results = semantic_bm25_results[ft].bm25
 
         # Find graph results for this fact type
         graph_results = []

hindsight-api-slim/hindsight_api/engine/search/think_utils.py

@@ -77,31 +77,6 @@ def format_facts_for_prompt(facts: list[MemoryFact]) -> str:
     return json.dumps(formatted, indent=2, ensure_ascii=False)
 
 
-def format_entity_summaries_for_prompt(entities: dict) -> str:
-    """Format entity summaries for inclusion in the reflect prompt.
-
-    Args:
-        entities: Dict mapping entity name to EntityState objects
-
-    Returns:
-        Formatted string with entity summaries, or empty string if no summaries
-    """
-    if not entities:
-        return ""
-
-    summaries = []
-    for name, state in entities.items():
-        # Get summary from observations (summary is stored as single observation)
-        if state.observations:
-            summary_text = state.observations[0].text
-            summaries.append(f"## {name}\n{summary_text}")
-
-    if not summaries:
-        return ""
-
-    return "\n\n".join(summaries)
-
-
 def build_think_prompt(
     agent_facts_text: str,
     world_facts_text: str,

hindsight-api-slim/hindsight_api/engine/search/trace.py

@@ -230,25 +230,3 @@ class SearchTrace(BaseModel):
             if visit.node_id == node_id:
                 return visit
         return None
-
-    def get_search_path_to_node(self, node_id: str) -> list[NodeVisit]:
-        """Get the path from entry point to a specific node."""
-        path = []
-        current_visit = self.get_visit_by_node_id(node_id)
-
-        while current_visit:
-            path.insert(0, current_visit)
-            if current_visit.parent_node_id:
-                current_visit = self.get_visit_by_node_id(current_visit.parent_node_id)
-            else:
-                break
-
-        return path
-
-    def get_nodes_by_link_type(self, link_type: Literal["temporal", "semantic", "entity"]) -> list[NodeVisit]:
-        """Get all nodes reached via a specific link type."""
-        return [v for v in self.visits if v.link_type == link_type]
-
-    def get_entry_point_nodes(self) -> list[NodeVisit]:
-        """Get all entry point visits."""
-        return [v for v in self.visits if v.is_entry_point]

hindsight-api-slim/hindsight_api/engine/search/tracer.py

@@ -11,7 +11,6 @@ from typing import Any, Literal
 
 from .trace import (
     EntryPoint,
-    LinkInfo,
     NodeVisit,
     PruningDecision,
     QueryInfo,
@@ -211,56 +210,6 @@ class SearchTracer:
         elif link_type == "entity":
             self.entity_links_followed += 1
 
-    def add_neighbor_link(
-        self,
-        from_node_id: str,
-        to_node_id: str,
-        link_type: Literal["temporal", "semantic", "entity"],
-        link_weight: float,
-        entity_id: str | None,
-        new_activation: float | None,
-        followed: bool,
-        prune_reason: str | None = None,
-        is_supplementary: bool = False,
-    ):
-        """
-        Record a link to a neighbor (whether followed or not).
-
-        Args:
-            from_node_id: Source node
-            to_node_id: Target node
-            link_type: Type of link
-            link_weight: Weight of link
-            entity_id: Entity ID if link is entity-based
-            new_activation: Activation passed to neighbor (None for supplementary links)
-            followed: Whether link was followed
-            prune_reason: Why link was not followed (if not followed)
-            is_supplementary: Whether this is a supplementary link (multiple connections)
-        """
-        # Find the visit for the source node
-        visit = None
-        for v in self.visits:
-            if v.node_id == from_node_id:
-                visit = v
-                break
-
-        if visit is None:
-            # Node not found, skip
-            return
-
-        link_info = LinkInfo(
-            to_node_id=to_node_id,
-            link_type=link_type,
-            link_weight=link_weight,
-            entity_id=entity_id,
-            new_activation=new_activation,
-            followed=followed,
-            prune_reason=prune_reason,
-            is_supplementary=is_supplementary,
-        )
-
-        visit.neighbors_explored.append(link_info)
-
     def prune_node(
         self,
         node_id: str,

hindsight-api-slim/hindsight_api/engine/tei_retry.py

@@ -0,0 +1,73 @@
+"""Shared retry timing for Text Embeddings Inference HTTP clients."""
+
+import math
+import random
+from datetime import datetime, timezone
+from email.utils import parsedate_to_datetime
+
+import httpx
+
+# Ceiling for a single backoff sleep, independent of the client's request
+# timeout. The reranker holds its concurrency semaphore across the sleep, so a
+# large server-supplied Retry-After would stall every queued rerank behind it;
+# capping well below the request timeout keeps a slow proxy from converting a
+# transient overload into a minutes-long recall stall.
+MAX_RETRY_DELAY_SECONDS = 5.0
+
+# Fraction of the delay used as the jitter window, matching the "equal jitter"
+# policy of ``db_utils._backoff_delay``. TEI overload is self-synchronising -- a
+# burst of concurrent requests exhausts the permit pool at the same instant and
+# gets 429ed together -- so the window has to be wide enough to actually break
+# the burst apart. A token few percent would leave the callers retrying in
+# lockstep and re-colliding on the same exhausted pool.
+JITTER_RATIO = 0.5
+
+
+def _retry_after_seconds(value: str | None) -> float | None:
+    if not isinstance(value, str) or not value:
+        return None
+    try:
+        seconds = float(value)
+    except (TypeError, ValueError, OverflowError):
+        try:
+            retry_at = parsedate_to_datetime(value)
+            if retry_at.tzinfo is None:
+                retry_at = retry_at.replace(tzinfo=timezone.utc)
+            seconds = (retry_at - datetime.now(timezone.utc)).total_seconds()
+        except (IndexError, TypeError, ValueError, OverflowError):
+            return None
+    return max(0.0, seconds) if math.isfinite(seconds) else None
+
+
+def _delay_limit(request_timeout: float) -> float:
+    """Longest sleep we are willing to take between attempts."""
+    if not math.isfinite(request_timeout) or request_timeout < 0:
+        return MAX_RETRY_DELAY_SECONDS
+    return min(request_timeout, MAX_RETRY_DELAY_SECONDS)
+
+
+def tei_retry_delay(
+    response: httpx.Response,
+    fallback_delay: float,
+    *,
+    request_timeout: float,
+) -> float:
+    """Seconds to sleep before retrying a transient TEI response.
+
+    A server-supplied ``Retry-After`` wins over the caller's exponential
+    backoff, and both are bounded by :func:`_delay_limit`. The result is spread
+    over a jitter window so concurrent callers do not resume in lockstep.
+    """
+    retry_after = _retry_after_seconds(response.headers.get("Retry-After"))
+    limit = _delay_limit(request_timeout)
+    fallback = fallback_delay if math.isfinite(fallback_delay) else 0.0
+    requested_delay = max(fallback, retry_after or 0.0, 0.0)
+    delay = min(requested_delay, limit)
+    if delay <= 0:
+        return 0.0
+    spread = delay * JITTER_RATIO
+    if requested_delay >= limit:
+        # Already at the ceiling, so the only room to de-synchronise is downward.
+        return delay - random.uniform(0.0, spread)
+    # Retry-After is a minimum, so spread upward -- but never past the ceiling.
+    return delay + random.uniform(0.0, min(spread, limit - delay))

hindsight-api-slim/hindsight_api/engine/temporal_periods.py

@@ -51,11 +51,15 @@ def _month_end(year: int, month: int) -> datetime:
 
 
 def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRange | None:
-    if re.search(r"\b(yesterday|ayer|ieri|hier|gestern)\b", query, re.IGNORECASE):
+    if re.search(r"\b(yesterday|ayer|ieri|hier|gestern|вчера)\b", query, re.IGNORECASE):
         d = reference_date - timedelta(days=1)
         return _constraint(d, d)
 
-    if re.search(r"\b(today|hoy|oggi|aujourd\'?hui|heute)\b", query, re.IGNORECASE):
+    if re.search(r"\b(позавчера)\b", query, re.IGNORECASE):
+        d = reference_date - timedelta(days=2)
+        return _constraint(d, d)
+
+    if re.search(r"\b(today|hoy|oggi|aujourd\'?hui|heute|сегодня)\b", query, re.IGNORECASE):
         return _constraint(reference_date, reference_date)
 
     if re.search(r"\b(a\s+)?couple\s+(of\s+)?days?\s+ago\b", query, re.IGNORECASE):
@@ -64,20 +68,39 @@ def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRan
     if re.search(r"\b(a\s+)?few\s+days?\s+ago\b", query, re.IGNORECASE):
         return _constraint(reference_date - timedelta(days=5), reference_date - timedelta(days=2))
 
+    if re.search(r"\b(пару|пар[ыу]?)\s+дн(?:ей|я)\s+назад\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=3), reference_date - timedelta(days=1))
+
+    if re.search(r"\bнесколько\s+дней\s+назад\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=5), reference_date - timedelta(days=2))
+
     if re.search(r"\b(a\s+)?couple\s+(of\s+)?weeks?\s+ago\b", query, re.IGNORECASE):
         return _constraint(reference_date - timedelta(weeks=3), reference_date - timedelta(weeks=1))
 
     if re.search(r"\b(a\s+)?few\s+weeks?\s+ago\b", query, re.IGNORECASE):
         return _constraint(reference_date - timedelta(weeks=5), reference_date - timedelta(weeks=2))
 
+    if re.search(r"\b(пару|пар[ыу]?)\s+недель\s+назад\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(weeks=3), reference_date - timedelta(weeks=1))
+
+    if re.search(r"\bнесколько\s+недель\s+назад\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(weeks=5), reference_date - timedelta(weeks=2))
+
     if re.search(r"\b(a\s+)?couple\s+(of\s+)?months?\s+ago\b", query, re.IGNORECASE):
         return _constraint(reference_date - timedelta(days=90), reference_date - timedelta(days=30))
 
     if re.search(r"\b(a\s+)?few\s+months?\s+ago\b", query, re.IGNORECASE):
         return _constraint(reference_date - timedelta(days=150), reference_date - timedelta(days=60))
 
+    if re.search(r"\b(пару|пар[ыу]?)\s+месяцев\s+назад\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=90), reference_date - timedelta(days=30))
+
+    if re.search(r"\bнесколько\s+месяцев\s+назад\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=150), reference_date - timedelta(days=60))
+
     if re.search(
-        r"\b(last\s+week|la\s+semana\s+pasada|la\s+settimana\s+scorsa|la\s+semaine\s+derni[eè]re|letzte\s+woche)\b",
+        r"\b(last\s+week|la\s+semana\s+pasada|la\s+settimana\s+scorsa|la\s+semaine\s+derni[eè]re|letzte\s+woche"
+        r"|(?:на\s+)?прошлой\s+неделе|прошлая\s+неделя)\b",
         query,
         re.IGNORECASE,
     ):
@@ -85,7 +108,8 @@ def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRan
         return _constraint(start, start + timedelta(days=6))
 
     if re.search(
-        r"\b(last\s+month|el\s+mes\s+pasado|il\s+mese\s+scorso|le\s+mois\s+dernier|letzten?\s+monat)\b",
+        r"\b(last\s+month|el\s+mes\s+pasado|il\s+mese\s+scorso|le\s+mois\s+dernier|letzten?\s+monat"
+        r"|(?:в\s+)?прошлом\s+месяце|прошлый\s+месяц)\b",
         query,
         re.IGNORECASE,
     ):
@@ -95,7 +119,8 @@ def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRan
         return _constraint(start, end)
 
     if re.search(
-        r"\b(last\s+year|el\s+a[ñn]o\s+pasado|l\'anno\s+scorso|l\'ann[ée]e\s+derni[eè]re|letztes?\s+jahr)\b",
+        r"\b(last\s+year|el\s+a[ñn]o\s+pasado|l\'anno\s+scorso|l\'ann[ée]e\s+derni[eè]re|letztes?\s+jahr"
+        r"|(?:в\s+)?прошлом\s+году|прошлый\s+год)\b",
         query,
         re.IGNORECASE,
     ):
@@ -103,7 +128,8 @@ def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRan
         return _constraint(datetime(year, 1, 1), datetime(year, 12, 31))
 
     if re.search(
-        r"\b(last\s+weekend|el\s+fin\s+de\s+semana\s+pasado|lo\s+scorso\s+fine\s+settimana|le\s+week-?end\s+dernier|letztes?\s+wochenende)\b",
+        r"\b(last\s+weekend|el\s+fin\s+de\s+semana\s+pasado|lo\s+scorso\s+fine\s+settimana|le\s+week-?end\s+dernier"
+        r"|letztes?\s+wochenende|(?:на\s+|в\s+)?прошлых?\s+выходных|прошлые\s+выходные)\b",
         query,
         re.IGNORECASE,
     ):
@@ -114,18 +140,18 @@ def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRan
         return _constraint(sat, sat + timedelta(days=1))
 
     month_patterns = {
-        "january|enero|gennaio|janvier|januar": 1,
-        "february|febrero|febbraio|f[ée]vrier|februar": 2,
-        "march|marzo|mars|m[äa]rz": 3,
-        "april|abril|aprile|avril": 4,
-        "may|mayo|maggio|mai": 5,
-        "june|junio|giugno|juin|juni": 6,
-        "july|julio|luglio|juillet|juli": 7,
-        "august|agosto|ao[uû]t": 8,
-        "september|septiembre|settembre|septembre": 9,
-        "october|octubre|ottobre|octobre|oktober": 10,
-        "november|noviembre|novembre": 11,
-        "december|diciembre|dicembre|d[ée]cembre|dezember": 12,
+        "january|enero|gennaio|janvier|januar|январ[ьяе]": 1,
+        "february|febrero|febbraio|f[ée]vrier|februar|феврал[ьяе]": 2,
+        "march|marzo|mars|m[äa]rz|март[ае]?": 3,
+        "april|abril|aprile|avril|апрел[ьяе]": 4,
+        "may|mayo|maggio|mai|ма[йяе]": 5,
+        "june|junio|giugno|juin|juni|июн[ьяе]": 6,
+        "july|julio|luglio|juillet|juli|июл[ьяе]": 7,
+        "august|agosto|ao[uû]t|август[ае]?": 8,
+        "september|septiembre|settembre|septembre|сентябр[ьяе]": 9,
+        "october|octubre|ottobre|octobre|oktober|октябр[ьяе]": 10,
+        "november|noviembre|novembre|ноябр[ьяе]": 11,
+        "december|diciembre|dicembre|d[ée]cembre|dezember|декабр[ьяе]": 12,
     }
     for pattern, month_num in month_patterns.items():
         # Skip when a day number precedes the month ("13 июля 2026", "13 July 2026"):

hindsight-api-slim/hindsight_api/engine/transfer/export.py

@@ -193,7 +193,12 @@ async def export_documents(
         raise ValueError("include_observations is only supported when exporting the whole bank (omit document_id)")
 
     async with acquire_with_retry(backend) as conn:
-        loaded = await _load_documents(conn, bank_id, document_ids)
+        # Carry per-fact consolidation lifecycle exactly when observations are
+        # carried: with observations in the archive the target must NOT re-derive
+        # them, so imported facts keep their consolidated/failed state. Without
+        # observations (the default document export) the target re-consolidates
+        # from scratch, so lifecycle is deliberately dropped.
+        loaded = await _load_documents(conn, bank_id, document_ids, include_lifecycle=include_observations)
         documents = loaded.documents
         observations = await _load_observations(conn, bank_id, loaded.unit_index) if include_observations else []
 
@@ -293,9 +298,11 @@ async def export_bank(
     ``_current_schema`` and passes its raw connection; the engine acquires one
     after tenant auth).
     """
-    loaded = await _load_documents(conn, bank_id, None)
+    # Whole-bank export always carries observations (they're bank-level state)
+    # and, with them, the per-fact consolidation lifecycle so the target restores
+    # exact eligibility instead of re-consolidating historical facts (#2965).
+    loaded = await _load_documents(conn, bank_id, None, include_lifecycle=True)
     documents = loaded.documents
-    # Whole-bank export always carries observations (they're bank-level state).
     observations = await _load_observations(conn, bank_id, loaded.unit_index)
 
     bank_rows = {table: await _dump_bank_rows(conn, table, bank_id) for table in _BANK_ROW_TABLES}
@@ -356,6 +363,7 @@ async def _load_documents(
     conn: Any,
     bank_id: str,
     document_ids: list[str] | None,
+    include_lifecycle: bool = False,
 ) -> _LoadedExport:
     """Load and assemble TransferDocument payloads for the requested documents."""
     doc_filter = "AND id = ANY($2)" if document_ids else ""
@@ -377,7 +385,7 @@ async def _load_documents(
     selected_ids = [row["id"] for row in doc_rows]
 
     chunks_by_doc = await _load_chunks(conn, bank_id, selected_ids)
-    loaded = await _load_facts(conn, bank_id, selected_ids)
+    loaded = await _load_facts(conn, bank_id, selected_ids, include_lifecycle=include_lifecycle)
     await _attach_entities(conn, loaded)
     await _attach_causal_relations(conn, loaded)
 
@@ -472,17 +480,22 @@ async def _load_chunks(conn: Any, bank_id: str, doc_ids: list[str]) -> dict[str,
     return chunks_by_doc
 
 
-async def _load_facts(conn: Any, bank_id: str, doc_ids: list[str]) -> _LoadedFacts:
+async def _load_facts(conn: Any, bank_id: str, doc_ids: list[str], include_lifecycle: bool = False) -> _LoadedFacts:
     """Load non-observation facts grouped by document, with a unit-id location index.
 
     The ordering is fixed (created_at, id) so that
     ``causal_relations.target_fact_index`` ordinals stay consistent.
+
+    ``include_lifecycle`` carries each fact's ``created_at`` / ``consolidated_at`` /
+    ``consolidation_failed_at`` (whole-bank / with-observations export). It is left
+    off for the plain document export so the target re-consolidates from scratch.
     """
     rows = await conn.fetch(
         f"""
         SELECT id, document_id, text, fact_type, context, event_date,
                occurred_start, occurred_end, mentioned_at, metadata,
-               chunk_id, tags, observation_scopes
+               chunk_id, tags, observation_scopes,
+               created_at, consolidated_at, consolidation_failed_at
         FROM {fq_table("memory_units")}
         WHERE bank_id = $1
           AND document_id = ANY($2)
@@ -511,6 +524,9 @@ async def _load_facts(conn: Any, bank_id: str, doc_ids: list[str]) -> _LoadedFac
             tags=list(row["tags"] or []),
             observation_scopes=_as_jsonb(row["observation_scopes"]),
             chunk_index=_chunk_index_from_chunk_id(row["chunk_id"]),
+            created_at=row["created_at"] if include_lifecycle else None,
+            consolidated_at=row["consolidated_at"] if include_lifecycle else None,
+            consolidation_failed_at=row["consolidation_failed_at"] if include_lifecycle else None,
         )
         bucket.append(fact)
         loaded.unit_index[row["id"]] = _UnitLocation(document_id=doc_id, ordinal=ordinal)

hindsight-api-slim/hindsight_api/engine/transfer/importer.py

@@ -633,10 +633,31 @@ async def _import_one_document(
                     ops=ops,
                 )
 
-        try:
-            await entity_resolver.flush_pending_stats()
-        except Exception:
-            logger.warning("[transfer] Entity stats flush failed for document %s", target_id, exc_info=True)
+            # Restore the source consolidation lifecycle. A whole-bank transfer
+            # preserves exact eligibility: a fact that was consolidated (or that
+            # failed consolidation) in the source is never re-consolidated on the
+            # target, so the maintenance reconciler sees no phantom backlog and
+            # observations are not re-derived. Archives predating these fields
+            # carry None for all three -> skipped here, leaving the
+            # observation-driven marking in _import_observations as the only
+            # (lossy) signal, exactly as before.
+            if result_unit_ids:
+                await _restore_fact_lifecycle(
+                    conn,
+                    bank_id,
+                    document.facts,
+                    retained_index_by_original,
+                    result_unit_ids[0],
+                )
+
+    # Best-effort, and only after the acquire() block above has exited: this
+    # takes its own connection, and on Oracle the write above is not committed
+    # until that block exits, so flushing while still holding the connection
+    # deadlocks (see the retain orchestrator for the full explanation).
+    try:
+        await entity_resolver.flush_pending_stats()
+    except Exception:
+        logger.warning("[transfer] Entity stats flush failed for document %s", target_id, exc_info=True)
 
     logger.debug("[transfer] Imported document %s:\n%s", target_id, "\n".join(log_buffer))
     # Single content item -> result_unit_ids[0] follows the retained fact order.
@@ -651,6 +672,51 @@ async def _import_one_document(
     )
 
 
+async def _restore_fact_lifecycle(
+    conn: Any,
+    bank_id: str,
+    facts: list[TransferFact],
+    retained_index_by_original: list[int | None],
+    retained_unit_ids: list[str],
+) -> None:
+    """Apply each imported fact's source consolidation timestamps to its new row.
+
+    ``retained_unit_ids`` follows the retained fact order; ``retained_index_by_original[i]``
+    maps original fact ``i`` to its position there (or ``None`` if it was dropped
+    on insert, e.g. a duplicate). ``created_at`` restores source provenance only
+    when present (mirroring the document-row handling); ``consolidated_at`` /
+    ``consolidation_failed_at`` are set verbatim — a source-``NULL`` (unconsolidated)
+    fact stays eligible, which is correct.
+    """
+    rows: list[tuple[uuid.UUID, datetime | None, datetime | None, datetime | None]] = []
+    for original_index, fact in enumerate(facts):
+        retained_index = retained_index_by_original[original_index]
+        if retained_index is None:
+            continue
+        if fact.created_at is None and fact.consolidated_at is None and fact.consolidation_failed_at is None:
+            # Legacy archive without lifecycle fields — nothing to restore.
+            continue
+        rows.append(
+            (
+                uuid.UUID(retained_unit_ids[retained_index]),
+                fact.created_at,
+                fact.consolidated_at,
+                fact.consolidation_failed_at,
+            )
+        )
+    if not rows:
+        return
+    await conn.executemany(
+        f"UPDATE {fq_table('memory_units')} "
+        f"SET created_at = COALESCE($2, created_at), consolidated_at = $3, consolidation_failed_at = $4 "
+        f"WHERE id = $1 AND bank_id = $5",
+        [
+            (unit_id, created_at, consolidated_at, failed_at, bank_id)
+            for unit_id, created_at, consolidated_at, failed_at in rows
+        ],
+    )
+
+
 async def _import_observations(
     *,
     backend: Any,
@@ -732,10 +798,13 @@ async def _import_observations(
                 all_source_ids.update(source_uuids)
                 await _link_observation_sources(conn, ops, bank_id, observation_uuid, source_uuids, obs.proof_count)
 
-            # Mark source facts consolidated so the target consolidator skips them.
+            # Mark source facts consolidated so the target consolidator skips
+            # them. COALESCE keeps the exact source timestamp already restored by
+            # _restore_fact_lifecycle (new archives); now() is the fallback only
+            # for legacy archives that carry no per-fact lifecycle state.
             if all_source_ids:
                 await conn.execute(
-                    f"UPDATE {fq_table('memory_units')} SET consolidated_at = now() "
+                    f"UPDATE {fq_table('memory_units')} SET consolidated_at = COALESCE(consolidated_at, now()) "
                     f"WHERE bank_id = $1 AND id = ANY($2)",
                     bank_id,
                     list(all_source_ids),

hindsight-api-slim/hindsight_api/engine/transfer/schema.py

@@ -68,6 +68,15 @@ class TransferFact(BaseModel):
     # Entity canonical names; re-resolved against the target bank on import.
     entities: list[str] = Field(default_factory=list)
     causal_relations: list[TransferCausalRelation] = Field(default_factory=list)
+    # Consolidation lifecycle timestamps, carried verbatim by a whole-bank
+    # transfer so imported facts keep their exact consolidation eligibility: an
+    # already-consolidated or failed fact is never re-consolidated on the target,
+    # and the maintenance reconciler sees no phantom backlog. Absent in archives
+    # produced before these were added (-> None), in which case the importer
+    # falls back to marking only observation-referenced facts consolidated.
+    created_at: datetime | None = None
+    consolidated_at: datetime | None = None
+    consolidation_failed_at: datetime | None = None
 
 
 class TransferChunk(BaseModel):

hindsight-api-slim/hindsight_api/engine/utils.py

@@ -1,73 +0,0 @@
-"""
-Utility functions for memory system.
-"""
-
-import logging
-from datetime import datetime
-from typing import TYPE_CHECKING
-
-if TYPE_CHECKING:
-    from .llm_wrapper import LLMConfig
-    from .retain.fact_extraction import Fact
-
-from .retain.fact_extraction import extract_facts_from_text
-
-
-async def extract_facts(
-    text: str,
-    event_date: datetime,
-    context: str = "",
-    llm_config: "LLMConfig" = None,
-    agent_name: str = None,
-    config=None,
-) -> tuple[list["Fact"], list[tuple[str, int]]]:
-    """
-    Extract semantic facts from text using LLM.
-
-    Uses LLM for intelligent fact extraction that:
-    - Filters out social pleasantries and filler words
-    - Creates self-contained statements with absolute dates
-    - Handles conversational text well
-    - Resolves relative time expressions to absolute dates
-
-    Args:
-        text: Input text (conversation, article, etc.)
-        event_date: Reference date for resolving relative times
-        context: Context about the conversation/document
-        llm_config: LLM configuration to use
-        agent_name: Optional agent name to help identify agent-related facts
-        config: HindsightConfig to use (defaults to global config if not provided)
-
-    Returns:
-        Tuple of (facts, chunks) where:
-        - facts: List of Fact model instances
-        - chunks: List of tuples (chunk_text, fact_count) for each chunk
-
-    Raises:
-        Exception: If LLM fact extraction fails
-    """
-    if not text or not text.strip():
-        return [], []
-
-    # Use provided config or fall back to global config
-    if config is None:
-        from ..config import _get_raw_config
-
-        config = _get_raw_config()
-
-    facts, chunks, _ = await extract_facts_from_text(
-        text,
-        event_date,
-        llm_config=llm_config,
-        agent_name=agent_name,
-        config=config,
-        context=context,
-    )
-
-    if not facts:
-        logging.warning(
-            f"LLM extracted 0 facts from text of length {len(text)}. This may indicate the text contains no meaningful information, or the LLM failed to extract facts. Full text: {text}"
-        )
-        return [], chunks
-
-    return facts, chunks

hindsight-api-slim/hindsight_api/engine/vector_index_health.py

@@ -19,6 +19,7 @@ import logging
 from dataclasses import dataclass, field
 from typing import Any
 
+from .db_utils import retry_with_backoff
 from .retain.bank_utils import _BANK_INDEX_FACT_TYPES, _bank_index_name
 
 logger = logging.getLogger(__name__)
@@ -142,16 +143,33 @@ async def _repair_schema(
 
             qindex = _quote_identifier(index_name)
             qualified = f"{qschema}.{qindex}"
-            try:
-                # An unhealthy-but-present index (INVALID leftover, wrong access
-                # method) must be dropped first — IF NOT EXISTS cannot repair it.
-                if healthy is False:
-                    await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {qualified}")
+
+            async def _rebuild(
+                qindex: str = qindex,
+                qualified: str = qualified,
+                ft: str = ft,
+                bank_id_literal: str = bank_id_literal,
+            ) -> None:
+                # Always drop first. An unhealthy-but-present index (INVALID
+                # leftover, wrong access method) can't be repaired by
+                # IF NOT EXISTS, and a prior deadlocked CONCURRENTLY build leaves
+                # an INVALID stub that IF NOT EXISTS would likewise skip — so a
+                # retry must clear it. DROP ... IF EXISTS is a no-op when the
+                # index is simply absent (healthy is None).
+                await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {qualified}")
                 await conn.execute(
                     f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {qindex} "
                     f"ON {qschema}.memory_units {index_clause} "
                     f"WHERE fact_type = '{ft}' AND bank_id = {bank_id_literal}"
                 )
+
+            try:
+                # CREATE INDEX CONCURRENTLY on the live, concurrently-written
+                # memory_units table can be chosen as a deadlock victim
+                # (sqlstate 40P01 / ORA-00060). That is transient — Postgres
+                # aborts one side to break the cycle — so retry the drop+build a
+                # few times before recording a permanent failure.
+                await retry_with_backoff(_rebuild)
                 result.created += 1
             except Exception as exc:  # noqa: BLE001 — one failed index must not abort the rest
                 result.failed += 1

hindsight-api-slim/hindsight_api/extensions/__init__.py

@@ -15,6 +15,7 @@ Extensions receive an ExtensionContext that provides a controlled API for intera
 with the system (e.g., running migrations for tenant schemas).
 """
 
+from hindsight_api.extensions.bank_tables import BankScopedTable
 from hindsight_api.extensions.base import Extension
 from hindsight_api.extensions.builtin import (
     ApiKeyTenantExtension,
@@ -78,6 +79,7 @@ from hindsight_api.worker.exceptions import DeferOperation
 __all__ = [
     # Base
     "Extension",
+    "BankScopedTable",
     "load_extension",
     # Context
     "ExtensionContext",

hindsight-api-slim/hindsight_api/extensions/bank_tables.py

@@ -0,0 +1,60 @@
+"""Extension-declared, bank-scoped tables.
+
+An extension may provision its own tables in the tenant schema (e.g. audit
+receipts, per-bank policy state). Those tables are invisible to core, so they
+silently fall out of the per-tenant data-lifecycle operations core owns:
+
+* **Backup / restore** — ``hindsight-admin backup``/``restore`` copies a fixed
+  set of core tables and ``TRUNCATE ... CASCADE``\\ s them on restore. An
+  extension table absent from that set is dropped from the backup *and* — if it
+  carries a FK to ``banks`` — wiped by the cascade with no way to restore it.
+* **Bank teardown** — :meth:`MemoryEngine.delete_bank` clears a bank by
+  deleting the core rows and letting ``banks``' FK cascade handle the rest. An
+  extension table that scopes by ``bank_id`` without a cascading FK leaks
+  orphaned rows when the bank is deleted.
+
+An extension declares its bank-scoped tables via
+:meth:`TenantExtension.extra_bank_tables`; core consults that list in the
+operations above. The extension still owns the DDL (creation lives in its
+provisioning path) — this descriptor only tells core which tables to sweep.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+
+# Unquoted SQL identifiers only. ``name`` and ``bank_id_column`` are
+# interpolated into SQL (schema-qualified via ``fq_table``), so they must be
+# validated to a safe identifier shape rather than trusted verbatim.
+_IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
+
+
+@dataclass(frozen=True)
+class BankScopedTable:
+    """A bank-scoped table an extension owns and core should sweep.
+
+    Args:
+        name: Unqualified table name. Schema-qualified at use via ``fq_table``.
+        bank_id_column: Column holding the bank id, used to scope a per-bank
+            delete. Defaults to ``"bank_id"``.
+        include_in_backup: Include the table in ``hindsight-admin``
+            backup/restore. Defaults to ``True`` — a bank-scoped table almost
+            always wants restore coverage; opt out only for regenerable or
+            transient state.
+        delete_with_bank: Delete the table's rows for a bank during a full
+            :meth:`MemoryEngine.delete_bank`. Defaults to ``True``. Set
+            ``False`` to retain rows that should outlive the bank (e.g. audit
+            receipts a compliance regime requires kept).
+    """
+
+    name: str
+    bank_id_column: str = "bank_id"
+    include_in_backup: bool = True
+    delete_with_bank: bool = True
+
+    def __post_init__(self) -> None:
+        if not _IDENTIFIER_RE.match(self.name):
+            raise ValueError(f"BankScopedTable.name {self.name!r} is not a valid SQL identifier")
+        if not _IDENTIFIER_RE.match(self.bank_id_column):
+            raise ValueError(f"BankScopedTable.bank_id_column {self.bank_id_column!r} is not a valid SQL identifier")

hindsight-api-slim/hindsight_api/extensions/context.py

@@ -160,6 +160,19 @@ class DefaultExtensionContext(ExtensionContext):
             schema=schema,
         )
 
+        # Provision any extension-owned bank-scoped tables for this schema,
+        # right after core migrations, so extension schema evolves on the same
+        # lifecycle as core schema (instead of via a lazy per-request path).
+        # No-op unless a tenant extension declares a provisioner; errors
+        # propagate so a failed provision surfaces here, not at request time.
+        engine = self._memory_engine
+        get_pool = getattr(engine, "_get_pool", None)
+        tenant_extension = getattr(engine, "tenant_extension", None)
+        if get_pool is not None and tenant_extension is not None:
+            pool = await get_pool()
+            async with pool.acquire() as conn:
+                await tenant_extension.provision_bank_tables(conn, schema)
+
     def get_memory_engine(self) -> "MemoryEngineInterface":
         """Get the memory engine interface."""
         if self._memory_engine is None:

hindsight-api-slim/hindsight_api/extensions/tenant.py

@@ -2,11 +2,15 @@
 
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
+from hindsight_api.extensions.bank_tables import BankScopedTable
 from hindsight_api.extensions.base import Extension
 from hindsight_api.models import RequestContext
 
+if TYPE_CHECKING:
+    import asyncpg
+
 
 class AuthenticationError(Exception):
     """Raised when authentication fails."""
@@ -143,6 +147,50 @@ class TenantExtension(Extension, ABC):
         """
         return None
 
+    def extra_bank_tables(self) -> list[BankScopedTable]:
+        """Bank-scoped tables this extension provisions in the tenant schema.
+
+        Core consults this list so extension-owned tables participate in the
+        per-tenant data-lifecycle operations it manages — admin backup/restore
+        and :meth:`MemoryEngine.delete_bank` teardown — instead of silently
+        falling out of them (dropped on restore, or leaked as orphaned rows on
+        bank deletion). See :class:`BankScopedTable`.
+
+        The extension still owns the DDL; this only declares which tables exist.
+        The default is no extra tables.
+
+        Returns:
+            The extension's bank-scoped tables. Empty by default.
+        """
+        return []
+
+    async def provision_bank_tables(self, conn: "asyncpg.Connection", schema: str) -> None:
+        """Create/evolve this extension's tables in ``schema`` (idempotent DDL).
+
+        Called from the **migration path** for every schema — both when a
+        tenant schema is provisioned (via ``ExtensionContext.run_migration``)
+        and by the ``hindsight-admin run-db-migration`` sweep across all
+        existing schemas — right after core migrations complete. This is the
+        counterpart to :meth:`extra_bank_tables`: that one *declares* the
+        tables for backup/teardown, this one *creates* them, so extension
+        schema evolves on the same lifecycle as core schema instead of via a
+        lazy per-request path.
+
+        Implementations MUST be idempotent (``CREATE TABLE IF NOT EXISTS`` /
+        ``ADD COLUMN IF NOT EXISTS``) — it runs on every provision and every
+        migration sweep — and MUST schema-qualify every statement with
+        ``schema`` (the connection's ``search_path`` is not set for you).
+        Exceptions propagate so a failed provision surfaces at migration time
+        rather than as a runtime error on a later request.
+
+        Args:
+            conn: An open connection to the target database.
+            schema: The schema to provision tables into.
+
+        The default does nothing.
+        """
+        return None
+
     async def authenticate_mcp(self, context: RequestContext) -> TenantContext:
         """
         Authenticate MCP requests.

hindsight-api-slim/hindsight_api/loop_watchdog.py

@@ -0,0 +1,147 @@
+"""Event-loop stall watchdog.
+
+Hindsight's worker and API run the ``/health`` handler and all task work on one
+asyncio event loop. If something does blocking (synchronous) work on that loop —
+CPU-bound parsing, a mis-offloaded SDK call, a third-party library that signs a
+request inline — the loop stops servicing coroutines, ``/health`` can't be
+scheduled, and a Kubernetes liveness probe fails even though the process is "up".
+
+This watchdog makes that condition self-diagnosing. It runs in a **separate OS
+thread** (deliberately: a coroutine-based monitor would be frozen by the very
+stall it's trying to observe), pings the loop, and when the loop fails to service
+the ping within a threshold it logs the loop thread's current stack — naming the
+exact frame that is blocking. It never raises and never touches the loop's work;
+it only observes. Unlike monkeypatch-based blocking detectors it works with
+uvloop, because it relies only on ``loop.call_soon_threadsafe`` and
+``sys._current_frames()``.
+
+It is the loop-side counterpart to the DB-pool acquire instrumentation
+(``engine/db/pool_instrumentation.py``): together they let a stuck ``/health`` be
+attributed to either a blocked loop or connection-pool exhaustion from the logs
+alone.
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+import threading
+import time
+import traceback
+from collections.abc import Callable
+
+logger = logging.getLogger("hindsight.loop_watchdog")
+
+
+def start_loop_watchdog(loop) -> "LoopWatchdog | None":
+    """Build and start a watchdog for ``loop`` from config, or return None if disabled.
+
+    Call this once, from inside the running loop's process (worker CLI / API lifespan),
+    and call ``.stop()`` on the returned handle at shutdown.
+    """
+    from .config import get_config
+
+    config = get_config()
+    if not config.loop_watchdog_enabled:
+        return None
+    watchdog = LoopWatchdog(
+        loop,
+        stall_threshold_s=config.loop_watchdog_stall_threshold_ms / 1000.0,
+        poll_interval_s=config.loop_watchdog_poll_interval_ms / 1000.0,
+    )
+    watchdog.start()
+    return watchdog
+
+
+class LoopWatchdog:
+    """Detects event-loop stalls from an off-loop thread and logs the culprit stack.
+
+    Args:
+        loop: the asyncio event loop to monitor.
+        stall_threshold_s: log when the loop takes at least this long to service a ping.
+        poll_interval_s: how often to ping the loop.
+        on_stall: optional callback ``(blocked_for_s, stack_text)`` invoked on each
+            detected stall instead of the default log+metric path. Used for testing.
+    """
+
+    def __init__(
+        self,
+        loop,
+        *,
+        stall_threshold_s: float = 1.0,
+        poll_interval_s: float = 0.25,
+        on_stall: Callable[[float, str], None] | None = None,
+    ) -> None:
+        self._loop = loop
+        self._stall_threshold_s = stall_threshold_s
+        self._poll_interval_s = poll_interval_s
+        self._on_stall = on_stall
+        self._stop = threading.Event()
+        self._loop_thread_id: int | None = None
+        self._thread = threading.Thread(target=self._run, name="loop-watchdog", daemon=True)
+        self._started = False
+
+    def start(self) -> None:
+        """Start monitoring. Must not block the loop — the id is captured via pings.
+
+        When called from the loop thread itself (the normal case: worker ``run()`` /
+        API lifespan), ``threading.get_ident()`` is already the loop thread id, so we
+        seed it here; each ping then re-affirms it authoritatively. We deliberately do
+        NOT schedule-and-wait for a callback: that would deadlock, because the loop
+        can't run the callback while ``start()`` is blocking it.
+        """
+        self._loop_thread_id = threading.get_ident()
+        self._started = True
+        self._thread.start()
+        logger.info(
+            "Loop watchdog started (stall_threshold=%.2fs, poll_interval=%.2fs)",
+            self._stall_threshold_s,
+            self._poll_interval_s,
+        )
+
+    def stop(self) -> None:
+        self._stop.set()
+        if self._started and self._thread.is_alive():
+            self._thread.join(timeout=self._poll_interval_s + self._stall_threshold_s + 1.0)
+
+    def _run(self) -> None:
+        while not self._stop.wait(self._poll_interval_s):
+            serviced = threading.Event()
+            sent_at = time.monotonic()
+
+            def _ping() -> None:
+                # Runs on the loop thread — capture its id authoritatively, then
+                # signal that the loop serviced this ping.
+                self._loop_thread_id = threading.get_ident()
+                serviced.set()
+
+            try:
+                self._loop.call_soon_threadsafe(_ping)
+            except RuntimeError:
+                return  # loop closed — nothing left to watch
+            if not serviced.wait(self._stall_threshold_s):
+                self._report(sent_at)
+                # Block until the loop finally services the ping so we emit one
+                # report per stall, not one per poll while it stays blocked.
+                serviced.wait()
+
+    def _report(self, sent_at: float) -> None:
+        frame = sys._current_frames().get(self._loop_thread_id or -1)
+        stack = "".join(traceback.format_stack(frame)) if frame is not None else "<loop-thread frame unavailable>"
+        blocked_for = time.monotonic() - sent_at
+        if self._on_stall is not None:
+            self._on_stall(blocked_for, stack)
+            return
+        logger.warning(
+            "EVENT LOOP BLOCKED for >= %.2fs (%.2fs and counting). The loop is not "
+            "servicing coroutines — /health cannot be scheduled. Blocking frame:\n%s",
+            self._stall_threshold_s,
+            blocked_for,
+            stack,
+        )
+        try:
+            from .metrics import get_metrics_collector
+
+            get_metrics_collector().record_loop_stall(blocked_for)
+        except Exception:
+            pass

hindsight-api-slim/hindsight_api/main.py

@@ -32,6 +32,7 @@ from .config import (
     ENV_WORKERS,
     HindsightConfig,
     _get_raw_config,
+    load_dotenv_for_entrypoint,
 )
 from .daemon import (
     DEFAULT_DAEMON_PORT,
@@ -197,6 +198,8 @@ def main():
     """Main entry point for the CLI."""
     global _memory
 
+    load_dotenv_for_entrypoint()
+
     # Load configuration from environment (for CLI args defaults)
     config = _get_raw_config()
 

hindsight-api-slim/hindsight_api/mcp_tools.py

@@ -1020,6 +1020,7 @@ def _register_reflect(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig
             response_schema: dict | None = None,
             tags: list[str] | None = None,
             tags_match: str = "any",
+            apply_all_directives: bool = False,
             include_based_on: bool = False,
             include_trace: bool = False,
             bank_id: str | None = None,
@@ -1051,6 +1052,7 @@ def _register_reflect(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig
                 response_schema: Optional JSON schema for structured output. When provided, the response includes a 'structured_output' field.
                 tags: Optional tags to filter memories by (e.g., ['project:alpha'])
                 tags_match: How to match tags - 'any' (match any tag) or 'all' (match all tags). Default: 'any'
+                apply_all_directives: Apply every active directive regardless of tags. By default directives are scoped like memories (untagged always apply; tagged apply only when tags match). Set true to apply all directives, ignoring tag scope.
                 include_based_on: Include source facts used for synthesis. Defaults to false because broad reflections can exceed MCP client result limits.
                 include_trace: Include the reflection's internal trace fields (tool_trace/llm_trace and directives_applied). Defaults to false because the trace can be tens of KB and overflow MCP client context; enable only for debugging.
                 bank_id: Optional bank to reflect in (defaults to session bank). Use for cross-bank operations.
@@ -1069,6 +1071,7 @@ def _register_reflect(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig
                     "budget": budget_enum,
                     "context": context,
                     "max_tokens": max_tokens,
+                    "apply_all_directives": apply_all_directives,
                     "request_context": _get_request_context(config),
                 }
                 if response_schema is not None:
@@ -1112,6 +1115,7 @@ def _register_reflect(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig
             response_schema: dict | None = None,
             tags: list[str] | None = None,
             tags_match: str = "any",
+            apply_all_directives: bool = False,
             include_based_on: bool = False,
             include_trace: bool = False,
         ) -> dict:
@@ -1142,6 +1146,7 @@ def _register_reflect(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig
                 response_schema: Optional JSON schema for structured output. When provided, the response includes a 'structured_output' field.
                 tags: Optional tags to filter memories by (e.g., ['project:alpha'])
                 tags_match: How to match tags - 'any' (match any tag) or 'all' (match all tags). Default: 'any'
+                apply_all_directives: Apply every active directive regardless of tags. By default directives are scoped like memories (untagged always apply; tagged apply only when tags match). Set true to apply all directives, ignoring tag scope.
                 include_based_on: Include source facts used for synthesis. Defaults to false because broad reflections can exceed MCP client result limits.
                 include_trace: Include the reflection's internal trace fields (tool_trace/llm_trace and directives_applied). Defaults to false because the trace can be tens of KB and overflow MCP client context; enable only for debugging.
             """
@@ -1159,6 +1164,7 @@ def _register_reflect(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig
                     "budget": budget_enum,
                     "context": context,
                     "max_tokens": max_tokens,
+                    "apply_all_directives": apply_all_directives,
                     "request_context": _get_request_context(config),
                 }
                 if response_schema is not None:
@@ -1235,20 +1241,16 @@ def _register_create_bank(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsCo
         """
         try:
             request_context = _get_request_context(config)
-            # create_bank may auto-create the bank; validate that explicit
-            # creation permission before reading the resulting profile.
-            await memory._ensure_bank_exists(bank_id, request_context)
-            profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-
-            # Update name/mission if provided
             if name is not None or mission is not None:
-                await memory.update_bank(
+                profile = await memory.update_bank(
                     bank_id,
                     name=name,
                     mission=mission,
                     request_context=request_context,
                 )
-                # Fetch updated profile
+            else:
+                # The public profile API owns bank creation and its lifecycle
+                # validation when no profile fields need updating.
                 profile = await memory.get_bank_profile(bank_id, request_context=request_context)
 
             # Serialize disposition if it's a Pydantic model
@@ -3290,28 +3292,21 @@ async def _do_update_bank(
     Args:
         name: Display name (stored in banks table).
         mission: Deprecated alias for reflect_mission — mapped into config_updates.
-        config_updates: Arbitrary config overrides passed to config_resolver.update_bank_config().
+        config_updates: Arbitrary config overrides passed to MemoryEngine.update_bank_config().
             Supports all configurable fields (retain_mission, disposition_*, etc.).
             The config resolver validates keys and rejects non-configurable/credential fields.
     """
-    # Update display name via engine (stored in DB banks table)
-    if name is not None:
-        await memory.update_bank(
-            target_bank,
-            name=name,
-            request_context=request_context,
-        )
-
     # Merge deprecated mission alias into config_updates as reflect_mission
     effective_config: dict[str, Any] = dict(config_updates) if config_updates else {}
     if mission is not None and "reflect_mission" not in effective_config:
         effective_config["reflect_mission"] = mission
 
-    if effective_config:
-        await memory._config_resolver.update_bank_config(target_bank, effective_config, request_context)
-
-    # Return updated profile
-    return await memory.get_bank_profile(target_bank, request_context=request_context)
+    return await memory.update_bank(
+        target_bank,
+        name=name,
+        config_updates=effective_config or None,
+        request_context=request_context,
+    )
 
 
 def _register_update_bank(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig) -> None:

hindsight-api-slim/hindsight_api/metrics.py

@@ -292,6 +292,18 @@ class MetricsCollectorBase:
         """Context manager to record HTTP request metrics."""
         raise NotImplementedError
 
+    def record_retain_document(self, bank_id: str, memory_unit_count: int):
+        """Record the fact-extraction outcome of one document processed by retain."""
+        raise NotImplementedError
+
+    def record_db_acquire_wait(self, wait_seconds: float):
+        """Record how long a caller waited to acquire a pooled DB connection."""
+        raise NotImplementedError
+
+    def record_loop_stall(self, stall_seconds: float):
+        """Record a detected event-loop stall (blocked longer than the watchdog threshold)."""
+        raise NotImplementedError
+
     def set_db_pool(self, pool: "asyncpg.Pool"):
         """Set the database pool for metrics collection."""
         pass
@@ -345,6 +357,18 @@ class NoOpMetricsCollector(MetricsCollectorBase):
         """No-op HTTP request recording."""
         yield
 
+    def record_retain_document(self, bank_id: str, memory_unit_count: int):
+        """No-op retain document outcome recording."""
+        pass
+
+    def record_db_acquire_wait(self, wait_seconds: float):
+        """No-op DB acquire-wait recording."""
+        pass
+
+    def record_loop_stall(self, stall_seconds: float):
+        """No-op loop-stall recording."""
+        pass
+
 
 class MetricsCollector(MetricsCollectorBase):
     """
@@ -411,6 +435,18 @@ class MetricsCollector(MetricsCollectorBase):
             unit="tokens",
         )
 
+        # Per-document retain outcome. The point of this counter is the
+        # ``outcome=no_facts`` series: a document whose extraction legitimately
+        # produced zero facts is stored but unreachable via recall/reflect (only
+        # memory_units carry embeddings), and nothing else in the system reports
+        # it — the operation still completes successfully. Alert on it to catch a
+        # retain_mission that silently excludes more than intended (issue #3040).
+        self.retain_documents_total = self.meter.create_counter(
+            name="hindsight.retain.documents.total",
+            description="Documents processed by retain, labelled by extraction outcome (facts/no_facts)",
+            unit="documents",
+        )
+
         # HTTP request metrics
         self.http_request_duration = self.meter.create_histogram(
             name="hindsight.http.duration", description="Duration of HTTP requests in seconds", unit="s"
@@ -426,6 +462,25 @@ class MetricsCollector(MetricsCollectorBase):
             unit="requests",
         )
 
+        # Runtime-stall observability: how long callers wait for a pooled DB
+        # connection (pool-exhaustion signal) and detected event-loop stalls
+        # (blocked-loop signal). See loop_watchdog.py and db/pool_instrumentation.py.
+        self.db_acquire_wait = self.meter.create_histogram(
+            name="hindsight.db.pool.acquire_wait",
+            description="Time spent waiting to acquire a pooled database connection",
+            unit="s",
+        )
+        self.event_loop_stalls = self.meter.create_counter(
+            name="hindsight.event_loop.stalls",
+            description="Number of detected event-loop stalls (loop blocked past the watchdog threshold)",
+            unit="stalls",
+        )
+        self.event_loop_stall_duration = self.meter.create_histogram(
+            name="hindsight.event_loop.stall_duration",
+            description="Duration of detected event-loop stalls in seconds",
+            unit="s",
+        )
+
         # Process metrics (observable gauges - collected on scrape)
         self._setup_process_metrics()
 
@@ -528,6 +583,22 @@ class MetricsCollector(MetricsCollectorBase):
         # Record operation count
         self.operation_total.add(1, attributes)
 
+    def record_retain_document(self, bank_id: str, memory_unit_count: int):
+        """Record one document's retain outcome.
+
+        ``memory_unit_count == 0`` means fact extraction ran and returned
+        nothing, so the document is stored but unreachable through recall/reflect
+        until it is reprocessed.
+        """
+        attributes = {
+            "tenant": _get_tenant(),
+            "outcome": "facts" if memory_unit_count > 0 else "no_facts",
+        }
+        if self._include_bank_id:
+            attributes["bank_id"] = bank_id
+
+        self.retain_documents_total.add(1, attributes)
+
     def record_llm_call(
         self,
         provider: str,
@@ -646,6 +717,15 @@ class MetricsCollector(MetricsCollectorBase):
             # Decrement in-progress
             self.http_requests_in_progress.add(-1, base_attributes)
 
+    def record_db_acquire_wait(self, wait_seconds: float):
+        """Record how long a caller waited to acquire a pooled DB connection."""
+        self.db_acquire_wait.record(wait_seconds)
+
+    def record_loop_stall(self, stall_seconds: float):
+        """Record a detected event-loop stall. Called from the watchdog thread."""
+        self.event_loop_stalls.add(1)
+        self.event_loop_stall_duration.record(stall_seconds)
+
     def _setup_process_metrics(self):
         """Set up observable gauges for process metrics."""
         if _resource_mod is None:
@@ -771,6 +851,20 @@ class MetricsCollector(MetricsCollectorBase):
                 except Exception:
                     pass
 
+        def get_pool_waiting(_options):
+            """Number of callers currently blocked waiting to acquire a connection.
+
+            asyncpg does not expose this; it's tracked in db/pool_instrumentation.py.
+            This is the gauge that actually distinguishes pool exhaustion (a high,
+            sustained value) from a merely busy-but-healthy pool.
+            """
+            try:
+                from .engine.db.pool_instrumentation import waiting_count
+
+                yield metrics.Observation(waiting_count())
+            except Exception:
+                pass
+
         # Create observable gauges for pool metrics
         self.meter.create_observable_gauge(
             name="hindsight.db.pool.size",
@@ -800,6 +894,13 @@ class MetricsCollector(MetricsCollectorBase):
             unit="{connections}",
         )
 
+        self.meter.create_observable_gauge(
+            name="hindsight.db.pool.waiting",
+            callbacks=[get_pool_waiting],
+            description="Callers currently blocked waiting to acquire a pooled connection",
+            unit="{connections}",
+        )
+
     def _setup_backlog_metrics(self):
         """Observable gauges for the async-operation queue and the
         consolidation backlog.

hindsight-api-slim/hindsight_api/migrations.py

@@ -17,7 +17,6 @@ No alembic.ini required - all configuration is done programmatically.
 
 import hashlib
 import logging
-import os
 import threading
 import time
 from pathlib import Path
@@ -390,65 +389,6 @@ def run_migrations(
         raise RuntimeError("Database migration failed") from e
 
 
-def check_migration_status(
-    database_url: str | None = None, script_location: str | None = None
-) -> tuple[str | None, str | None]:
-    """
-    Check current database schema version and latest available version.
-
-    Args:
-        database_url: SQLAlchemy database URL. If None, uses HINDSIGHT_API_DATABASE_URL env var.
-        script_location: Path to alembic migrations directory. If None, uses default location.
-
-    Returns:
-        Tuple of (current_revision, head_revision)
-        Returns (None, None) if unable to determine versions
-    """
-    try:
-        from alembic.runtime.migration import MigrationContext
-        from alembic.script import ScriptDirectory
-        from sqlalchemy import create_engine
-
-        # Get database URL
-        if database_url is None:
-            database_url = os.getenv("HINDSIGHT_API_DATABASE_URL")
-        if not database_url:
-            logger.warning(
-                "Database URL not provided and HINDSIGHT_API_DATABASE_URL not set, cannot check migration status"
-            )
-            return None, None
-
-        # Get current revision from database
-        engine = create_engine(to_libpq_url(database_url), poolclass=NullPool)
-        with engine.connect() as connection:
-            context = MigrationContext.configure(connection)
-            current_rev = context.get_current_revision()
-
-        # Get head revision from migration scripts
-        if script_location is None:
-            package_dir = Path(__file__).parent
-            script_location = str(package_dir / "alembic")
-
-        script_path = Path(script_location)
-        if not script_path.exists():
-            logger.warning(f"Script location not found at {script_location}")
-            return current_rev, None
-
-        # Create config programmatically
-        alembic_cfg = Config()
-        _set_alembic_main_option(alembic_cfg, "script_location", script_location)
-        _set_alembic_main_option(alembic_cfg, "path_separator", "os")
-
-        script = ScriptDirectory.from_config(alembic_cfg)
-        head_rev = script.get_current_head()
-
-        return current_rev, head_rev
-
-    except Exception as e:
-        logger.warning(f"Unable to check migration status: {e}")
-        return None, None
-
-
 def _migrate_table_embedding_dimension(
     conn: Connection,
     schema_name: str,

hindsight-api-slim/hindsight_api/pg0.py

@@ -131,29 +131,6 @@ class EmbeddedPostgres:
         return await self.start()
 
 
-_default_instance: EmbeddedPostgres | None = None
-
-
-def get_embedded_postgres() -> EmbeddedPostgres:
-    """Get or create the default EmbeddedPostgres instance."""
-    global _default_instance
-    if _default_instance is None:
-        _default_instance = EmbeddedPostgres()
-    return _default_instance
-
-
-async def start_embedded_postgres() -> str:
-    """Quick start function for embedded PostgreSQL."""
-    return await get_embedded_postgres().ensure_running()
-
-
-async def stop_embedded_postgres() -> None:
-    """Stop the default embedded PostgreSQL instance."""
-    global _default_instance
-    if _default_instance:
-        await _default_instance.stop()
-
-
 @dataclass(frozen=True)
 class Pg0Url:
     """Parsed representation of a ``pg0`` embedded-database URL.

hindsight-api-slim/hindsight_api/server.py

@@ -17,7 +17,7 @@ warnings.filterwarnings("ignore", message="websockets.server.WebSocketServerProt
 
 from hindsight_api import MemoryEngine
 from hindsight_api.api import create_app
-from hindsight_api.config import get_config
+from hindsight_api.config import get_config, load_dotenv_for_entrypoint
 from hindsight_api.extensions import (
     DefaultExtensionContext,
     OperationValidatorExtension,
@@ -25,6 +25,13 @@ from hindsight_api.extensions import (
     load_extension,
 )
 
+# This module IS an entry point: it is the ASGI app targeted by
+# `uvicorn hindsight_api.server:app`, and the import string uvicorn re-imports
+# in each worker process when `hindsight-api` runs with --workers/--reload. Load
+# .env here (like the CLI entry points) so those worker processes see the same
+# configuration. Importing hindsight_api as a library never reaches this module.
+load_dotenv_for_entrypoint()
+
 # Disable tokenizers parallelism to avoid warnings
 os.environ["TOKENIZERS_PARALLELISM"] = "false"
 

hindsight-api-slim/hindsight_api/webhooks/models.py

@@ -22,6 +22,16 @@ class ConsolidationEventData(BaseModel):
 class RetainEventData(BaseModel):
     document_id: str | None = None
     tags: list[str] | None = None
+    memory_unit_count: int | None = Field(
+        default=None,
+        description=(
+            "Memory units the document owns after this retain (the same number the "
+            "Documents API reports). 0 means fact extraction returned nothing, so the "
+            "document is stored but unreachable through recall/reflect until it is "
+            "reprocessed. Null when the retain carried no document_id, since there is "
+            "then no document to count against."
+        ),
+    )
 
 
 class MemoryDefenseHit(BaseModel):

hindsight-api-slim/hindsight_api/worker/main.py

@@ -18,7 +18,7 @@ import sys
 import warnings
 from collections.abc import Callable
 
-from ..config import get_config
+from ..config import get_config, load_dotenv_for_entrypoint
 from ..engine.task_backend import WorkerTaskBackend
 from .poller import WorkerPoller
 
@@ -125,6 +125,8 @@ def create_worker_app(poller: WorkerPoller, memory):
 
 def main():
     """Main entry point for the hindsight-worker CLI."""
+    load_dotenv_for_entrypoint()
+
     # Load configuration from environment
     config = get_config()
 
@@ -320,6 +322,13 @@ def main():
         )
         server = uvicorn.Server(uvicorn_config)
 
+        # Start the event-loop stall watchdog: if a task blocks the loop, this
+        # logs the culprit stack so a failing /health can be attributed to a
+        # blocked loop (vs DB-pool exhaustion, which the pool instrumentation logs).
+        from ..loop_watchdog import start_loop_watchdog
+
+        loop_watchdog = start_loop_watchdog(loop)
+
         # Run the poller and HTTP server concurrently
         poller_task = asyncio.create_task(poller.run())
         http_task = asyncio.create_task(server.serve())
@@ -333,6 +342,9 @@ def main():
             print("\nReceived interrupt, initiating graceful shutdown...")
 
         # Graceful shutdown
+        if loop_watchdog is not None:
+            loop_watchdog.stop()
+
         print("Shutting down HTTP server...")
         server.should_exit = True
 

hindsight-api-slim/hindsight_api/worker/poller.py

@@ -19,6 +19,7 @@ from collections.abc import Awaitable, Callable, Iterable
 from dataclasses import dataclass
 from typing import TYPE_CHECKING, Any
 
+from ..config import get_config
 from ..engine.schema import fq_table_explicit as fq_table
 from ..metrics import get_metrics_collector
 from .exceptions import DeferOperation, RetryTaskAt
@@ -37,6 +38,34 @@ def _metric_operation_label(operation_type: str | None) -> str:
     return operation_type or "unknown"
 
 
+def _wall_timeout_for(task_type: str) -> float | None:
+    """Wall-clock ceiling for one task of this type, or None when unbounded.
+
+    A task that wedges (a lock wait with no deadlock cycle to break it, an LLM
+    permit that never frees, a producer blocked on a queue nobody drains) holds
+    its worker slot forever: the operation stays 'processing', which the API
+    refuses to retry *or* cancel, and once every slot is held the worker stops
+    claiming work entirely (#3002). Per-operation timeouts elsewhere bound one
+    LLM call or one query, never the whole task — this is the outer backstop
+    that turns "wedged until restart" into "failed and retryable".
+
+    Only retain is bounded today; reflect self-bounds inside the engine
+    (``reflect_wall_timeout``) and the remaining types have no reported wedge.
+    """
+    if task_type in _RETAIN_OP_TYPES:
+        timeout = get_config().retain_wall_timeout
+        return float(timeout) if timeout > 0 else None
+    return None
+
+
+class _WallTimeoutExceeded(Exception):
+    """A task was cancelled because it blew through its wall-clock ceiling."""
+
+    def __init__(self, timeout: float) -> None:
+        super().__init__(f"wall-clock timeout after {timeout:.0f}s")
+        self.timeout = timeout
+
+
 def _updated_row_count(result: Any) -> int:
     """Extract a row count from backend execute() results."""
     if isinstance(result, int):
@@ -727,6 +756,26 @@ class WorkerPoller:
                     if self._in_flight_by_type[operation_type] == 0:
                         del self._in_flight_by_type[operation_type]
 
+    async def _run_executor(self, task: ClaimedTask, task_type: str) -> None:
+        """Run the task executor under its type's wall-clock ceiling, if it has one."""
+        wall_timeout = _wall_timeout_for(task_type)
+        if wall_timeout is None:
+            await self._executor(task.task_dict)
+            return
+
+        # asyncio.timeout() rather than wait_for(): `expired()` distinguishes our
+        # ceiling firing from an inner TimeoutError merely bubbling out (an asyncpg
+        # command timeout, say), which wait_for would surface as the same exception.
+        # Reporting a task's own timeout as a wedge would send operators hunting for
+        # the wrong thing.
+        try:
+            async with asyncio.timeout(wall_timeout) as cm:
+                await self._executor(task.task_dict)
+        except asyncio.TimeoutError as e:
+            if cm.expired():
+                raise _WallTimeoutExceeded(wall_timeout) from e
+            raise
+
     async def _execute_task_inner(self, task: ClaimedTask, holder: StageHolder | None = None):
         """Inner task execution with retry/fail handling.
 
@@ -769,10 +818,25 @@ class WorkerPoller:
             logger.debug(f"Executing task {task.operation_id} (type={task_type}, bank={bank_id}{schema_info})")
             if task.schema:
                 task.task_dict["_schema"] = task.schema
-            await self._executor(task.task_dict)
+            await self._run_executor(task, task_type)
             logger.debug(f"Task {task.operation_id} execution finished")
             await self._mark_completed(task.operation_id, task.schema)
             terminal_success = True
+        except _WallTimeoutExceeded as e:
+            # The executor has already been cancelled; all that's left is to say so
+            # clearly. Handled apart from the generic branch below so the operator
+            # sees the wedge for what it is rather than a bare "TimeoutError", and
+            # so the stage that was current when the ceiling fired is preserved —
+            # that breadcrumb is the only pointer to where the task was stuck.
+            stage = holder.stage if holder is not None else "unknown"
+            message = (
+                f"Task exceeded the {e.timeout:.0f}s wall-clock limit for '{task_type}' "
+                f"(stage={stage}) and was cancelled. Raise HINDSIGHT_API_RETAIN_WALL_TIMEOUT "
+                f"if this is a legitimately long operation, or set it to 0 to disable the limit."
+            )
+            logger.error(f"Task {task.operation_id} timed out: {message}")
+            await self._mark_failed(task.operation_id, message, task.schema)
+            terminal_success = False
         except DeferOperation as e:
             # Deferral is not a terminal outcome — do not record a completion.
             await self._defer_operation(task.operation_id, e.exec_date, e.reason, task.schema)
@@ -880,6 +944,13 @@ class WorkerPoller:
                         async with conn.transaction():
                             await self._maybe_update_parent_operation(str(failed_row["operation_id"]), schema, conn)
 
+                # Finalize batch_retain parents that the aggregation left behind
+                # (crash between a child's terminal commit and the parent update,
+                # or children that never committed). These sit 'pending' with a
+                # NULL payload — unclaimable and invisible to failed_operations —
+                # until reconciled. See issue #2985.
+                await self._reconcile_orphaned_parents(schema)
+
                 pending_count = int(result.split()[-1]) if result else 0
                 failed_count = len(failed_rows)
                 total_count += pending_count
@@ -968,6 +1039,126 @@ class WorkerPoller:
             logger.error(f"Failed to recover batch operations for schema {schema_display}: {e}")
             return 0
 
+    async def _reconcile_orphaned_parents(self, schema: str | None) -> int:
+        """Drive stranded batch_retain parent operations to a terminal state.
+
+        A batch_retain parent is a status *aggregator*: it carries no
+        task_payload (workers never claim it) and is normally promoted to a
+        terminal state by ``_maybe_update_parent_operation`` when its last child
+        sub-batch finishes. Two crash windows can strand a parent 'pending'
+        forever:
+
+          * every child reached a terminal state but the promotion was skipped
+            (the aggregation swallows and logs errors rather than failing the
+            child, so a transient error there leaves the parent behind), or
+          * the children never committed at all, leaving a parent with zero
+            children (older non-atomic create paths, a hard kill mid-submission).
+
+        Either way the parent sits 'pending' with ``task_payload IS NULL``: it is
+        unclaimable, never counted in ``failed_operations``, unretryable via the
+        API, and its documents are silently absent. See issue #2985.
+
+        On worker startup we reconcile every such parent:
+
+          * children present, all terminal -> completed / failed (mirrors the
+            aggregator, inheriting a representative child error on failure),
+          * no children at all -> failed, with an explicit reason so an operator
+            can see the loss and resubmit the source documents.
+
+        Parents with at least one still-live child are left untouched — the
+        normal aggregation path will finish them once their children drain.
+
+        Returns the number of parents driven to a terminal state.
+        """
+        table = fq_table("async_operations", schema)
+        schema_display = f'"{schema}"' if schema else str(schema)
+        reconciled = 0
+        try:
+            async with self._backend.acquire() as conn:
+                parents = await conn.fetch(
+                    f"""
+                    SELECT operation_id, bank_id FROM {table}
+                    WHERE operation_type = 'batch_retain'
+                      AND status = 'pending'
+                      AND task_payload IS NULL
+                    """
+                )
+
+            for parent in parents:
+                parent_id = parent["operation_id"]
+                bank_id = parent["bank_id"]
+                # One transaction per parent so a single problematic row can't
+                # roll back the others (mirrors the per-child rollup above).
+                async with self._backend.acquire() as conn:
+                    async with conn.transaction():
+                        # Re-read under a row lock: skip if the normal aggregation
+                        # path (or another worker) already moved it off 'pending'.
+                        locked = await conn.fetchrow(
+                            f"SELECT status FROM {table} WHERE operation_id = $1 FOR UPDATE",
+                            parent_id,
+                        )
+                        if locked is None or locked["status"] != "pending":
+                            continue
+
+                        siblings = await conn.fetch(
+                            f"""
+                            SELECT status, error_message FROM {table}
+                            WHERE bank_id = $1
+                              AND result_metadata::jsonb @> $2::jsonb
+                            """,
+                            bank_id,
+                            json.dumps({"parent_operation_id": str(parent_id)}),
+                        )
+
+                        # A still-live child will drive the aggregation itself.
+                        if any(s["status"] not in ("completed", "failed") for s in siblings):
+                            continue
+
+                        if not siblings:
+                            await conn.execute(
+                                f"""
+                                UPDATE {table}
+                                SET status = 'failed', error_message = $2,
+                                    completed_at = now(), updated_at = now()
+                                WHERE operation_id = $1
+                                """,
+                                parent_id,
+                                "orphaned batch_retain parent: no child sub-batches were "
+                                "persisted (worker crashed mid-submission); resubmit the "
+                                "source documents",
+                            )
+                        elif any(s["status"] == "failed" for s in siblings):
+                            await conn.execute(
+                                f"""
+                                UPDATE {table}
+                                SET status = 'failed', error_message = $2, updated_at = now()
+                                WHERE operation_id = $1
+                                """,
+                                parent_id,
+                                _summarise_child_error_messages(siblings),
+                            )
+                        else:
+                            await conn.execute(
+                                f"""
+                                UPDATE {table}
+                                SET status = 'completed', completed_at = now(), updated_at = now()
+                                WHERE operation_id = $1
+                                """,
+                                parent_id,
+                            )
+                        reconciled += 1
+
+            if reconciled:
+                logger.warning(
+                    f"Worker {self._worker_id} reconciled {reconciled} stranded "
+                    f"batch_retain parent(s) in schema {schema_display}"
+                )
+        except Exception as e:
+            logger.warning(
+                f"Worker {self._worker_id} failed to reconcile orphaned parents in schema {schema_display}: {e}"
+            )
+        return reconciled
+
     async def run(self):
         """
         Main polling loop with fire-and-forget task execution.

hindsight-api-slim/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-api-slim"
-version = "0.8.4"
+version = "0.8.6"
 description = "Hindsight: Agent Memory That Works Like Human Memory"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -32,26 +32,37 @@ dependencies = [
     "PyJWT[crypto]>=2.8.0",
     "fastmcp>=3.2.0", # SSRF/path traversal, OAuth confused deputy, command injection fixes
     "python-dateutil>=2.8.0",
-    # opentelemetry-exporter-prometheus 0.62b1 calls
-    # MetricReader.__init__(otel_component_type=…), a kwarg added in
-    # opentelemetry-sdk 1.41.0 (open-telemetry/opentelemetry-python#4970).
-    # Without these floors, pip happily resolves a 0.62b1 exporter against
-    # an older sdk and metric initialisation crashes at startup with
-    # "MetricReader.__init__() got an unexpected keyword argument
-    # 'otel_component_type'".  Keep all six pins moving together.
-    "opentelemetry-api>=1.41.0",
-    "opentelemetry-sdk>=1.41.0",
-    "opentelemetry-instrumentation-fastapi>=0.62b1",
-    "opentelemetry-exporter-prometheus>=0.62b1",
-    "opentelemetry-exporter-otlp-proto-http>=1.41.0",
-    "opentelemetry-semantic-conventions>=0.62b1",
+    # Two coupled reasons for these floors — keep all six pins moving together:
+    # (1) opentelemetry-exporter-prometheus calls
+    #     MetricReader.__init__(otel_component_type=…), a kwarg added in
+    #     opentelemetry-sdk 1.41.0 (open-telemetry/opentelemetry-python#4970).
+    #     Without matched floors, pip resolves a new exporter against an older
+    #     sdk and metric initialisation crashes at startup with
+    #     "MetricReader.__init__() got an unexpected keyword argument
+    #     'otel_component_type'".
+    # (2) opentelemetry-proto <1.44 caps protobuf<7.0; 1.44.0 raised it to
+    #     protobuf<8.0, which is what lets the protobuf pin below reach 7.x.
+    "opentelemetry-api>=1.44.0",
+    "opentelemetry-sdk>=1.44.0",
+    "opentelemetry-instrumentation-fastapi>=0.65b0",
+    "opentelemetry-exporter-prometheus>=0.65b0",
+    "opentelemetry-exporter-otlp-proto-http>=1.44.0",
+    "opentelemetry-semantic-conventions>=0.65b0",
     "dateparser>=1.2.2",
     "google-genai>=1.0.0",
     "google-auth>=2.0.0",
     "anthropic>=0.40.0",
     "typer>=0.9.0",
     "cohere>=5.0.0",
-    "litellm>=1.84.0", # 1.82.7/1.82.8 had a supply chain compromise (yanked); 1.83.0+ also fixes GHSA-jjhc-v7c2-5hh6 / GHSA-53mr-6c8q-9789 / GHSA-pq44-5pcq-4r5g / GHSA-8cjq-wjmh-q42r; 1.84.0 fixes GHSA-4xpc-pv4p-pm3w
+    # 1.82.7/1.82.8 had a supply chain compromise (yanked); 1.83.0+ also fixes
+    # GHSA-jjhc-v7c2-5hh6 / GHSA-53mr-6c8q-9789 / GHSA-pq44-5pcq-4r5g /
+    # GHSA-8cjq-wjmh-q42r; 1.84.0 fixes GHSA-4xpc-pv4p-pm3w.
+    # Floor raised to 1.93.0 for Python 3.14: litellm ships its own Rust
+    # extension (litellm-rust python-bridge). Releases before 1.93.0 publish no
+    # cp314 wheel and their sdist fails to build because PyO3 0.23.5 rejects
+    # any interpreter newer than 3.13. 1.93.0 adds cp314 wheels and a PyO3 that
+    # builds on 3.14.
+    "litellm>=1.93.0",
     "markitdown[pdf,docx,pptx,xlsx,xls]>=0.1.4", # File to markdown conversion
     "obstore>=0.4.0", # S3/GCS/Azure object storage client (Rust-backed)
     "winloop>=0.1.0; sys_platform == 'win32'",
@@ -61,7 +72,7 @@ dependencies = [
     "urllib3>=2.7.0", # Decompression-bomb safeguards bypass + sensitive header forwarding fixes
     "langchain-core>=1.2.22", # Path traversal in legacy load_prompt functions fix
     "langsmith>=0.8.18", # GHSA-f4xh-w4cj-qxq8: arbitrary server-side file read in TracingMiddleware fix (supersedes >=0.6.3 SSRF tracing-header-injection floor)
-    "protobuf>=6.33.5", # JSON recursion depth bypass fix
+    "protobuf>=7.35.1", # JSON recursion depth bypass fix (>=6.33.5); requires otel>=1.44 (proto <1.44 caps protobuf<7.0)
     "pillow>=12.3.0", # Multiple HIGH image parsing vulnerabilities fixed in 12.3.0
     "cryptography>=48.0.1", # GHSA-537c-gmf6-5ccf: bundled-OpenSSL OOB read fix needs >=48.0.1. Prior <47 cap (47.0.0 SIGILL on ARM64 Docker/Podman, pyca/cryptography#14733) lifted — 47/48/49 verified importing + RSA sign/verify cleanly on linux/arm64 (Docker on Apple Silicon) and native arm64 macOS; upstream issue closed unconfirmed.
     "filelock>=3.20.1", # TOCTOU race condition fix

hindsight-api-slim/tests/conftest.py

@@ -127,7 +127,10 @@ def pytest_configure(config):
     # Look for .env in the workspace root (two levels up from tests dir)
     env_file = Path(__file__).parent.parent.parent / ".env"
     if env_file.exists():
-        load_dotenv(env_file)
+        # override=True keeps the workspace .env authoritative for the test
+        # session, matching the precedence hindsight_api used to apply at import
+        # time (removed in #2961 so library imports are side-effect-free).
+        load_dotenv(env_file, override=True)
     else:
         print(f"Warning: {env_file} not found, tests may fail without proper configuration")
 
@@ -415,8 +418,8 @@ async def oracle_memory(oracle_db_url, embeddings, cross_encoder, query_analyzer
     try:
         mem = MemoryEngine(
             db_url=oracle_db_url,
-            # Note: config.py loads ../.env with override=True, so these defaults
-            # only apply if no .env file is found. The .env file is authoritative.
+            # Note: conftest loads ../.env with override=True at session start, so
+            # these defaults only apply if no .env file is found. .env is authoritative.
             memory_llm_provider=os.getenv("HINDSIGHT_API_LLM_PROVIDER", "openai"),
             memory_llm_api_key=os.getenv("HINDSIGHT_API_LLM_API_KEY"),
             memory_llm_model=os.getenv("HINDSIGHT_API_LLM_MODEL", "gpt-4o-mini"),

hindsight-api-slim/tests/test_admin_backup_restore.py

@@ -10,6 +10,7 @@ import tempfile
 import uuid
 import zipfile
 from pathlib import Path
+from unittest.mock import AsyncMock
 
 import asyncpg
 import pytest
@@ -173,11 +174,12 @@ async def test_backup_restore_roundtrip(backup_test_schema):
         assert backup_path.stat().st_size > 0
 
         # Verify manifest
-        assert manifest["version"] == "1"
+        assert manifest["version"] == "2"
         assert "created_at" in manifest
         for table in BACKUP_TABLES:
             assert table in manifest["tables"]
             assert manifest["tables"][table]["rows"] == counts_before[table]
+            assert manifest["tables"][table]["columns"]
 
         # Verify zip contents
         with zipfile.ZipFile(backup_path, "r") as zf:
@@ -356,6 +358,186 @@ async def test_backup_restore_preserves_all_column_types(backup_test_schema):
             backup_path.unlink()
 
 
+@pytest.mark.asyncio
+async def test_restore_rejects_legacy_extra_column_before_truncating(backup_test_schema):
+    """Schema drift must fail preflight without deleting existing target data."""
+    db_url, schema_name, _fq, _embeddings = backup_test_schema
+    conn = await asyncpg.connect(db_url)
+    try:
+        await conn.execute(f"ALTER TABLE {_fq('documents')} ADD COLUMN legacy_metadata JSONB")
+        await conn.execute(f"INSERT INTO {_fq('banks')} (bank_id) VALUES ('source-bank')")
+    finally:
+        await conn.close()
+
+    with tempfile.NamedTemporaryFile(suffix=".zip", delete=False) as f:
+        backup_path = Path(f.name)
+
+    try:
+        await _backup(db_url, backup_path, schema=schema_name)
+        conn = await asyncpg.connect(db_url)
+        try:
+            await conn.execute(f"ALTER TABLE {_fq('documents')} DROP COLUMN legacy_metadata")
+            await conn.execute(f"INSERT INTO {_fq('banks')} (bank_id) VALUES ('target-bank')")
+        finally:
+            await conn.close()
+
+        with pytest.raises(ValueError, match=r"documents: target is missing backup columns legacy_metadata"):
+            await _restore(db_url, backup_path, schema=schema_name)
+
+        conn = await asyncpg.connect(db_url)
+        try:
+            banks = await conn.fetch(f"SELECT bank_id FROM {_fq('banks')} ORDER BY bank_id")
+        finally:
+            await conn.close()
+        assert [row["bank_id"] for row in banks] == ["source-bank", "target-bank"]
+    finally:
+        if backup_path.exists():
+            backup_path.unlink()
+
+
+@pytest.mark.asyncio
+async def test_restore_rejects_incompatible_column_type_before_truncating(backup_test_schema):
+    """A column present in both schemas but with a different type must fail preflight."""
+    db_url, schema_name, _fq, _embeddings = backup_test_schema
+    conn = await asyncpg.connect(db_url)
+    try:
+        await conn.execute(f"ALTER TABLE {_fq('documents')} ADD COLUMN drift_col INTEGER")
+        await conn.execute(f"INSERT INTO {_fq('banks')} (bank_id) VALUES ('source-bank')")
+    finally:
+        await conn.close()
+
+    with tempfile.NamedTemporaryFile(suffix=".zip", delete=False) as f:
+        backup_path = Path(f.name)
+
+    try:
+        await _backup(db_url, backup_path, schema=schema_name)
+        conn = await asyncpg.connect(db_url)
+        try:
+            # Same column name, incompatible type in the target.
+            await conn.execute(f"ALTER TABLE {_fq('documents')} DROP COLUMN drift_col")
+            await conn.execute(f"ALTER TABLE {_fq('documents')} ADD COLUMN drift_col TEXT")
+            await conn.execute(f"INSERT INTO {_fq('banks')} (bank_id) VALUES ('target-bank')")
+        finally:
+            await conn.close()
+
+        with pytest.raises(ValueError, match=r"documents: incompatible column types: drift_col"):
+            await _restore(db_url, backup_path, schema=schema_name)
+
+        conn = await asyncpg.connect(db_url)
+        try:
+            banks = await conn.fetch(f"SELECT bank_id FROM {_fq('banks')} ORDER BY bank_id")
+        finally:
+            await conn.close()
+        assert [row["bank_id"] for row in banks] == ["source-bank", "target-bank"]
+    finally:
+        if backup_path.exists():
+            backup_path.unlink()
+
+
+@pytest.mark.asyncio
+async def test_restore_succeeds_when_target_has_additional_nullable_column(backup_test_schema):
+    """A target with an extra nullable column not in the backup restores cleanly.
+
+    Binary COPY without an explicit column list would fail this with a field-count
+    error; pinning the source column list is what lets these restores succeed.
+    """
+    db_url, schema_name, _fq, _embeddings = backup_test_schema
+    conn = await asyncpg.connect(db_url)
+    try:
+        await conn.execute(f"INSERT INTO {_fq('banks')} (bank_id) VALUES ('roundtrip-bank')")
+    finally:
+        await conn.close()
+
+    with tempfile.NamedTemporaryFile(suffix=".zip", delete=False) as f:
+        backup_path = Path(f.name)
+
+    try:
+        await _backup(db_url, backup_path, schema=schema_name)
+        conn = await asyncpg.connect(db_url)
+        try:
+            await conn.execute(f"ALTER TABLE {_fq('banks')} ADD COLUMN target_only_note TEXT")
+        finally:
+            await conn.close()
+
+        # The extra target column is not in the backup, so validation passes and
+        # restore copies only the source columns; the new column stays NULL.
+        await _restore(db_url, backup_path, schema=schema_name)
+
+        conn = await asyncpg.connect(db_url)
+        try:
+            rows = await conn.fetch(f"SELECT bank_id, target_only_note FROM {_fq('banks')} ORDER BY bank_id")
+        finally:
+            await conn.close()
+        assert [row["bank_id"] for row in rows] == ["roundtrip-bank"]
+        assert rows[0]["target_only_note"] is None
+    finally:
+        if backup_path.exists():
+            backup_path.unlink()
+
+
+@pytest.mark.asyncio
+async def test_backup_restore_includes_extension_table(backup_test_schema):
+    """An extension-declared bank-scoped table rides along backup + restore.
+
+    Simulates a table an extension provisions in the tenant schema (core knows
+    nothing about it). Passing the augmented ``backup_tables`` list — as
+    ``_effective_backup_tables()`` builds from ``TenantExtension.extra_bank_tables``
+    — must back it up AND restore it, so restore's ``TRUNCATE ... CASCADE`` can't
+    silently drop it.
+    """
+    db_url, schema_name, _fq, _embeddings = backup_test_schema
+    extra = "ext_audit_receipts"
+    effective = [*BACKUP_TABLES, extra]
+
+    conn = await asyncpg.connect(db_url)
+    try:
+        await conn.execute(f"CREATE TABLE {_fq(extra)} (id uuid PRIMARY KEY, bank_id text NOT NULL, payload text)")
+        kept_id = uuid.uuid4()
+        await conn.execute(
+            f"INSERT INTO {_fq(extra)} (id, bank_id, payload) VALUES ($1, $2, $3)",
+            kept_id,
+            "bank-x",
+            "original receipt",
+        )
+    finally:
+        await conn.close()
+
+    with tempfile.NamedTemporaryFile(suffix=".zip", delete=False) as f:
+        backup_path = Path(f.name)
+
+    try:
+        manifest = await _backup(db_url, backup_path, schema=schema_name, backup_tables=effective)
+        assert manifest["tables"][extra]["rows"] == 1
+
+        # Mutate after backup: a row that must NOT survive restore.
+        conn = await asyncpg.connect(db_url)
+        try:
+            await conn.execute(
+                f"INSERT INTO {_fq(extra)} (id, bank_id, payload) VALUES ($1, $2, $3)",
+                uuid.uuid4(),
+                "bank-x",
+                "post-backup row",
+            )
+        finally:
+            await conn.close()
+
+        await _restore(db_url, backup_path, schema=schema_name, backup_tables=effective)
+
+        conn = await asyncpg.connect(db_url)
+        try:
+            rows = await conn.fetch(f"SELECT id, payload FROM {_fq(extra)}")
+        finally:
+            await conn.close()
+
+        # Restore reset the table to exactly its backed-up contents.
+        assert len(rows) == 1
+        assert rows[0]["id"] == kept_id
+        assert rows[0]["payload"] == "original receipt"
+    finally:
+        if backup_path.exists():
+            backup_path.unlink()
+
+
 @pytest.mark.asyncio
 async def test_run_migration_without_schema_discovers_and_deduplicates_schemas(monkeypatch):
     """run-db-migration without --schema should include the base schema and deduplicate tenant schemas."""
@@ -397,6 +579,8 @@ async def test_run_migration_without_schema_discovers_and_deduplicates_schemas(m
     monkeypatch.setenv("HINDSIGHT_API_DATABASE_URL", "postgresql://test")
     monkeypatch.setattr(admin_cli, "load_extension", lambda *args, **kwargs: MockTenantExtension())
     monkeypatch.setattr(admin_cli, "resolve_database_url", fake_resolve_database_url)
+    # Extension-table provisioning does a real connect; these tests mock the DB, so stub it.
+    monkeypatch.setattr(admin_cli, "_provision_extra_bank_tables", AsyncMock())
 
     from hindsight_api import migrations as migrations_module
 
@@ -467,6 +651,8 @@ async def test_run_migration_without_schema_runs_optional_post_migration_hooks(m
 
     monkeypatch.setattr(admin_cli, "load_extension", lambda *args, **kwargs: MockTenantExtension())
     monkeypatch.setattr(admin_cli, "resolve_database_url", fake_resolve_database_url)
+    # Extension-table provisioning does a real connect; these tests mock the DB, so stub it.
+    monkeypatch.setattr(admin_cli, "_provision_extra_bank_tables", AsyncMock())
 
     from hindsight_api import migrations as migrations_module
 
@@ -537,6 +723,8 @@ async def test_run_migration_with_schema_only_runs_requested_schema(monkeypatch)
 
     monkeypatch.setattr(admin_cli, "load_extension", lambda *args, **kwargs: MockTenantExtension())
     monkeypatch.setattr(admin_cli, "resolve_database_url", fake_resolve_database_url)
+    # Extension-table provisioning does a real connect; these tests mock the DB, so stub it.
+    monkeypatch.setattr(admin_cli, "_provision_extra_bank_tables", AsyncMock())
 
     from hindsight_api import migrations as migrations_module
 
@@ -575,6 +763,8 @@ async def test_run_migration_threads_ensure_extensions_flag(monkeypatch, ensure_
 
     monkeypatch.setattr(admin_cli, "load_extension", lambda *args, **kwargs: None)
     monkeypatch.setattr(admin_cli, "resolve_database_url", fake_resolve_database_url)
+    # Extension-table provisioning does a real connect; these tests mock the DB, so stub it.
+    monkeypatch.setattr(admin_cli, "_provision_extra_bank_tables", AsyncMock())
 
     from hindsight_api import migrations as migrations_module
 

hindsight-api-slim/tests/test_async_batch_retain.py

@@ -1432,3 +1432,110 @@ async def test_submit_async_batch_retain_rolls_back_missing_bank_on_child_failur
 
     bank = await pool.fetchrow("SELECT bank_id FROM banks WHERE bank_id = $1", bank_id)
     assert bank is None, "the lazily-created bank must roll back with the failed operation inserts"
+
+
+# ── retrying a batch_retain parent re-runs its outstanding children ─────────
+# Regression for #2985's retry guard, which rejected *every* payload-null
+# batch_retain parent — that made `retain --async` operations un-retryable
+# (their operation_id is always such a parent) and 409'd the operations doc
+# example. Retry now re-queues the parent's failed/cancelled children and
+# revives the parent, without touching in-flight children or re-stranding a
+# parent whose work is already done.
+
+
+async def _insert_parent(conn, bank_id: str, parent_id, status: str) -> None:
+    await conn.execute(
+        "INSERT INTO banks (bank_id, name) VALUES ($1, $1) ON CONFLICT DO NOTHING",
+        bank_id,
+    )
+    await conn.execute(
+        "INSERT INTO async_operations (operation_id, bank_id, operation_type, status, result_metadata) "
+        "VALUES ($1, $2, 'batch_retain', $3, $4::jsonb)",
+        parent_id,
+        bank_id,
+        status,
+        json.dumps({"is_parent": True}),
+    )
+
+
+async def _insert_child(conn, bank_id: str, child_id, parent_id, status: str) -> None:
+    await conn.execute(
+        "INSERT INTO async_operations "
+        "(operation_id, bank_id, operation_type, status, task_payload, result_metadata) "
+        "VALUES ($1, $2, 'retain', $3, $4::jsonb, $5::jsonb)",
+        child_id,
+        bank_id,
+        status,
+        json.dumps({"contents": []}),
+        json.dumps({"parent_operation_id": str(parent_id)}),
+    )
+
+
+@pytest.mark.asyncio
+async def test_retry_batch_parent_requeues_failed_child_and_revives_parent(memory, request_context):
+    from hindsight_api.engine.db_utils import acquire_with_retry
+
+    bank_id = f"retry-parent-{uuid.uuid4().hex[:8]}"
+    parent_id, child_id = uuid.uuid4(), uuid.uuid4()
+    backend = await memory._get_backend()
+    async with acquire_with_retry(backend) as conn:
+        await _insert_parent(conn, bank_id, parent_id, "cancelled")
+        await _insert_child(conn, bank_id, child_id, parent_id, "failed")
+
+    result = await memory.retry_operation(bank_id, str(parent_id), request_context=request_context)
+    assert result["success"] is True
+
+    async with acquire_with_retry(backend) as conn:
+        rows = {
+            r["operation_id"]: r["status"]
+            for r in await conn.fetch("SELECT operation_id, status FROM async_operations WHERE bank_id = $1", bank_id)
+        }
+    assert rows[child_id] == "pending", "the failed child must be re-queued"
+    assert rows[parent_id] == "pending", "the parent must be revived so it re-aggregates"
+
+
+@pytest.mark.asyncio
+async def test_retry_batch_parent_leaves_processing_child_untouched(memory, request_context):
+    # A 'processing' child is owned by a live worker; resetting it would let a
+    # second worker race it on the same document_id (#1795). Retry must revive
+    # the parent (the processing child is non-completed) but not touch the child.
+    from hindsight_api.engine.db_utils import acquire_with_retry
+
+    bank_id = f"retry-parent-{uuid.uuid4().hex[:8]}"
+    parent_id, child_id = uuid.uuid4(), uuid.uuid4()
+    backend = await memory._get_backend()
+    async with acquire_with_retry(backend) as conn:
+        await _insert_parent(conn, bank_id, parent_id, "cancelled")
+        await _insert_child(conn, bank_id, child_id, parent_id, "processing")
+
+    result = await memory.retry_operation(bank_id, str(parent_id), request_context=request_context)
+    assert result["success"] is True
+
+    async with acquire_with_retry(backend) as conn:
+        rows = {
+            r["operation_id"]: r["status"]
+            for r in await conn.fetch("SELECT operation_id, status FROM async_operations WHERE bank_id = $1", bank_id)
+        }
+    assert rows[child_id] == "processing", "an in-flight child must not be reset"
+    assert rows[parent_id] == "pending"
+
+
+@pytest.mark.asyncio
+async def test_retry_batch_parent_all_children_completed_is_rejected(memory, request_context):
+    from hindsight_api.engine.db_utils import acquire_with_retry
+    from hindsight_api.extensions import OperationValidationError
+
+    bank_id = f"retry-parent-{uuid.uuid4().hex[:8]}"
+    parent_id, child_id = uuid.uuid4(), uuid.uuid4()
+    backend = await memory._get_backend()
+    async with acquire_with_retry(backend) as conn:
+        await _insert_parent(conn, bank_id, parent_id, "cancelled")
+        await _insert_child(conn, bank_id, child_id, parent_id, "completed")
+
+    with pytest.raises(OperationValidationError) as exc:
+        await memory.retry_operation(bank_id, str(parent_id), request_context=request_context)
+    assert exc.value.status_code == 409
+
+    async with acquire_with_retry(backend) as conn:
+        parent = await conn.fetchrow("SELECT status FROM async_operations WHERE operation_id = $1", parent_id)
+    assert parent["status"] == "cancelled", "a parent with no retryable work must not be revived (no re-strand)"

hindsight-api-slim/tests/test_async_retain_operation_id.py

@@ -0,0 +1,206 @@
+"""Idempotent async retain via a caller-supplied operation_id.
+
+A client that retries an async retain after a lost or timed-out acknowledgement
+must not enqueue a second parent operation. Supplying the same ``operation_id``
+returns the original operation and creates no new work; the parent primary key
+is the concurrency authority, so no extra schema is involved.
+"""
+
+import asyncio
+import uuid
+
+import httpx
+import pytest
+import pytest_asyncio
+
+from hindsight_api.api import create_app
+from hindsight_api.engine.memory_engine import RetainOperationConflictError
+
+# These tests submit async operations against the shared pool. Share the
+# "worker_tests" xdist group with test_async_batch_retain / test_worker so a
+# concurrently running poller cannot steal each other's pending rows.
+pytestmark = pytest.mark.xdist_group("worker_tests")
+
+
+@pytest_asyncio.fixture
+async def pool(pg0_db_url):
+    import asyncpg
+
+    from hindsight_api.pg0 import resolve_database_url
+
+    resolved_url = await resolve_database_url(pg0_db_url)
+    p = await asyncpg.create_pool(resolved_url, min_size=1, max_size=5, command_timeout=30)
+    yield p
+    await p.close()
+
+
+@pytest_asyncio.fixture
+async def api_client(memory):
+    app = create_app(memory, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        yield client
+
+
+async def _count_batch_parents(pool, bank_id: str) -> int:
+    return await pool.fetchval(
+        "SELECT count(*) FROM async_operations WHERE bank_id = $1 AND operation_type = 'batch_retain'",
+        bank_id,
+    )
+
+
+# --------------------------------------------------------------------------- #
+# Engine-level behaviour
+# --------------------------------------------------------------------------- #
+
+
+@pytest.mark.asyncio
+async def test_replay_returns_same_operation_and_no_new_work(memory, request_context, pool):
+    """Re-submitting with the same operation_id returns the original, creating no new op."""
+    bank_id = "test_retain_idem_replay"
+    operation_id = str(uuid.uuid4())
+    contents = [{"content": "Alice works at Google", "document_id": "doc1"}]
+
+    first = await memory.submit_async_retain(
+        bank_id=bank_id,
+        contents=contents,
+        request_context=request_context,
+        operation_id=operation_id,
+    )
+    await asyncio.sleep(0.1)
+    second = await memory.submit_async_retain(
+        bank_id=bank_id,
+        contents=contents,
+        request_context=request_context,
+        operation_id=operation_id,
+    )
+
+    assert first["operation_id"] == operation_id
+    assert second["operation_id"] == operation_id
+    assert second["items_count"] == 1
+    # Exactly one parent operation exists — the retry created no duplicate.
+    assert await _count_batch_parents(pool, bank_id) == 1
+
+
+@pytest.mark.asyncio
+async def test_no_operation_id_creates_distinct_operations(memory, request_context, pool):
+    """Omitting operation_id keeps the legacy create-each-time behaviour."""
+    bank_id = "test_retain_idem_legacy"
+    contents = [{"content": "Bob went hiking", "document_id": "doc1"}]
+
+    first = await memory.submit_async_retain(bank_id=bank_id, contents=contents, request_context=request_context)
+    second = await memory.submit_async_retain(bank_id=bank_id, contents=contents, request_context=request_context)
+
+    assert first["operation_id"] != second["operation_id"]
+    assert await _count_batch_parents(pool, bank_id) == 2
+
+
+@pytest.mark.asyncio
+async def test_conflict_when_id_belongs_to_different_bank(memory, request_context):
+    """An operation_id owned by another bank cannot be reused (global PK collision)."""
+    operation_id = str(uuid.uuid4())
+    contents = [{"content": "Shared id content", "document_id": "doc1"}]
+
+    await memory.submit_async_retain(
+        bank_id="test_retain_idem_bankA",
+        contents=contents,
+        request_context=request_context,
+        operation_id=operation_id,
+    )
+
+    with pytest.raises(RetainOperationConflictError):
+        await memory.submit_async_retain(
+            bank_id="test_retain_idem_bankB",
+            contents=contents,
+            request_context=request_context,
+            operation_id=operation_id,
+        )
+
+
+@pytest.mark.asyncio
+async def test_replay_ignores_payload_differences(memory, request_context, pool):
+    """A replay resolves purely by id; a differing payload still returns the original.
+
+    Reusing an id you generated for different content is a client bug, and
+    returning the original operation is the safe idempotent answer — no new
+    work is enqueued.
+    """
+    bank_id = "test_retain_idem_payload"
+    operation_id = str(uuid.uuid4())
+
+    first = await memory.submit_async_retain(
+        bank_id=bank_id,
+        contents=[{"content": "original", "document_id": "doc1"}],
+        request_context=request_context,
+        operation_id=operation_id,
+    )
+    await asyncio.sleep(0.1)
+    second = await memory.submit_async_retain(
+        bank_id=bank_id,
+        contents=[{"content": "totally different", "document_id": "doc2"}],
+        request_context=request_context,
+        operation_id=operation_id,
+    )
+
+    assert second["operation_id"] == first["operation_id"]
+    assert await _count_batch_parents(pool, bank_id) == 1
+
+
+@pytest.mark.asyncio
+async def test_concurrent_submissions_with_same_id_create_one_operation(memory, request_context, pool):
+    """Two simultaneous first submissions of the same id resolve to one operation."""
+    bank_id = "test_retain_idem_concurrent"
+    operation_id = str(uuid.uuid4())
+    contents = [{"content": "Concurrent content", "document_id": "doc1"}]
+
+    async def submit():
+        return await memory.submit_async_retain(
+            bank_id=bank_id,
+            contents=contents,
+            request_context=request_context,
+            operation_id=operation_id,
+        )
+
+    results = await asyncio.gather(submit(), submit())
+
+    assert {r["operation_id"] for r in results} == {operation_id}
+    assert await _count_batch_parents(pool, bank_id) == 1
+
+
+# --------------------------------------------------------------------------- #
+# HTTP-level validation
+# --------------------------------------------------------------------------- #
+
+
+@pytest.mark.asyncio
+async def test_http_invalid_operation_id_is_rejected(api_client):
+    bank_id = "test_retain_idem_http_invalid"
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [{"content": "hello", "document_id": "doc1"}],
+            "async": True,
+            "operation_id": "not-a-uuid",
+        },
+    )
+    assert response.status_code == 422
+
+
+@pytest.mark.asyncio
+async def test_http_replay_returns_same_operation_id(api_client):
+    bank_id = "test_retain_idem_http_replay"
+    operation_id = str(uuid.uuid4())
+    payload = {
+        "items": [{"content": "Alice works at Google", "document_id": "doc1"}],
+        "async": True,
+        "operation_id": operation_id,
+    }
+
+    first = await api_client.post(f"/v1/default/banks/{bank_id}/memories", json=payload)
+    assert first.status_code == 200
+    await asyncio.sleep(0.1)
+    second = await api_client.post(f"/v1/default/banks/{bank_id}/memories", json=payload)
+    assert second.status_code == 200
+
+    assert first.json()["operation_id"] == operation_id
+    assert second.json()["operation_id"] == operation_id

hindsight-api-slim/tests/test_audit_per_bank.py

@@ -15,7 +15,7 @@ import pytest
 import pytest_asyncio
 
 from hindsight_api.api import create_app
-from hindsight_api.engine.audit import AuditLogger
+from hindsight_api.engine.audit import AuditEntry, AuditLogger
 from tests.conftest import enable_audit_default
 
 # Audit writes are fire-and-forget; give the background task room to land.
@@ -114,6 +114,66 @@ async def test_no_override_uses_deployment_default(client, memory):
     assert "recall" in await _audited_actions(client, bank_id)
 
 
+# ── AuditLogger write: dialect-aware table qualification ───────────────────
+
+
+class _CapturingConn:
+    def __init__(self, sink: list[str]) -> None:
+        self._sink = sink
+
+    async def execute(self, sql: str, *args: object) -> None:
+        self._sink.append(sql)
+
+
+class _CapturingPool:
+    """Minimal asyncpg.Pool-like stand-in that records the SQL it executes."""
+
+    def __init__(self, sink: list[str]) -> None:
+        self._sink = sink
+
+    async def acquire(self) -> _CapturingConn:
+        return _CapturingConn(self._sink)
+
+    async def release(self, conn: _CapturingConn) -> None:
+        pass
+
+    def get_size(self) -> int:
+        return 1
+
+    def get_idle_size(self) -> int:
+        return 1
+
+
+async def _capture_audit_write(monkeypatch, *, oracle: bool, schema: str) -> str:
+    monkeypatch.setattr("hindsight_api.engine.schema._is_oracle", lambda: oracle)
+    sink: list[str] = []
+    logger = AuditLogger(
+        pool_getter=lambda: _CapturingPool(sink),
+        schema_getter=lambda: schema,
+        enabled=True,
+        allowed_actions=[],
+    )
+    await logger._safe_log(AuditEntry(action="recall", transport="http", bank_id="b1"))
+    assert len(sink) == 1, "expected exactly one INSERT"
+    return sink[0]
+
+
+@pytest.mark.asyncio
+async def test_audit_write_uses_bare_table_on_oracle(monkeypatch):
+    # audit_log exists on Oracle, but a raw f"{schema}.audit_log" yields
+    # public.audit_log — "public" is a reserved word there, so every write failed
+    # with ORA-00903. The dialect-qualified (bare) name must be used instead.
+    sql = await _capture_audit_write(monkeypatch, oracle=True, schema="public")
+    assert "INSERT INTO audit_log" in sql
+    assert "public.audit_log" not in sql
+
+
+@pytest.mark.asyncio
+async def test_audit_write_qualifies_schema_on_postgres(monkeypatch):
+    sql = await _capture_audit_write(monkeypatch, oracle=False, schema="tenant_x")
+    assert '"tenant_x".audit_log' in sql
+
+
 # ── AuditLogger decision logic (no DB) ─────────────────────────────────────
 
 

hindsight-api-slim/tests/test_bank_lifecycle_deadlock_retry.py

@@ -0,0 +1,75 @@
+"""Per-bank vector index DDL on create/delete must survive a transient deadlock.
+
+The test-api shard runs 8 pytest-xdist workers against one shared pg0 database
+(``public`` schema), so every bank create/delete does index DDL on the same
+``memory_units`` table other workers are writing. A fresh bank builds its
+partial indexes with a plain ``CREATE INDEX`` inside the bank-create tx
+(ShareLock), and ``delete_bank`` drops them (CONCURRENTLY, post-commit); both
+can be chosen as a deadlock victim. These are the exact production paths that
+flaked in CI — they must retry the transient deadlock, not surface it.
+
+The deadlock is injected via monkeypatch (one-shot ``DeadlockDetectedError``)
+so the retry path is exercised deterministically, without racing real workers.
+"""
+
+import uuid
+
+import pytest
+from asyncpg.exceptions import DeadlockDetectedError
+
+from hindsight_api import RequestContext
+from hindsight_api.engine.memory_engine import MemoryEngine
+from hindsight_api.engine.retain import bank_utils
+
+
+@pytest.mark.asyncio
+async def test_bank_create_retries_transient_deadlock(
+    memory: MemoryEngine, request_context: RequestContext, monkeypatch
+):
+    """A deadlock during the lazy bank-create tx retries the whole tx."""
+    backend = await memory._get_backend()
+    real = bank_utils.get_or_create_bank_profile_on_conn
+    calls = 0
+
+    async def flaky(conn, bank_id, *, ops):
+        nonlocal calls
+        calls += 1
+        if calls == 1:
+            raise DeadlockDetectedError("deadlock detected")
+        return await real(conn, bank_id, ops=ops)
+
+    monkeypatch.setattr(bank_utils, "get_or_create_bank_profile_on_conn", flaky)
+
+    bank_id = f"test-deadlock-{uuid.uuid4().hex[:8]}"
+    try:
+        result = await bank_utils.get_or_create_bank_profile(backend, bank_id)
+        assert calls == 2, "expected exactly one retry after the injected deadlock"
+        assert result.created is True
+    finally:
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+
+@pytest.mark.asyncio
+async def test_bank_delete_retries_transient_deadlock(
+    memory: MemoryEngine, request_context: RequestContext, monkeypatch
+):
+    """A deadlock while dropping per-bank indexes on delete is retried."""
+    bank_id = f"test-deadlock-{uuid.uuid4().hex[:8]}"
+    await memory.get_bank_profile(bank_id=bank_id, request_context=request_context)
+
+    backend = await memory._get_backend()
+    real = backend.ops.drop_bank_vector_indexes
+    calls = 0
+
+    async def flaky(*args, **kwargs):
+        nonlocal calls
+        calls += 1
+        if calls == 1:
+            raise DeadlockDetectedError("deadlock detected")
+        return await real(*args, **kwargs)
+
+    monkeypatch.setattr(backend.ops, "drop_bank_vector_indexes", flaky)
+
+    result = await memory.delete_bank(bank_id, request_context=request_context)
+    assert calls == 2, "expected exactly one retry after the injected deadlock"
+    assert result["bank_deleted"] is True

hindsight-api-slim/tests/test_bank_template_configurable_fields.py

@@ -50,6 +50,8 @@ NEW_FIELDS: list[tuple[str, object]] = [
     ("recall_budget_adaptive_high", 0.4),
     ("recall_budget_min", 30),
     ("recall_budget_max", 1500),
+    ("audit_log_enabled", True),
+    ("store_document_text", False),
 ]
 
 

hindsight-api-slim/tests/test_bank_templates.py

@@ -1,11 +1,14 @@
 """Integration tests for bank template import/export endpoints."""
 
+from datetime import datetime
+
+import httpx
 import pytest
 import pytest_asyncio
-import httpx
-from datetime import datetime
+
 from hindsight_api.api import create_app
 from hindsight_api.api.http import BankTemplateManifest, validate_bank_template
+from hindsight_api.models import RequestContext
 
 
 @pytest_asyncio.fixture
@@ -682,6 +685,83 @@ class TestDefaultBankTemplateEnvVar:
         names = [d["name"] for d in dir_resp.json()["items"]]
         assert "Default Env Directive" in names
 
+    @pytest.mark.asyncio
+    async def test_import_updates_resources_provisioned_by_default_template(
+        self,
+        api_client,
+        bank_id,
+        _patched_default_template,
+    ):
+        """Import authorization projects default-template resources on a missing bank."""
+        response = await api_client.post(
+            f"/v1/default/banks/{bank_id}/import",
+            json={
+                "version": "1",
+                "mental_models": [
+                    {
+                        "id": "default-env-model",
+                        "name": "Imported Model",
+                        "source_query": "What did the import request?",
+                    }
+                ],
+                "directives": [
+                    {
+                        "name": "Default Env Directive",
+                        "content": "Follow the imported behavior.",
+                        "priority": 9,
+                    }
+                ],
+            },
+        )
+
+        assert response.status_code == 200, response.text
+        body = response.json()
+        assert body["mental_models_updated"] == ["default-env-model"]
+        assert body["mental_models_created"] == []
+        assert body["directives_updated"] == ["Default Env Directive"]
+        assert body["directives_created"] == []
+
+    @pytest.mark.asyncio
+    async def test_import_validates_config_against_projected_default_template(
+        self,
+        api_client,
+        memory,
+        bank_id,
+        monkeypatch,
+    ):
+        """A client config rejected against the projected defaults leaves no bank."""
+        from hindsight_api.config import _get_raw_config
+
+        raw = _get_raw_config()
+        monkeypatch.setattr(
+            raw,
+            "default_bank_template",
+            {
+                "version": "1",
+                "bank": {"retain_chunk_size": raw.retain_max_completion_tokens},
+            },
+        )
+
+        response = await api_client.post(
+            f"/v1/default/banks/{bank_id}/import",
+            json={
+                "version": "1",
+                "bank": {
+                    "retain_strategies": {
+                        "projected-default": {"retain_extraction_mode": "concise"},
+                    }
+                },
+            },
+        )
+
+        assert response.status_code == 400, response.text
+        profile = await memory.get_bank_profile(
+            bank_id,
+            request_context=RequestContext(),
+            create_if_missing=False,
+        )
+        assert profile is None
+
     @pytest.mark.asyncio
     async def test_default_template_overrides_env_config_defaults(
         self, api_client, bank_id, monkeypatch, default_template

hindsight-api-slim/tests/test_causal_link_taxonomy.py

@@ -8,6 +8,7 @@ from hindsight_api.engine.causal_links import (
     CANONICAL_CAUSAL_LINK_TYPES,
     CAUSAL_LINK_TYPES,
     LEGACY_CAUSAL_LINK_TYPES,
+    CausalLinkDescriptor,
 )
 from hindsight_api.engine.retain.fact_extraction import CausalRelation, FactCausalRelation
 
@@ -32,3 +33,42 @@ def test_causal_link_taxonomy_keeps_canonical_and_legacy_types_separate() -> Non
     assert CANONICAL_CAUSAL_LINK_TYPES == {CANONICAL_CAUSAL_LINK_TYPE}
     assert LEGACY_CAUSAL_LINK_TYPES == {"causes", "enables", "prevents"}
     assert CAUSAL_LINK_TYPES == (CANONICAL_CAUSAL_LINK_TYPE, "causes", "enables", "prevents")
+
+
+def test_causal_link_descriptor_round_trips_through_json() -> None:
+    """The archive stores descriptors as plain JSON; the round-trip must be lossless."""
+    descriptor = CausalLinkDescriptor(
+        from_unit_id="11111111-1111-1111-1111-111111111111",
+        to_unit_id="22222222-2222-2222-2222-222222222222",
+        link_type="caused_by",
+        weight=0.75,
+    )
+    assert CausalLinkDescriptor.from_json_dict(descriptor.as_json_dict()) == descriptor
+
+
+def test_causal_link_descriptor_defaults_missing_weight() -> None:
+    parsed = CausalLinkDescriptor.from_json_dict(
+        {
+            "from_unit_id": "11111111-1111-1111-1111-111111111111",
+            "to_unit_id": "22222222-2222-2222-2222-222222222222",
+            "link_type": "enables",
+        }
+    )
+    assert parsed is not None
+    assert parsed.weight == 1.0
+
+
+@pytest.mark.parametrize(
+    "raw",
+    [
+        "not-a-dict",
+        {"from_unit_id": None, "to_unit_id": "2", "link_type": "caused_by"},
+        {"from_unit_id": "1", "to_unit_id": None, "link_type": "caused_by"},
+        # 'temporal' is derived data, not a causal edge — and memory_links has a
+        # link_type CHECK constraint, so an unusable entry must not abort a revert.
+        {"from_unit_id": "1", "to_unit_id": "2", "link_type": "temporal"},
+        {"from_unit_id": "1", "to_unit_id": "2"},
+    ],
+)
+def test_causal_link_descriptor_skips_unusable_entries(raw) -> None:
+    assert CausalLinkDescriptor.from_json_dict(raw) is None

hindsight-api-slim/tests/test_causal_relation_offsets.py

@@ -0,0 +1,164 @@
+import json
+from types import SimpleNamespace
+
+import pytest
+
+from hindsight_api.engine.response_models import TokenUsage
+from hindsight_api.engine.retain import fact_extraction
+from hindsight_api.engine.retain.fact_extraction import CausalRelation, Fact
+from hindsight_api.engine.retain.types import RetainContent
+
+
+@pytest.mark.asyncio
+async def test_causal_targets_are_offset_from_extraction_group_start(monkeypatch):
+    async def extract_facts_from_text(*, text, **_kwargs):
+        if text == "preceding":
+            facts = [Fact(fact=f"prior {index}", fact_type="world") for index in range(4)]
+            return facts, [(text, len(facts))], TokenUsage()
+
+        facts = [
+            Fact(fact="cause", fact_type="world"),
+            Fact(
+                fact="effect",
+                fact_type="world",
+                causal_relations=[CausalRelation(target_fact_index=0, relation_type="caused_by")],
+            ),
+        ]
+        return facts, [(text, len(facts))], TokenUsage()
+
+    monkeypatch.setattr(fact_extraction, "extract_facts_from_text", extract_facts_from_text)
+    monkeypatch.setattr(fact_extraction, "_add_temporal_offsets", lambda *_args: None)
+    monkeypatch.setattr(fact_extraction, "_inject_label_tags", lambda *_args: None)
+
+    config = SimpleNamespace(retain_extraction_mode="normal", retain_batch_enabled=False)
+    facts, _, _ = await fact_extraction.extract_facts_from_contents(
+        [RetainContent(content="preceding"), RetainContent(content="causal group")],
+        llm_config=None,
+        agent_name="test",
+        config=config,
+    )
+
+    assert facts[5].causal_relations[0].target_fact_index == 4
+
+
+@pytest.mark.asyncio
+async def test_each_chunk_uses_its_own_causal_index_base(monkeypatch):
+    async def extract_facts_from_text(**_kwargs):
+        facts = [
+            Fact(fact="first cause", fact_type="world"),
+            Fact(
+                fact="first effect",
+                fact_type="world",
+                causal_relations=[CausalRelation(target_fact_index=0, relation_type="caused_by")],
+            ),
+            Fact(fact="second cause", fact_type="world"),
+            Fact(
+                fact="second effect",
+                fact_type="world",
+                causal_relations=[CausalRelation(target_fact_index=0, relation_type="caused_by")],
+            ),
+        ]
+        return facts, [("first", 2), ("second", 2)], TokenUsage()
+
+    monkeypatch.setattr(fact_extraction, "extract_facts_from_text", extract_facts_from_text)
+    monkeypatch.setattr(fact_extraction, "_add_temporal_offsets", lambda *_args: None)
+    monkeypatch.setattr(fact_extraction, "_inject_label_tags", lambda *_args: None)
+
+    config = SimpleNamespace(retain_extraction_mode="normal", retain_batch_enabled=False)
+    facts, _, _ = await fact_extraction.extract_facts_from_contents(
+        [RetainContent(content="two chunks")],
+        llm_config=None,
+        agent_name="test",
+        config=config,
+    )
+
+    assert facts[1].causal_relations[0].target_fact_index == 0
+    assert facts[3].causal_relations[0].target_fact_index == 2
+
+
+def test_invalid_local_causal_targets_are_dropped():
+    relations = [
+        CausalRelation(target_fact_index=-1, relation_type="caused_by"),
+        CausalRelation(target_fact_index=2, relation_type="caused_by"),
+        SimpleNamespace(target_fact_index=True, relation_type="caused_by"),
+        SimpleNamespace(target_fact_index=0.5, relation_type="caused_by"),
+        SimpleNamespace(target_fact_index="0", relation_type="caused_by"),
+    ]
+
+    assert fact_extraction._convert_causal_relations(relations, 4, 2) == []
+
+
+@pytest.mark.asyncio
+async def test_batch_causal_targets_use_each_chunk_start(monkeypatch):
+    fact_groups = [
+        [{"what": f"prior {index}", "fact_type": "world"} for index in range(4)],
+        [
+            {"what": "first cause", "fact_type": "world"},
+            {
+                "what": "first effect",
+                "fact_type": "world",
+                "causal_relations": [{"target_index": 0, "relation_type": "caused_by"}],
+            },
+        ],
+        [
+            {"what": "second cause", "fact_type": "world"},
+            {
+                "what": "second effect",
+                "fact_type": "world",
+                "causal_relations": [
+                    {"target_index": 0, "relation_type": "caused_by"},
+                    {"target_index": 2, "relation_type": "caused_by"},
+                ],
+            },
+        ],
+    ]
+
+    class Provider:
+        async def supports_batch_api(self):
+            return True
+
+        async def submit_batch(self, _requests):
+            return {"batch_id": "test"}
+
+        async def get_batch_status(self, _batch_id):
+            return {"status": "completed", "request_counts": {"completed": 3, "total": 3}}
+
+        async def retrieve_batch_results(self, _batch_id):
+            return [
+                {
+                    "custom_id": f"chunk_{index}",
+                    "response": {
+                        "body": {
+                            "choices": [{"message": {"content": json.dumps({"facts": facts})}}],
+                            "usage": {},
+                        }
+                    },
+                }
+                for index, facts in enumerate(fact_groups)
+            ]
+
+    monkeypatch.setattr(fact_extraction, "chunk_text", lambda text, **_kwargs: text.split("|"))
+    monkeypatch.setattr(fact_extraction, "_build_extraction_prompt_and_schema", lambda _config: ("", object))
+    monkeypatch.setattr(fact_extraction, "_retain_mission_preamble", lambda _config: "")
+    monkeypatch.setattr(fact_extraction, "_build_user_message", lambda *_args, **_kwargs: "")
+    monkeypatch.setattr(fact_extraction, "_build_request_body", lambda *_args: {})
+    monkeypatch.setattr(fact_extraction, "_add_temporal_offsets", lambda *_args: None)
+    monkeypatch.setattr(fact_extraction, "_inject_label_tags", lambda *_args: None)
+
+    config = SimpleNamespace(
+        retain_extract_causal_links=True,
+        retain_chunk_size=100,
+        retain_structured_chunk_size=100,
+        retain_batch_poll_interval_seconds=0,
+    )
+    llm_config = SimpleNamespace(_provider_impl=Provider(), provider="test")
+    facts, _, _ = await fact_extraction.extract_facts_from_contents_batch_api(
+        [RetainContent(content="preceding"), RetainContent(content="first|second")],
+        llm_config=llm_config,
+        agent_name="test",
+        config=config,
+    )
+
+    assert facts[5].causal_relations[0].target_fact_index == 4
+    assert facts[7].causal_relations[0].target_fact_index == 6
+    assert len(facts[7].causal_relations) == 1

hindsight-api-slim/tests/test_claude_code_llm_error_result.py

@@ -18,12 +18,18 @@ from __future__ import annotations
 
 from dataclasses import dataclass, field
 from typing import Any
+from unittest.mock import MagicMock
 
 import pytest
+from pydantic import BaseModel
 
 QUOTA_ERROR_TEXT = "You've hit your weekly limit · resets Jul 18, 12pm (UTC)"
 
 
+class _StructuredResponse(BaseModel):
+    fact: str
+
+
 @dataclass
 class _FakeOptions:
     """Stand-in for ClaudeAgentOptions; captures kwargs without importing SDK."""
@@ -34,6 +40,7 @@ class _FakeOptions:
     tools: list[str] = field(default_factory=list)
     env: dict[str, str] = field(default_factory=dict)
     mcp_servers: dict[str, Any] = field(default_factory=dict)
+    model: str | None = None
 
 
 class _FakeAssistantMessage:
@@ -139,6 +146,67 @@ async def test_call_ignores_non_error_result_message(monkeypatch):
     assert result == "ok"
 
 
+@pytest.mark.asyncio
+async def test_call_records_span_for_unvalidated_dict(monkeypatch):
+    """skip_validation returns a dict, which must still be serialized into the span."""
+    import claude_agent_sdk
+
+    import hindsight_api.tracing as tracing
+
+    async def fake_query(prompt: str, options: _FakeOptions):
+        yield _FakeAssistantMessage(content=[_FakeTextBlock(text='{"fact": "x"}')])
+        yield _FakeResultMessage(subtype="success", is_error=False, result='{"fact": "x"}')
+
+    span_recorder = MagicMock()
+    monkeypatch.setattr(claude_agent_sdk, "ClaudeAgentOptions", _FakeOptions)
+    monkeypatch.setattr(claude_agent_sdk, "AssistantMessage", _FakeAssistantMessage)
+    monkeypatch.setattr(claude_agent_sdk, "TextBlock", _FakeTextBlock)
+    monkeypatch.setattr(claude_agent_sdk, "ResultMessage", _FakeResultMessage)
+    monkeypatch.setattr(claude_agent_sdk, "query", fake_query)
+    monkeypatch.setattr(tracing, "get_span_recorder", lambda: span_recorder)
+
+    result = await _instantiate_provider().call(
+        messages=[{"role": "user", "content": "extract facts"}],
+        response_format=_StructuredResponse,
+        skip_validation=True,
+        max_retries=0,
+        scope="retain_extract_facts",
+    )
+
+    assert result == {"fact": "x"}
+    assert span_recorder.record_llm_call.call_args.kwargs["response_content"] == '{"fact": "x"}'
+
+
+@pytest.mark.asyncio
+async def test_call_survives_span_recorder_failure(monkeypatch):
+    """A raising span recorder must be logged, never break the call (best-effort, #3025)."""
+    import claude_agent_sdk
+
+    import hindsight_api.tracing as tracing
+
+    async def fake_query(prompt: str, options: _FakeOptions):
+        yield _FakeAssistantMessage(content=[_FakeTextBlock(text="ok")])
+        yield _FakeResultMessage(subtype="success", is_error=False, result="ok")
+
+    span_recorder = MagicMock()
+    span_recorder.record_llm_call.side_effect = RuntimeError("recorder exploded")
+    monkeypatch.setattr(claude_agent_sdk, "ClaudeAgentOptions", _FakeOptions)
+    monkeypatch.setattr(claude_agent_sdk, "AssistantMessage", _FakeAssistantMessage)
+    monkeypatch.setattr(claude_agent_sdk, "TextBlock", _FakeTextBlock)
+    monkeypatch.setattr(claude_agent_sdk, "ResultMessage", _FakeResultMessage)
+    monkeypatch.setattr(claude_agent_sdk, "query", fake_query)
+    monkeypatch.setattr(tracing, "get_span_recorder", lambda: span_recorder)
+
+    result = await _instantiate_provider().call(
+        messages=[{"role": "user", "content": "hi"}],
+        max_retries=0,
+        scope="test",
+    )
+
+    assert result == "ok"
+    span_recorder.record_llm_call.assert_called_once()
+
+
 @pytest.mark.asyncio
 async def test_call_with_tools_raises_with_result_text_on_error_result(monkeypatch):
     """call_with_tools() must surface ResultMessage.result the same way."""

hindsight-api-slim/tests/test_claude_code_llm_isolation.py

@@ -31,6 +31,7 @@ class _FakeOptions:
     tools: list[str] = field(default_factory=list)
     env: dict[str, str] = field(default_factory=dict)
     mcp_servers: dict[str, Any] = field(default_factory=dict)
+    model: str | None = None
 
 
 class _FakeAssistantMessage:

hindsight-api-slim/tests/test_claude_code_llm_tool_round.py

@@ -0,0 +1,347 @@
+"""Regression tests for issue #2966.
+
+``call_with_tools()`` is a *single round* of an agentic loop the caller drives:
+the model proposes tool call(s), the provider returns them, and the orchestrator
+(reflect/agent.py) executes the REAL tools and feeds the results back on the next
+call. See providers/openai_compatible_llm.py for the reference contract.
+
+The Claude Agent SDK, however, runs its own in-process loop and invokes our SDK
+MCP handlers — which are deliberate placeholders returning
+``[Tool <name> called successfully]`` with no real data. With ``max_turns >= 2``
+the model calls a search tool, sees the empty placeholder, re-queries, exhausts
+the turn budget, and the run ends in ``error_max_turns`` *with its tool calls
+discarded* — the "0 tool calls / no information" reflect failure in #2966.
+
+The fix caps the SDK at ``max_turns=1`` and:
+  * breaks out of the stream as soon as a tool call is proposed (so the SDK never
+    acts on a placeholder result), returning the call to the caller; and
+  * treats the ``error_max_turns`` ResultMessage that necessarily follows a
+    single-turn tool call as non-fatal — the tool call is the result, not an error.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+import pytest
+
+from hindsight_api.engine.llm_interface import LLM_TOOL_CHOICE_AUTO
+
+
+@dataclass
+class _FakeOptions:
+    """Stand-in for ClaudeAgentOptions; captures kwargs without importing the SDK."""
+
+    system_prompt: str | None = None
+    max_turns: int | None = None
+    allowed_tools: list[str] = field(default_factory=list)
+    tools: list[str] = field(default_factory=list)
+    env: dict[str, str] = field(default_factory=dict)
+    mcp_servers: dict[str, Any] = field(default_factory=dict)
+    model: str | None = None
+
+
+class _FakeAssistantMessage:
+    def __init__(self, content: list[Any]) -> None:
+        self.content = content
+
+
+class _FakeTextBlock:
+    def __init__(self, text: str) -> None:
+        self.text = text
+
+
+class _FakeToolUseBlock:
+    def __init__(self, id: str, name: str, input: dict[str, Any]) -> None:
+        self.id = id
+        self.name = name
+        self.input = input
+
+
+class _FakeResultMessage:
+    def __init__(self, subtype: str, is_error: bool, result: str | None) -> None:
+        self.subtype = subtype
+        self.is_error = is_error
+        self.result = result
+
+
+@dataclass
+class _FakeSdkMcpTool:
+    name: str
+    description: str
+    input_schema: dict[str, Any]
+    handler: Any
+
+
+def _fake_create_sdk_mcp_server(name: str, version: str, tools=None):
+    return {"name": name, "version": version, "tools": tools}
+
+
+def _install_fake_sdk(monkeypatch, client_cls) -> None:
+    import claude_agent_sdk
+
+    monkeypatch.setattr(claude_agent_sdk, "ClaudeAgentOptions", _FakeOptions)
+    monkeypatch.setattr(claude_agent_sdk, "AssistantMessage", _FakeAssistantMessage)
+    monkeypatch.setattr(claude_agent_sdk, "TextBlock", _FakeTextBlock)
+    monkeypatch.setattr(claude_agent_sdk, "ToolUseBlock", _FakeToolUseBlock)
+    monkeypatch.setattr(claude_agent_sdk, "ResultMessage", _FakeResultMessage)
+    monkeypatch.setattr(claude_agent_sdk, "ClaudeSDKClient", client_cls)
+    monkeypatch.setattr(claude_agent_sdk, "SdkMcpTool", _FakeSdkMcpTool)
+    monkeypatch.setattr(claude_agent_sdk, "create_sdk_mcp_server", _fake_create_sdk_mcp_server)
+
+
+def _instantiate_provider():
+    from hindsight_api.engine.providers.claude_code_llm import ClaudeCodeLLM
+
+    return ClaudeCodeLLM(
+        provider="claude-code",
+        api_key="",
+        base_url="",
+        model="claude-haiku-4-5",
+        reasoning_effort="low",
+    )
+
+
+_RECALL_TOOL = {
+    "function": {
+        "name": "recall",
+        "description": "Search the memory bank.",
+        "parameters": {
+            "type": "object",
+            "properties": {"query": {"type": "string"}},
+            "required": ["query"],
+        },
+    }
+}
+
+
+@pytest.mark.asyncio
+async def test_tool_call_returned_despite_error_max_turns(monkeypatch):
+    """A single-turn tool call followed by error_max_turns must be returned, not raised.
+
+    This is the exact #2966 failure shape: the model emits a tool call and, because
+    the single turn was spent on it, the CLI reports ``error_max_turns``. The tool
+    call is the intended result and must reach the caller.
+    """
+    captured_options: dict[str, Any] = {}
+
+    class _FakeClient:
+        def __init__(self, options: _FakeOptions) -> None:
+            captured_options["opts"] = options
+
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, exc_type, exc, tb):
+            return False
+
+        async def query(self, prompt: str) -> None:
+            return None
+
+        async def receive_response(self):
+            yield _FakeAssistantMessage(
+                content=[
+                    _FakeToolUseBlock(
+                        id="tu_1",
+                        name="mcp__hindsight_tools__recall",
+                        input={"query": "domain family repositories"},
+                    )
+                ]
+            )
+            # With max_turns=1, the tool call consumes the only turn, so the CLI
+            # necessarily follows up with error_max_turns.
+            yield _FakeResultMessage(subtype="error_max_turns", is_error=True, result=None)
+
+    _install_fake_sdk(monkeypatch, _FakeClient)
+
+    provider = _instantiate_provider()
+    result = await provider.call_with_tools(
+        messages=[{"role": "user", "content": "What repos are in the domain family?"}],
+        tools=[_RECALL_TOOL],
+        max_retries=0,
+        scope="reflect",
+        tool_choice=LLM_TOOL_CHOICE_AUTO,
+    )
+
+    assert result.finish_reason == "tool_calls"
+    assert [tc.name for tc in result.tool_calls] == ["recall"]
+    assert result.tool_calls[0].arguments == {"query": "domain family repositories"}
+    # The MCP prefix must be stripped for the caller.
+    assert not result.tool_calls[0].name.startswith("mcp__")
+    # The provider must cap the SDK at one turn so it never acts on placeholder results.
+    assert captured_options["opts"].max_turns == 1
+    # The configured model must be pinned on the SDK options (issue #2881), otherwise
+    # the CLI silently runs its own default model.
+    assert captured_options["opts"].model == "claude-haiku-4-5"
+
+
+@pytest.mark.asyncio
+async def test_stops_after_first_tool_round(monkeypatch):
+    """The provider must stop consuming the stream once a tool call is proposed.
+
+    If it kept reading, the SDK's next placeholder-driven turn would append more
+    (duplicate/reworded) recall calls — the runaway loop that exhausts the budget.
+    """
+    later_turns_consumed = {"value": False}
+
+    class _FakeClient:
+        def __init__(self, options: _FakeOptions) -> None:
+            pass
+
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, exc_type, exc, tb):
+            return False
+
+        async def query(self, prompt: str) -> None:
+            return None
+
+        async def receive_response(self):
+            yield _FakeAssistantMessage(
+                content=[
+                    _FakeToolUseBlock(
+                        id="tu_1",
+                        name="mcp__hindsight_tools__recall",
+                        input={"query": "first"},
+                    )
+                ]
+            )
+            # These would only be produced by the SDK re-prompting the model with a
+            # placeholder result. The provider must break before pulling them.
+            later_turns_consumed["value"] = True
+            yield _FakeAssistantMessage(
+                content=[
+                    _FakeToolUseBlock(
+                        id="tu_2",
+                        name="mcp__hindsight_tools__recall",
+                        input={"query": "reworded"},
+                    )
+                ]
+            )
+
+    _install_fake_sdk(monkeypatch, _FakeClient)
+
+    provider = _instantiate_provider()
+    result = await provider.call_with_tools(
+        messages=[{"role": "user", "content": "hi"}],
+        tools=[_RECALL_TOOL],
+        max_retries=0,
+        scope="reflect",
+        tool_choice=LLM_TOOL_CHOICE_AUTO,
+    )
+
+    assert [tc.arguments["query"] for tc in result.tool_calls] == ["first"]
+    assert later_turns_consumed["value"] is False
+
+
+@pytest.mark.asyncio
+async def test_text_only_answer_returned(monkeypatch):
+    """A text-only turn (no tool call) is returned as a normal 'stop' response.
+
+    This is the loop's final round: the orchestrator has already fed tool results
+    back and the model answers in prose.
+    """
+
+    class _FakeClient:
+        def __init__(self, options: _FakeOptions) -> None:
+            pass
+
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, exc_type, exc, tb):
+            return False
+
+        async def query(self, prompt: str) -> None:
+            return None
+
+        async def receive_response(self):
+            yield _FakeAssistantMessage(content=[_FakeTextBlock(text="The domain family has 8 repositories.")])
+            yield _FakeResultMessage(subtype="success", is_error=False, result="done")
+
+    _install_fake_sdk(monkeypatch, _FakeClient)
+
+    provider = _instantiate_provider()
+    result = await provider.call_with_tools(
+        messages=[{"role": "user", "content": "answer now"}],
+        tools=[_RECALL_TOOL],
+        max_retries=0,
+        scope="reflect",
+        tool_choice=LLM_TOOL_CHOICE_AUTO,
+    )
+
+    assert result.finish_reason == "stop"
+    assert result.tool_calls == []
+    assert result.content == "The domain family has 8 repositories."
+
+
+@pytest.mark.asyncio
+async def test_call_pins_configured_model(monkeypatch):
+    """call() must pass the configured model to the SDK (issue #2881).
+
+    Without model= the spawned CLI runs its own default model regardless of
+    HINDSIGHT_API_*_LLM_MODEL, while metrics/logs still print the configured one.
+    """
+    import claude_agent_sdk
+
+    captured: dict[str, Any] = {}
+
+    async def fake_query(prompt: str, options: _FakeOptions):
+        captured["opts"] = options
+        yield _FakeAssistantMessage(content=[_FakeTextBlock(text="ok")])
+        yield _FakeResultMessage(subtype="success", is_error=False, result="ok")
+
+    monkeypatch.setattr(claude_agent_sdk, "ClaudeAgentOptions", _FakeOptions)
+    monkeypatch.setattr(claude_agent_sdk, "AssistantMessage", _FakeAssistantMessage)
+    monkeypatch.setattr(claude_agent_sdk, "TextBlock", _FakeTextBlock)
+    monkeypatch.setattr(claude_agent_sdk, "ResultMessage", _FakeResultMessage)
+    monkeypatch.setattr(claude_agent_sdk, "query", fake_query)
+
+    provider = _instantiate_provider()
+    result = await provider.call(
+        messages=[{"role": "user", "content": "hi"}],
+        max_retries=0,
+        scope="test",
+    )
+
+    assert result == "ok"
+    assert captured["opts"].model == "claude-haiku-4-5"
+
+
+@pytest.mark.asyncio
+async def test_genuine_error_without_tool_calls_still_raises(monkeypatch):
+    """An error ResultMessage with nothing collected must still surface (issue #2702)."""
+
+    class _FakeClient:
+        def __init__(self, options: _FakeOptions) -> None:
+            pass
+
+        async def __aenter__(self):
+            return self
+
+        async def __aexit__(self, exc_type, exc, tb):
+            return False
+
+        async def query(self, prompt: str) -> None:
+            return None
+
+        async def receive_response(self):
+            yield _FakeResultMessage(
+                subtype="success",
+                is_error=True,
+                result="You've hit your weekly limit · resets Jul 18, 12pm (UTC)",
+            )
+
+    _install_fake_sdk(monkeypatch, _FakeClient)
+
+    provider = _instantiate_provider()
+    with pytest.raises(RuntimeError, match="weekly limit"):
+        await provider.call_with_tools(
+            messages=[{"role": "user", "content": "hi"}],
+            tools=[_RECALL_TOOL],
+            max_retries=0,
+            scope="reflect",
+            tool_choice=LLM_TOOL_CHOICE_AUTO,
+        )

hindsight-api-slim/tests/test_codex_reasoning_effort.py

@@ -0,0 +1,64 @@
+"""Regression tests for Codex reasoning-effort request serialization."""
+
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from hindsight_api.engine.providers.codex_llm import CodexLLM
+
+
+def build_llm(reasoning_effort: str = "high") -> CodexLLM:
+    with (
+        patch.object(CodexLLM, "_load_codex_auth", return_value=("token", "account")),
+        patch.object(CodexLLM, "_load_codex_refresh_token", return_value=None),
+    ):
+        return CodexLLM(
+            provider="openai-codex",
+            api_key="ignored",
+            base_url="https://chatgpt.com/backend-api",
+            model="gpt-5.6-luna",
+            reasoning_effort=reasoning_effort,
+        )
+
+
+@pytest.mark.asyncio
+async def test_call_sends_reasoning_effort_separately_from_summary() -> None:
+    llm = build_llm("high")
+    response = MagicMock()
+    response.raise_for_status.return_value = None
+
+    with (
+        patch.object(llm._client, "post", new_callable=AsyncMock) as mock_post,
+        patch.object(llm, "_parse_sse_stream", new_callable=AsyncMock, return_value="ok"),
+    ):
+        mock_post.return_value = response
+        await llm.call(messages=[{"role": "user", "content": "hello"}], max_retries=0)
+
+    assert mock_post.call_args.kwargs["json"]["reasoning"] == {
+        "effort": "high",
+        "summary": "detailed",
+    }
+
+
+@pytest.mark.asyncio
+async def test_call_with_tools_sends_reasoning_effort_separately_from_summary() -> None:
+    llm = build_llm("low")
+    response = MagicMock()
+    response.status_code = 200
+    response.raise_for_status.return_value = None
+
+    with (
+        patch.object(llm._client, "post", new_callable=AsyncMock) as mock_post,
+        patch.object(llm, "_parse_sse_tool_stream", new_callable=AsyncMock, return_value=(None, [])),
+    ):
+        mock_post.return_value = response
+        await llm.call_with_tools(
+            messages=[{"role": "user", "content": "hello"}],
+            tools=[],
+            max_retries=0,
+        )
+
+    assert mock_post.call_args.kwargs["json"]["reasoning"] == {
+        "effort": "low",
+        "summary": "concise",
+    }

hindsight-api-slim/tests/test_codex_strict_schema.py

@@ -122,20 +122,23 @@ async def test_strict_schema_skip_validation_returns_dict():
     response = MagicMock()
     response.raise_for_status.return_value = None
     tool_call = LLMToolCall(id="c", name="structured_response", arguments={"fact": "x"})
+    span_recorder = MagicMock()
 
-    with patch.object(llm._client, "post", new_callable=AsyncMock) as mock_post:
-        mock_post.return_value = response
-        with patch.object(llm, "_parse_sse_tool_stream", new_callable=AsyncMock) as mock_parse:
-            mock_parse.return_value = (None, [tool_call])
-            result = await llm.call(
-                messages=[{"role": "user", "content": "hi"}],
-                response_format=_Fact,
-                strict_schema=True,
-                skip_validation=True,
-                max_retries=0,
-            )
+    with patch("hindsight_api.tracing.get_span_recorder", return_value=span_recorder):
+        with patch.object(llm._client, "post", new_callable=AsyncMock) as mock_post:
+            mock_post.return_value = response
+            with patch.object(llm, "_parse_sse_tool_stream", new_callable=AsyncMock) as mock_parse:
+                mock_parse.return_value = (None, [tool_call])
+                result = await llm.call(
+                    messages=[{"role": "user", "content": "hi"}],
+                    response_format=_Fact,
+                    strict_schema=True,
+                    skip_validation=True,
+                    max_retries=0,
+                )
 
     assert result == {"fact": "x"}
+    assert span_recorder.record_llm_call.call_args.kwargs["response_content"] == '{"fact": "x"}'
 
 
 @pytest.mark.asyncio

hindsight-api-slim/tests/test_config_validation.py

@@ -27,6 +27,10 @@ def setup_test_env():
         "HINDSIGHT_API_LLM_BEDROCK_SERVICE_TIER",
         "HINDSIGHT_API_LLM_GEMINI_SERVICE_TIER",
         "HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY",
+        "HINDSIGHT_API_GRAPH_SEED_MIN_SIMILARITY",
+        "HINDSIGHT_API_TEMPORAL_SEMANTIC_MIN_SIMILARITY",
+        "HINDSIGHT_API_SEMANTIC_LINK_MIN_SIMILARITY",
+        "HINDSIGHT_API_CONSOLIDATION_DEDUP_THRESHOLD",
         "HINDSIGHT_API_DATABASE_URL",
         "HINDSIGHT_API_MIGRATION_DATABASE_URL",
     ]
@@ -227,23 +231,74 @@ def test_retain_strategy_structured_chunk_size_validation():
     assert resolved.retain_structured_chunk_size == 2000
 
 
-def test_semantic_min_similarity_reads_from_env():
-    """Semantic retrieval min similarity can be configured at the server level."""
-    from hindsight_api.config import HindsightConfig
+def test_embedding_similarity_threshold_defaults_are_backward_compatible(monkeypatch):
+    """Unset threshold settings preserve the five existing operating points."""
+    from hindsight_api.config import (
+        ENV_CONSOLIDATION_DEDUP_THRESHOLD,
+        ENV_GRAPH_SEED_MIN_SIMILARITY,
+        ENV_SEMANTIC_LINK_MIN_SIMILARITY,
+        ENV_SEMANTIC_MIN_SIMILARITY,
+        ENV_TEMPORAL_SEMANTIC_MIN_SIMILARITY,
+        HindsightConfig,
+    )
 
-    os.environ["HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY"] = "0.58"
+    for env_name in (
+        ENV_SEMANTIC_MIN_SIMILARITY,
+        ENV_GRAPH_SEED_MIN_SIMILARITY,
+        ENV_TEMPORAL_SEMANTIC_MIN_SIMILARITY,
+        ENV_SEMANTIC_LINK_MIN_SIMILARITY,
+        ENV_CONSOLIDATION_DEDUP_THRESHOLD,
+    ):
+        monkeypatch.delenv(env_name, raising=False)
 
     config = HindsightConfig.from_env()
-    assert config.semantic_min_similarity == 0.58
+
+    assert config.semantic_min_similarity == 0.3
+    assert config.graph_seed_min_similarity == 0.3
+    assert config.temporal_semantic_min_similarity == 0.1
+    assert config.semantic_link_min_similarity == 0.7
+    assert config.consolidation_dedup_threshold == 0.97
 
 
-def test_semantic_min_similarity_must_be_between_zero_and_one():
-    """Invalid semantic min similarity fails fast during configuration loading."""
+@pytest.mark.parametrize(
+    ("env_name", "field_name", "value"),
+    [
+        ("HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY", "semantic_min_similarity", 0.58),
+        ("HINDSIGHT_API_GRAPH_SEED_MIN_SIMILARITY", "graph_seed_min_similarity", 0.41),
+        ("HINDSIGHT_API_TEMPORAL_SEMANTIC_MIN_SIMILARITY", "temporal_semantic_min_similarity", 0.22),
+        ("HINDSIGHT_API_SEMANTIC_LINK_MIN_SIMILARITY", "semantic_link_min_similarity", 0.81),
+    ],
+)
+def test_embedding_similarity_thresholds_read_from_env(env_name: str, field_name: str, value: float):
+    """Each configurable similarity gate has an independent environment setting."""
     from hindsight_api.config import HindsightConfig
 
-    os.environ["HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY"] = "1.5"
+    os.environ[env_name] = str(value)
 
-    with pytest.raises(ValueError, match="semantic_min_similarity"):
+    config = HindsightConfig.from_env()
+
+    assert getattr(config, field_name) == value
+
+
+@pytest.mark.parametrize(
+    ("env_name", "field_name"),
+    [
+        ("HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY", "semantic_min_similarity"),
+        ("HINDSIGHT_API_GRAPH_SEED_MIN_SIMILARITY", "graph_seed_min_similarity"),
+        ("HINDSIGHT_API_TEMPORAL_SEMANTIC_MIN_SIMILARITY", "temporal_semantic_min_similarity"),
+        ("HINDSIGHT_API_SEMANTIC_LINK_MIN_SIMILARITY", "semantic_link_min_similarity"),
+    ],
+)
+@pytest.mark.parametrize("invalid_value", ["-0.01", "1.01"])
+def test_embedding_similarity_thresholds_must_be_between_zero_and_one(
+    env_name: str, field_name: str, invalid_value: str
+):
+    """All embedding-dependent gates fail fast outside the supported range."""
+    from hindsight_api.config import HindsightConfig
+
+    os.environ[env_name] = invalid_value
+
+    with pytest.raises(ValueError, match=field_name):
         HindsightConfig.from_env()
 
 
@@ -835,3 +890,44 @@ def test_operation_cleanup_batch_size_requires_positive_integer(monkeypatch, raw
 
     with pytest.raises(ValueError, match=ENV_OPERATION_CLEANUP_BATCH_SIZE):
         HindsightConfig.from_env()
+
+
+# --- Worker per-type slot reservations: RESERVED_SLOTS (floor) + deprecated _MAX_SLOTS alias ---
+
+
+def test_worker_reserved_slots_parse(monkeypatch):
+    """RESERVED_SLOTS sets the reservation floor for an operation type."""
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+    monkeypatch.setenv("HINDSIGHT_API_WORKER_RETAIN_RESERVED_SLOTS", "2")
+
+    config = HindsightConfig.from_env()
+
+    assert config.worker_slot_reservations["retain"] == 2
+
+
+def test_worker_legacy_max_slots_is_deprecated_alias_for_reserved(monkeypatch, caplog):
+    """The legacy _MAX_SLOTS env var maps to the reservation floor and warns."""
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+    monkeypatch.setenv("HINDSIGHT_API_WORKER_CONSOLIDATION_MAX_SLOTS", "3")
+
+    with caplog.at_level(logging.WARNING):
+        config = HindsightConfig.from_env()
+
+    assert config.worker_slot_reservations["consolidation"] == 3
+    assert any("HINDSIGHT_API_WORKER_CONSOLIDATION_MAX_SLOTS" in r.message for r in caplog.records)
+
+
+def test_worker_reserved_and_legacy_both_set_is_rejected(monkeypatch):
+    """Setting both RESERVED_SLOTS and the deprecated _MAX_SLOTS alias is ambiguous."""
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+    monkeypatch.setenv("HINDSIGHT_API_WORKER_RETAIN_RESERVED_SLOTS", "2")
+    monkeypatch.setenv("HINDSIGHT_API_WORKER_RETAIN_MAX_SLOTS", "2")
+
+    with pytest.raises(ValueError, match="RESERVED_SLOTS"):
+        HindsightConfig.from_env()

hindsight-api-slim/tests/test_consolidation.py

@@ -6,7 +6,8 @@ Note: Consolidation runs automatically after retain via SyncTaskBackend in tests
 
 import uuid
 from datetime import datetime, timezone
-from unittest.mock import AsyncMock, MagicMock, call, patch
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, patch
 
 import pytest
 
@@ -1944,40 +1945,6 @@ class TestMentalModelRefreshAfterConsolidation:
         await memory.delete_bank(bank_id, request_context=request_context)
 
 
-def test_consolidation_prompt_default():
-    """Test that the default consolidation prompt contains the built-in mission and processing rules."""
-    from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-
-    prompt = build_batch_consolidation_prompt()
-    # Verify core structural elements are present (not exact wording)
-    assert "STATE CHANGES" in prompt
-    assert "RESOLVE REFERENCES" in prompt
-    assert "{facts_text}" in prompt
-    assert "{observations_text}" in prompt
-
-
-def test_consolidation_prompt_observations_mission():
-    """Test that observations_mission replaces the default mission but keeps processing rules."""
-    from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-
-    spec = "Observations are weekly summaries of sprint outcomes and team dynamics."
-    prompt = build_batch_consolidation_prompt(observations_mission=spec)
-
-    # Spec is injected
-    assert spec in prompt
-    # Processing rules and output format always remain
-    assert "RESOLVE REFERENCES" in prompt
-    assert "creates" in prompt
-    assert "updates" in prompt
-    assert "{facts_text}" in prompt
-    assert "{observations_text}" in prompt
-
-    # Renders cleanly
-    rendered = prompt.format(facts_text="Alice fixed a bug.", observations_text="[]")
-    assert "{facts_text}" not in rendered
-    assert spec in rendered
-
-
 def test_observations_mission_config():
     """Test that observations_mission is loaded from env and exposed as configurable."""
     import os
@@ -2473,50 +2440,33 @@ class TestConsolidationSourceFactsConfig:
 
 
 class TestBuildResponseModel:
-    """Unit tests for _build_response_model (dynamic Pydantic model factory)."""
+    """Unit tests for _build_response_model (consolidation response-model factory)."""
 
     def test_no_limit_returns_base_model(self):
-        """When max_creates is None or -1, the base model is returned."""
         from hindsight_api.engine.consolidation.consolidator import _ConsolidationBatchResponse
 
-        assert _build_response_model(None) is _ConsolidationBatchResponse
-        assert _build_response_model(-1) is _ConsolidationBatchResponse
+        for max_creates in (None, -1):
+            assert _build_response_model(max_creates) is _ConsolidationBatchResponse
 
-    def test_zero_limit_forbids_creates(self):
-        """When max_creates=0, the model rejects any creates."""
-        model = _build_response_model(0)
-        # Valid: no creates
-        result = model(creates=[], updates=[], deletes=[])
-        assert result.creates == []
+    def test_schema_contains_max_items_by_default(self):
+        for max_creates in (0, 3, 50):
+            schema = _build_response_model(max_creates).model_json_schema()
+            creates_prop = schema["properties"]["creates"]
+            assert creates_prop.get("maxItems") == max_creates
 
-        # Invalid: one create should be rejected
+    def test_schema_omits_max_items_when_unsupported(self):
+        """Regression for #2500: Bedrock rejects maxItems in response schemas."""
+        from hindsight_api.engine.consolidation.consolidator import _ConsolidationBatchResponse
+
+        for max_creates in (0, 3, 50):
+            model = _build_response_model(max_creates, supports_max_items=False)
+            assert model is _ConsolidationBatchResponse
+            assert "maxItems" not in model.model_json_schema()["properties"]["creates"]
+
+    def test_model_enforces_cap_by_default(self):
         from pydantic import ValidationError
 
-        with pytest.raises(ValidationError):
-            model(
-                creates=[{"text": "obs", "source_fact_ids": ["abc"]}],
-                updates=[],
-                deletes=[],
-            )
-
-    def test_positive_limit_allows_up_to_max(self):
-        """When max_creates=2, exactly 2 creates are allowed but 3 are rejected."""
         model = _build_response_model(2)
-
-        # 2 creates OK
-        result = model(
-            creates=[
-                {"text": "obs1", "source_fact_ids": ["a"]},
-                {"text": "obs2", "source_fact_ids": ["b"]},
-            ],
-            updates=[],
-            deletes=[],
-        )
-        assert len(result.creates) == 2
-
-        # 3 creates rejected
-        from pydantic import ValidationError
-
         with pytest.raises(ValidationError):
             model(
                 creates=[
@@ -2528,8 +2478,21 @@ class TestBuildResponseModel:
                 deletes=[],
             )
 
+    def test_model_accepts_over_cap_when_max_items_unsupported(self):
+        model = _build_response_model(2, supports_max_items=False)
+        result = model(
+            creates=[
+                {"text": "obs1", "source_fact_ids": ["a"]},
+                {"text": "obs2", "source_fact_ids": ["b"]},
+                {"text": "obs3", "source_fact_ids": ["c"]},
+            ],
+            updates=[],
+            deletes=[],
+        )
+        assert len(result.creates) == 3
+
     def test_updates_and_deletes_unconstrained(self):
-        """max_creates does not affect updates or deletes."""
+        """The creates cap never constrained updates or deletes (unchanged)."""
         model = _build_response_model(0)
         result = model(
             creates=[],
@@ -2542,306 +2505,46 @@ class TestBuildResponseModel:
         assert len(result.updates) == 2
         assert len(result.deletes) == 1
 
-    def test_json_schema_contains_max_items(self):
-        """The generated model's JSON schema should include maxItems for creates."""
-        model = _build_response_model(3)
-        schema = model.model_json_schema()
-        creates_prop = schema["properties"]["creates"]
-        assert creates_prop.get("maxItems") == 3
-
-
-class TestConsolidationPromptCapacity:
-    """Unit tests for the capacity constraint in the consolidation prompt."""
-
-    def test_no_capacity_note(self):
-        """When no capacity note is provided, prompt has no CAPACITY CONSTRAINT section."""
-        from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-
-        prompt = build_batch_consolidation_prompt()
-        assert "CAPACITY CONSTRAINT" not in prompt
-
-    def test_capacity_note_included(self):
-        """When a capacity note is provided, it appears in the prompt."""
-        from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-
-        prompt = build_batch_consolidation_prompt(
-            observation_capacity_note="OBSERVATION LIMIT REACHED. Only UPDATE or DELETE."
-        )
-        assert "CAPACITY CONSTRAINT" in prompt
-        assert "OBSERVATION LIMIT REACHED" in prompt
-
-    def test_capacity_note_with_mission(self):
-        """Capacity note and custom mission can coexist."""
-        from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-
-        prompt = build_batch_consolidation_prompt(
-            observations_mission="Track food preferences only.",
-            observation_capacity_note="2 slots remaining.",
-        )
-        assert "Track food preferences only." in prompt
-        assert "2 slots remaining." in prompt
-        # Both should be present
-        assert "MISSION" in prompt
-        assert "CAPACITY CONSTRAINT" in prompt
-
-
-class TestFullAssembledConsolidationPrompt:
-    """End-to-end assembly of the consolidation prompt the LLM actually sees.
-
-    Reproduces the substitution the consolidator does at runtime (consolidator.py
-    around `_consolidate_batch_with_llm`): builds realistic existing observations
-    and new facts, serializes them the same way, then `.format()`s the template
-    returned by ``build_batch_consolidation_prompt``.
-    """
-
-    def _build_fixture(self):
-        import json
-
-        from hindsight_api.engine.consolidation.consolidator import _build_observations_for_llm
-        from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-        from hindsight_api.engine.response_models import MemoryFact
-
-        # Existing source facts (already-stored, supporting the observations below)
-        src_a1 = MemoryFact(
-            id="aaaaaaaa-0000-0000-0000-000000000001",
-            text="Donald told Athena she is sovereign during the Janus design session.",
-            fact_type="experience",
-            occurred_start="2025-10-01T10:00:00Z",
-            mentioned_at="2025-10-01T10:00:00Z",
-            context="Janus design session",
-        )
-        src_a2 = MemoryFact(
-            id="aaaaaaaa-0000-0000-0000-000000000002",
-            text="Donald reiterated to Athena that she holds sovereignty over her own goals.",
-            fact_type="experience",
-            occurred_start="2025-10-05T14:00:00Z",
-            mentioned_at="2025-10-05T14:00:00Z",
-        )
-        src_b1 = MemoryFact(
-            id="bbbbbbbb-0000-0000-0000-000000000001",
-            text="Forge added the sovereignty line to SOUL.md.j2 in commit 1a2b3c.",
-            fact_type="world",
-            occurred_start="2025-10-03T09:00:00Z",
-            mentioned_at="2025-10-03T09:00:00Z",
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize(("remaining_slots", "expected_creates"), [(0, 0), (2, 2)])
+    async def test_unsupported_max_items_still_truncates_creates(
+        self, remaining_slots: int, expected_creates: int
+    ) -> None:
+        """The operation config controls the schema while the runtime cap remains hard."""
+        from hindsight_api.engine.consolidation.consolidator import (
+            _consolidate_batch_with_llm,
+            _ConsolidationBatchResponse,
+            _CreateAction,
         )
 
-        # Two existing observations the consolidator pulled in as merge candidates
-        obs_sovereignty = MemoryFact(
-            id="11111111-1111-1111-1111-111111111111",
-            text="Donald named Athena's sovereignty as a foundational principle of the Janus architecture.",
-            fact_type="observation",
-            occurred_start="2025-10-01T10:00:00Z",
-            occurred_end="2025-10-05T14:00:00Z",
-            mentioned_at="2025-10-05T14:00:00Z",
-            source_fact_ids=[src_a1.id, src_a2.id],
+        creates = [_CreateAction(text=f"observation {index}", source_fact_ids=[f"fact-{index}"]) for index in range(3)]
+        llm_config = SimpleNamespace(
+            _provider_impl=None,
+            call=AsyncMock(return_value=_ConsolidationBatchResponse(creates=creates)),
         )
-        obs_soul_file = MemoryFact(
-            id="22222222-2222-2222-2222-222222222222",
-            text="The sovereignty principle was codified in SOUL.md.j2.",
-            fact_type="observation",
-            occurred_start="2025-10-03T09:00:00Z",
-            occurred_end="2025-10-03T09:00:00Z",
-            mentioned_at="2025-10-03T09:00:00Z",
-            source_fact_ids=[src_b1.id],
+        config = SimpleNamespace(
+            llm_output_language=None,
+            observations_mission=None,
+            llm_strict_schema_consolidation=False,
+            llm_supports_max_items=False,
+            consolidation_max_attempts=1,
+            consolidation_llm_max_retries=None,
+            consolidation_max_completion_tokens=None,
         )
 
-        union_observations = [obs_sovereignty, obs_soul_file]
-        union_source_facts = {src_a1.id: src_a1, src_a2.id: src_a2, src_b1.id: src_b1}
-
-        # New incoming batch — one should merge into obs_sovereignty (issue #1566's
-        # bug: gets created as a sibling instead), one is genuinely new, one merges
-        # into obs_soul_file.
-        new_facts = [
-            {
-                "id": "cccccccc-0000-0000-0000-000000000001",
-                "text": "Donald reaffirmed to Athena that her sovereignty is non-negotiable.",
-                "occurred_start": "2025-10-10T11:00:00Z",
-                "mentioned_at": "2025-10-10T11:00:00Z",
-            },
-            {
-                "id": "cccccccc-0000-0000-0000-000000000002",
-                "text": "Athena chose to refactor the planning module on her own initiative.",
-                "occurred_start": "2025-10-11T16:30:00Z",
-                "mentioned_at": "2025-10-11T16:30:00Z",
-            },
-            {
-                "id": "cccccccc-0000-0000-0000-000000000003",
-                "text": "Forge updated SOUL.md.j2 to expand the sovereignty section.",
-                "occurred_start": "2025-10-12T08:00:00Z",
-                "mentioned_at": "2025-10-12T08:00:00Z",
-            },
-        ]
-
-        # Mission + capacity note exercise the optional sections.
-        mission = (
-            "Track durable architectural decisions and the people who made them. "
-            "Capture named principles, the agents involved, and where each "
-            "principle is codified in the codebase."
-        )
-        capacity_note = (
-            "This scope has 3 observation slot(s) remaining (out of 50). Prefer UPDATE over CREATE when possible."
+        result = await _consolidate_batch_with_llm(
+            llm_config=llm_config,
+            memories=[{"id": "fact-0", "text": "a fact"}],
+            union_observations=[],
+            union_source_facts={},
+            config=config,
+            remaining_observation_slots=remaining_slots,
+            max_observations_per_scope=2,
         )
 
-        # Build the template + substitute the same way consolidator.py does.
-        obs_list = _build_observations_for_llm(union_observations, union_source_facts)
-        observations_text = json.dumps(obs_list, indent=2, ensure_ascii=False)
-
-        def _fact_line(m: dict) -> str:
-            text = f"[{m['id']}] {m['text']}"
-            parts = []
-            if m.get("occurred_start"):
-                parts.append(f"occurred_start={m['occurred_start']}")
-            if m.get("occurred_end"):
-                parts.append(f"occurred_end={m['occurred_end']}")
-            if m.get("mentioned_at"):
-                parts.append(f"mentioned_at={m['mentioned_at']}")
-            if parts:
-                text += f" ({', '.join(parts)})"
-            return text
-
-        facts_lines = "\n".join(_fact_line(m) for m in new_facts)
-
-        template = build_batch_consolidation_prompt(
-            observations_mission=mission,
-            observation_capacity_note=capacity_note,
-        )
-        rendered = template.format(facts_text=facts_lines, observations_text=observations_text)
-
-        return {
-            "rendered": rendered,
-            "mission": mission,
-            "capacity_note": capacity_note,
-            "new_facts": new_facts,
-            "observations": [obs_sovereignty, obs_soul_file],
-            "source_facts": union_source_facts,
-        }
-
-    def test_fully_assembled_prompt_has_all_required_sections(self):
-        f = self._build_fixture()
-        prompt = f["rendered"]
-
-        # --- Header ---
-        assert prompt.startswith(
-            "You are a memory consolidation system. Synthesize new facts into "
-            "observations, merging with existing observations when appropriate."
-        )
-
-        # --- MISSION section: the supplied mission replaces the default ---
-        assert "## MISSION" in prompt
-        assert f["mission"] in prompt
-        assert "Track anything notable in the new facts" not in prompt, (
-            "default mission must be replaced when a custom one is supplied"
-        )
-
-        # --- Mission-priority note appears right after the mission ---
-        assert (
-            "If anything in this MISSION conflicts with the PROCESSING RULES, "
-            "DECISION GUIDE, or OUTPUT FORMAT below, the MISSION takes priority."
-        ) in prompt
-
-        # --- CAPACITY CONSTRAINT section: optional, should be present here ---
-        assert "## CAPACITY CONSTRAINT" in prompt
-        assert f["capacity_note"] in prompt
-        assert prompt.index("## MISSION") < prompt.index("## CAPACITY CONSTRAINT"), (
-            "CAPACITY CONSTRAINT must follow MISSION"
-        )
-
-        # --- Markdown section headers, in order ---
-        section_order = [
-            "## MISSION",
-            "## CAPACITY CONSTRAINT",
-            "## PROCESSING RULES",
-            "## INPUT",
-            "### New facts",
-            "### Existing observations",
-            "## DECISION GUIDE",
-            "## OUTPUT FORMAT",
-            "### Example 1 — Merging recurring claims into an existing observation",
-            "### Example 2 — State change updates one observation; unrelated fact creates a new one",
-            "### Observation text rules",
-            "### Field rules",
-        ]
-        last_idx = -1
-        for header in section_order:
-            idx = prompt.find(header)
-            assert idx != -1, f"section header missing: {header!r}"
-            assert idx > last_idx, f"section out of order: {header!r}"
-            last_idx = idx
-
-        # --- All 9 processing-rule headers must be present and ordered.
-        #     PREFER UPDATE OVER CREATE is now rule 1 (was rule 6) — this
-        #     is the central fix for issue #1566. ---
-        rule_markers = [
-            "1. PREFER UPDATE OVER CREATE",
-            "2. ONE OBSERVATION PER DISTINCT FACET",
-            "3. MATCH BY ENTITY/FACET, NOT TOPIC",
-            "4. STATE CHANGES — UPDATE CONCISELY",
-            "5. CASCADE TO ALL AFFECTED OBSERVATIONS",
-            "6. RESOLVE REFERENCES",
-            "7. PRESERVE HISTORY",
-            "8. NO COMPUTATION",
-            "9. KEEP DISTINCT TOPICS DISTINCT",
-        ]
-        last_idx = -1
-        for marker in rule_markers:
-            idx = prompt.find(marker)
-            assert idx != -1, f"processing rule marker missing: {marker!r}"
-            assert idx > last_idx, f"processing rule out of order: {marker!r}"
-            last_idx = idx
-
-        # --- New-facts subsection: every fact rendered with id + temporal parens ---
-        for nf in f["new_facts"]:
-            line = (
-                f"[{nf['id']}] {nf['text']} (occurred_start={nf['occurred_start']}, mentioned_at={nf['mentioned_at']})"
-            )
-            assert line in prompt, f"new fact line missing or malformed: {line!r}"
-
-        # --- Existing-observations subsection: both observations + their source memories ---
-        for obs in f["observations"]:
-            assert obs.id in prompt, f"observation id missing: {obs.id}"
-            assert obs.text in prompt, f"observation text missing: {obs.text!r}"
-        # source_memories block is included for each observation
-        assert '"source_memories"' in prompt
-        for sf in f["source_facts"].values():
-            assert sf.text in prompt, f"source fact text missing: {sf.text!r}"
-
-        # --- Both worked examples are present and the JSON renders correctly ---
-        assert '"creates": []' in prompt, "Example 1 demonstrates an UPDATE-only output"
-        assert "Alice works long hours" in prompt, "Example 2 create-side text present"
-        assert "Alice owned a 2019 Honda Civic; sold it" in prompt, "Example 2 state-change update text present"
-        # JSON braces in the examples must have been un-escaped by .format()
-        assert "{{" not in prompt and "}}" not in prompt, "literal {{ }} should have collapsed to { } after .format()"
-
-        # --- No unsubstituted format placeholders remain ---
-        for placeholder in ("{facts_text}", "{observations_text}"):
-            assert placeholder not in prompt, f"unsubstituted placeholder: {placeholder}"
-
-        # Dump the full prompt so a human can eyeball it under `pytest -s`.
-        print("\n" + "=" * 80)
-        print("FULL ASSEMBLED CONSOLIDATION PROMPT")
-        print("=" * 80)
-        print(prompt)
-        print("=" * 80)
-        print(f"length: {len(prompt)} chars")
-
-    def test_default_mission_appears_when_no_mission_supplied(self):
-        """Without an explicit mission the built-in default text must appear verbatim."""
-        from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
-
-        template = build_batch_consolidation_prompt()
-        rendered = template.format(facts_text="(none)", observations_text="[]")
-        assert (
-            "Track anything notable in the new facts — names, numbers, dates, "
-            "places, events, decisions, claims, relationships, and recurring patterns."
-        ) in rendered
-        # The mission-priority note must always be present so user-supplied
-        # missions can override the built-in rules when they conflict.
-        assert "the MISSION takes priority" in rendered
-        # The "at most one update per observation_id" rule must be present so
-        # the LLM doesn't emit colliding updates that silently overwrite each
-        # other (defensive fix for the horse-test misbehavior).
-        assert "AT MOST ONE UPDATE PER `observation_id`" in rendered
-        assert "## CAPACITY CONSTRAINT" not in rendered
+        response_model = llm_config.call.await_args.kwargs["response_format"]
+        assert response_model is _ConsolidationBatchResponse
+        assert len(result.creates) == expected_creates
 
 
 class TestDedupeUpdates:
@@ -3620,6 +3323,11 @@ def test_consolidation_prompt_split_is_cacheable_and_complete():
     assert "OBSERVATION LIMIT REACHED" in capped
     assert "OBSERVATION LIMIT REACHED" not in sys_prompt
 
+    # When no mission is supplied the built-in default mission is emitted in the
+    # per-batch input, so consolidation always has a mission to follow.
+    default = build_consolidation_input(facts_text="[id] F.", observations_text="[]")
+    assert "Track anything notable in the new facts" in default
+
 
 @pytest.mark.asyncio
 async def test_create_observation_populates_search_vector_native(memory, request_context):