Add Python client get_version helper

Adds HindsightClient.get_version()/aget_version() convenience wrappers for
the existing /version endpoint, re-exports VersionResponse for typed callers,
and tests both paths against a mocked MonitoringApi.

Python parity for #2252 (TypeScript getVersion). Fixes #2248.

.claude/skills/code-review/SKILL.md

@@ -192,6 +192,18 @@ If a migration adds a new PostgreSQL table (look for `CREATE TABLE` / `op.create
 - The guard test `test_backup_tables_covers_entire_schema` in `tests/test_admin_backup_restore.py` enforces this — flag it as a **must fix** if a new table is absent from `BACKUP_TABLES`.
 - Oracle-only tables (e.g. `observation_sources`) are intentionally excluded — admin backup/restore is PostgreSQL-only.
 
+### 11b. Check new config flags update the env template
+
+If the diff adds a new configuration field (a new `ENV_*` / `HINDSIGHT_*` env var
+in `hindsight-api-slim/hindsight_api/config.py`):
+- **`.env.example`** (repo root) — must add the variable (commented if optional)
+  alongside the docs entry in `hindsight-docs/docs/developer/configuration.md`.
+  A flag added to `config.py` but absent from `.env.example` is a **should fix**.
+- **`hindsight-embed/hindsight_embed/env.example`** — the bundled copy must stay
+  byte-identical to the repo-root `.env.example` (it seeds embed/profile configs).
+  The `test_bundled_template_matches_repo_root` sync test fails on drift; if the
+  root file changed without re-copying, flag it as a **must fix**.
+
 ### 12. Review against other coding standards
 
 Check the diff for violations of the standards listed above:

.claude/skills/hs-release/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: hs-release
+description: Cut a core Hindsight release (vX.Y.Z) and open the changelog + blog PR. Use when asked to cut/start a release, bump the version, or publish a new Hindsight version.
+user_invocable: true
+---
+
+# Hindsight Release
+
+Cut a **core** Hindsight release and open the accompanying changelog/blog PR. This is for the core
+product version (API, clients, CLI, control plane, Helm). **Integrations are versioned
+independently** — use `scripts/release-integration.sh` for those, not this skill.
+
+The release is **irreversible and outward-facing**: it tags a version and pushes it straight to
+`main`, which triggers CI that publishes packages to PyPI / npm / Helm. Confirm the version number
+and that the intended fixes are already merged to `main` before you start.
+
+## Step 0 — Pre-flight
+
+1. **Decide the base.** A release is cut from the latest `origin/main`, never from a feature
+   branch. `git fetch origin --tags` first. Confirm the "couple of fixes" the user means are
+   actually merged to `main` (`git log v<prev>..origin/main --oneline`).
+2. **Find where `main` is checked out.** `main` is often already checked out in a sibling worktree
+   (`git worktree list`). You **cannot** check out `main` in a second worktree — run the release in
+   the worktree that already holds it. If that worktree is dirty with throwaway cruft
+   (`.next-*` tsconfig paths, screenshots), `git stash push -u`, fast-forward to `origin/main`,
+   run the release, then `git stash pop`.
+3. **Pitfall:** never pipe the checkout in an `&&` chain like
+   `git checkout main 2>&1 | tail && git reset --hard ...` — the pipe's exit status is `tail`'s
+   (always 0), so a failed checkout won't stop the chain and the `reset` fires on the **wrong
+   branch**. Check out as its own command and verify `git branch --show-current` before resetting.
+
+## Step 1 — Cut the release
+
+Run from the worktree on a clean `main`:
+
+```bash
+./scripts/release.sh <version>     # e.g. 0.8.1  (no leading v)
+```
+
+`release.sh` bumps the version in every component, regenerates the OpenAPI spec + all client SDKs,
+updates docs versioning, commits `Release v<version>`, tags `v<version>`, and **pushes the commit
+and tag directly to `main`**. The push triggers the `Release` GitHub Actions workflow that builds
+and publishes the packages. It is **not** a PR.
+
+Verify after: `gh run list --limit 5` should show the `Release v<version>` workflow running, and
+`git ls-remote --tags origin v<version>` should return the tag.
+
+## Step 2 — Changelog + blog PR (separate)
+
+Done **after** the tag exists, as its own PR (precedent: v0.8.0 = #2053, v0.8.1 = #2080). Work on a
+branch off the new `main`:
+
+```bash
+git checkout -b docs-changelog-<version> origin/main
+```
+
+Only spin up a separate worktree (`git worktree add ../hindsight-changelog-<version> -b
+docs-changelog-<version> origin/main`) if you can't get a clean checkout otherwise — e.g. `main` is
+held in another worktree and the current one has work you don't want to disturb.
+
+**Branch naming:** use the `docs-` (hyphen) convention, e.g. `docs-changelog-0.8.1`. A remote
+branch literally named `docs` exists, so any `docs/...` branch is rejected on push with
+`directory file conflict`.
+
+### Changelog
+
+```bash
+uv run --directory hindsight-dev generate-changelog <version>
+```
+
+LLM-summarizes the commits between the previous tag and `v<version>` and prepends an entry to
+`hindsight-docs/src/pages/changelog/index.md`. Requires `OPENAI_API_KEY` (already in the repo
+`.env`). It excludes `hindsight-integrations/` source, but new integrations whose commits also
+touched docs will still appear — that matches precedent, leave them in the **changelog**.
+
+### Blog post
+
+Hand-write `hindsight-docs/blog/YYYY-MM-DD-version-X-Y-Z.md` (mirror an existing one; patch
+releases are short — see `2026-06-02-version-0-7-2.md`). Guidance:
+
+- **Explain user impact, not internals/mechanism.** Lead with what the user can now do and what to
+  set. Config/env-var names are fine (developer-facing), code symbols and internals are not.
+- **Do not list integrations in the release blog.** The core blog covers core engine / API /
+  ops changes; each integration ships its own changelog. (Integrations may still appear in the
+  generated `changelog/index.md` — that's fine; just keep them out of the blog.)
+- Call out an upgrade recommendation when there are operational/data-integrity fixes.
+- Validate formatting: `npx prettier --check <blog file>`.
+
+### Sync the docs skill
+
+```bash
+./scripts/generate-docs-skill.sh
+```
+
+Refreshes `skills/hindsight-docs/references/changelog/index.md`. It will also bump
+`skills/hindsight-docs/references/openapi.json` by one version — `release.sh` regenerates the skill
+*before* bumping OpenAPI, so the skill copy lags a version in the release commit; this step syncs
+it. Expect a one-line `version` diff there; keep it.
+
+### Commit, push, PR
+
+```bash
+git add -A
+git commit --no-verify -m "docs: changelog and blog post for v<version>"
+git push -u origin docs-changelog-<version>
+gh pr create --base main --title "docs: changelog and blog post for v<version>" --body "..."
+```
+
+Expected files in the PR: the changelog entry, the new blog post, the regenerated skill changelog
+mirror, and the skill `openapi.json` version sync.
+
+## Cleanup
+
+If you created a temporary worktree, remove it once the PR is up
+(`git worktree remove ../hindsight-changelog-<version>`; the branch stays on origin). Restore any
+stash you popped in Step 0.

.env.example

@@ -47,6 +47,13 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 HINDSIGHT_API_HOST=0.0.0.0
 HINDSIGHT_API_PORT=8888
 HINDSIGHT_API_LOG_LEVEL=info
+# Optional retain chunking override for structured logs/transcripts.
+# Unset uses HINDSIGHT_API_RETAIN_CHUNK_SIZE as the structured-chunk limit.
+# HINDSIGHT_API_RETAIN_STRUCTURED_CHUNK_SIZE=
+
+# Dry-run extraction preview endpoint (POST /memories/dry-run-extract). Enabled by default; it makes
+# a real LLM call but stores nothing. Set to false to remove the endpoint (returns 404).
+# HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT=true
 
 # Base Path / Reverse Proxy Support (Optional)
 # Set these when deploying behind a reverse proxy with path-based routing
@@ -59,6 +66,7 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_READ_DATABASE_URL=  # Optional read-replica URL. When set, recall queries (semantic, BM25, graph, temporal) flow through a separate pool against this URL, offloading the primary. Typically points to a read-only endpoint (CNPG's <cluster>-ro service or Aurora reader endpoint).
 # HINDSIGHT_API_MIGRATION_DATABASE_URL=  # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
 # HINDSIGHT_API_DATABASE_SCHEMA=public  # PostgreSQL schema name (default: public)
+# HINDSIGHT_API_MIGRATION_CONCURRENCY=1  # Tenant schemas to migrate concurrently (PG only, each in its own process; per-schema work stays sequential). Each worker has ~1-2s startup cost + uses ~3 DB connections, so it only pays off with many schemas (tens+) or slow migrations; keep concurrency*3 <= spare max_connections. Default: 1 (sequential).
 
 # Vector Extension (Optional - uses pgvector by default)
 # Options: "pgvector" (default), "vchord", "pgvectorscale" (DiskANN)

.github/workflows/release-integration.yml

@@ -9,7 +9,11 @@ jobs:
   publish:
     runs-on: ubuntu-latest
     permissions:
-      id-token: write  # for PyPI trusted publishing
+      id-token: write  # for PyPI trusted publishing + build-provenance attestations
+      attestations: write  # for actions/attest-build-provenance (Obsidian assets)
+      # No `contents: write`: we never create releases in this repo. The Obsidian
+      # plugin's distribution release is pushed to its dedicated repo using
+      # OBSIDIAN_DIST_TOKEN (see the "Mirror Obsidian plugin" step below).
 
     steps:
     - uses: actions/checkout@v6
@@ -112,6 +116,71 @@ jobs:
       working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
       run: npm run build
 
+    # Build-provenance attestations for the Obsidian release assets (community-store
+    # recommendation). Runs after the build so main.js exists. The assets are
+    # released in the dedicated repo while the build runs here, so users verify at
+    # owner scope: `gh attestation verify main.js --owner vectorize-io`.
+    - name: Attest Obsidian plugin build provenance
+      if: steps.type.outputs.type == 'typescript' && steps.info.outputs.integration == 'obsidian'
+      uses: actions/attest-build-provenance@v2
+      with:
+        subject-path: |
+          hindsight-integrations/obsidian/main.js
+          hindsight-integrations/obsidian/styles.css
+
+    # ── Obsidian plugin — mirror to its dedicated repo + cut the BRAT release ──
+    # We do NOT create a GitHub Release in this monorepo: per-integration
+    # releases pollute the repo's release list (it's for the core product) and
+    # steal the "Latest" badge, and BRAT / the community store read a repo's
+    # *latest* release — not a tag — so they can't target a tag in a monorepo.
+    #
+    # Instead this monorepo stays the source of truth, and on each obsidian
+    # release we mirror hindsight-integrations/obsidian/ → the *root* of
+    # github.com/vectorize-io/hindsight-obsidian (git subtree, history
+    # preserved) and cut the BRAT / community-store release *there*.
+    #
+    # Requires secret OBSIDIAN_DIST_TOKEN — a token with `contents: write` on
+    # vectorize-io/hindsight-obsidian (fine-grained PAT or app installation
+    # token). The dedicated repo is generated; never edit it directly.
+    - name: Mirror Obsidian plugin to its dedicated repo
+      if: steps.type.outputs.type == 'typescript' && steps.info.outputs.integration == 'obsidian'
+      env:
+        DIST_TOKEN: ${{ secrets.OBSIDIAN_DIST_TOKEN }}
+      run: |
+        set -euo pipefail
+        VERSION="${{ steps.info.outputs.version }}"
+        DIST_REPO="vectorize-io/hindsight-obsidian"
+        OBS_DIR="hindsight-integrations/obsidian"
+
+        # `git subtree split` needs full history; the default checkout is shallow.
+        git fetch --unshallow 2>/dev/null || true
+        git config user.name "github-actions[bot]"
+        git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+
+        # The runner injects the default GITHUB_TOKEN as an http.extraheader via
+        # an *included* config file (/home/runner/work/_temp/git-credentials-*.config),
+        # so `git config --local --unset-all` can't remove it and it authenticates
+        # the push as github-actions[bot] (no access to the dedicated repo → 403).
+        # The documented way to drop an inherited extraheader is to RESET the list
+        # with an empty value: since command-line `-c` is read last, the empty
+        # value clears the accumulated headers (including the included one) at
+        # request-build time. The dist token then comes from the push URL → a
+        # single Authorization header.
+        git subtree split --prefix="$OBS_DIR" -b _obs_dist
+        git -c "http.https://github.com/.extraheader=" \
+          push "https://x-access-token:${DIST_TOKEN}@github.com/${DIST_REPO}.git" _obs_dist:main
+
+        # Cut the BRAT / community-store release. Bare version tag (e.g. 0.1.0)
+        # to match manifest.json — idempotent so re-runs just refresh the assets.
+        export GH_TOKEN="$DIST_TOKEN"
+        ASSETS="$OBS_DIR/main.js $OBS_DIR/manifest.json $OBS_DIR/styles.css"
+        NOTES="Hindsight for Obsidian v${VERSION}. Install via BRAT (add ${DIST_REPO}) or copy main.js/manifest.json/styles.css into <vault>/.obsidian/plugins/hindsight/."
+        if gh release view "$VERSION" --repo "$DIST_REPO" >/dev/null 2>&1; then
+          gh release upload "$VERSION" $ASSETS --repo "$DIST_REPO" --clobber
+        else
+          gh release create "$VERSION" $ASSETS --repo "$DIST_REPO" --title "$VERSION" --notes "$NOTES"
+        fi
+
     - name: Publish TypeScript package to npm
       if: steps.type.outputs.type == 'typescript'
       working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
@@ -121,7 +190,12 @@ jobs:
         EXIT_CODE=$?
         echo "$OUTPUT"
         if [ $EXIT_CODE -ne 0 ]; then
-          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+          # Treat "already published" as success so re-pointed-tag re-runs stay green.
+          # "cannot publish over" = the version exists. TLOG_CREATE_ENTRY_ERROR / 409
+          # "equivalent entry already exists in the transparency log" = the identical
+          # --provenance artifact was already logged on a prior run (Sigstore tlog is
+          # idempotent); the package is published, so this is benign.
+          if echo "$OUTPUT" | grep -qE "cannot publish over|TLOG_CREATE_ENTRY_ERROR|already exists in the transparency log"; then
             echo "Package version already published, skipping..."
             exit 0
           fi

.github/workflows/test.yml

@@ -32,9 +32,14 @@ jobs:
       integration-tests: ${{ steps.filter.outputs.integration-tests }}
       integrations-openclaw: ${{ steps.filter.outputs.integrations-openclaw }}
       integrations-ai-sdk: ${{ steps.filter.outputs.integrations-ai-sdk }}
+      integrations-agent-framework: ${{ steps.filter.outputs.integrations-agent-framework }}
+      integrations-composio: ${{ steps.filter.outputs.integrations-composio }}
       integrations-chat: ${{ steps.filter.outputs.integrations-chat }}
       integrations-claude-code: ${{ steps.filter.outputs.integrations-claude-code }}
+      integrations-cline: ${{ steps.filter.outputs.integrations-cline }}
       integrations-codex: ${{ steps.filter.outputs.integrations-codex }}
+      integrations-continue: ${{ steps.filter.outputs.integrations-continue }}
+      integrations-cursor-cli: ${{ steps.filter.outputs.integrations-cursor-cli }}
       integrations-crewai: ${{ steps.filter.outputs.integrations-crewai }}
       integrations-litellm: ${{ steps.filter.outputs.integrations-litellm }}
       integrations-pydantic-ai: ${{ steps.filter.outputs.integrations-pydantic-ai }}
@@ -44,7 +49,9 @@ jobs:
       integrations-llamaindex: ${{ steps.filter.outputs.integrations-llamaindex }}
       integrations-paperclip: ${{ steps.filter.outputs.integrations-paperclip }}
       integrations-opencode: ${{ steps.filter.outputs.integrations-opencode }}
+      integrations-cursor: ${{ steps.filter.outputs.integrations-cursor }}
       integrations-n8n: ${{ steps.filter.outputs.integrations-n8n }}
+      integrations-zapier: ${{ steps.filter.outputs.integrations-zapier }}
       integrations-cloudflare-oauth-proxy: ${{ steps.filter.outputs.integrations-cloudflare-oauth-proxy }}
       integrations-superagent: ${{ steps.filter.outputs.integrations-superagent }}
       integrations-lockfiles: ${{ steps.filter.outputs.integrations-lockfiles }}
@@ -58,6 +65,9 @@ jobs:
       integrations-vapi: ${{ steps.filter.outputs.integrations-vapi }}
       integrations-flowise: ${{ steps.filter.outputs.integrations-flowise }}
       integrations-google-adk: ${{ steps.filter.outputs.integrations-google-adk }}
+      integrations-obsidian: ${{ steps.filter.outputs.integrations-obsidian }}
+      integrations-omo: ${{ steps.filter.outputs.integrations-omo }}
+      integrations-haystack: ${{ steps.filter.outputs.integrations-haystack }}
       tools-agent-sdk: ${{ steps.filter.outputs.tools-agent-sdk }}
       integrations-roo-code: ${{ steps.filter.outputs.integrations-roo-code }}
       dev: ${{ steps.filter.outputs.dev }}
@@ -118,12 +128,22 @@ jobs:
             - 'hindsight-integrations/openclaw/**'
           integrations-ai-sdk:
             - 'hindsight-integrations/ai-sdk/**'
+          integrations-agent-framework:
+            - 'hindsight-integrations/agent-framework/**'
+          integrations-composio:
+            - 'hindsight-integrations/composio/**'
           integrations-chat:
             - 'hindsight-integrations/chat/**'
           integrations-claude-code:
             - 'hindsight-integrations/claude-code/**'
+          integrations-cline:
+            - 'hindsight-integrations/cline/**'
           integrations-codex:
             - 'hindsight-integrations/codex/**'
+          integrations-continue:
+            - 'hindsight-integrations/continue/**'
+          integrations-cursor-cli:
+            - 'hindsight-integrations/cursor-cli/**'
           integrations-crewai:
             - 'hindsight-integrations/crewai/**'
           integrations-litellm:
@@ -138,12 +158,18 @@ jobs:
             - 'hindsight-integrations/langgraph/**'
           integrations-llamaindex:
             - 'hindsight-integrations/llamaindex/**'
+          integrations-haystack:
+            - 'hindsight-integrations/haystack/**'
           integrations-paperclip:
             - 'hindsight-integrations/paperclip/**'
           integrations-opencode:
             - 'hindsight-integrations/opencode/**'
+          integrations-cursor:
+            - 'hindsight-integrations/cursor/**'
           integrations-n8n:
             - 'hindsight-integrations/n8n/**'
+          integrations-zapier:
+            - 'hindsight-integrations/zapier/**'
           integrations-cloudflare-oauth-proxy:
             - 'hindsight-integrations/cloudflare-oauth-proxy/**'
           integrations-superagent:
@@ -172,6 +198,10 @@ jobs:
             - 'hindsight-integrations/flowise/**'
           integrations-google-adk:
             - 'hindsight-integrations/google-adk/**'
+          integrations-obsidian:
+            - 'hindsight-integrations/obsidian/**'
+          integrations-omo:
+            - 'hindsight-integrations/omo/**'
           tools-agent-sdk:
             - 'hindsight-tools/hindsight-agent-sdk/**'
           integrations-roo-code:
@@ -432,6 +462,95 @@ jobs:
       working-directory: ./hindsight-integrations/claude-code
       run: python -m pytest tests/ -v
 
+  test-cursor-integration:
+    needs: [detect-changes]
+    if: >-
+      github.event_name != 'pull_request_review' &&
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-cursor == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version: '3.11'
+
+    - name: Install pytest
+      run: pip install pytest
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/cursor
+      run: python -m pytest tests/ -v
+
+  test-omo-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-omo == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version: '3.11'
+
+    - name: Install pytest
+      run: pip install pytest
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/omo
+      run: python -m pytest tests/ -v
+
+  test-cline-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-cline == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build cline integration
+      working-directory: ./hindsight-integrations/cline
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/cline
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/cline
+      run: uv run pytest tests -v
+
   test-codex-integration:
     needs: [detect-changes]
     if: >-
@@ -458,6 +577,43 @@ jobs:
       working-directory: ./hindsight-integrations/codex
       run: python -m pytest tests/ -v
 
+  test-cursor-cli-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-cursor-cli == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build cursor-cli integration
+      working-directory: ./hindsight-integrations/cursor-cli
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/cursor-cli
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/cursor-cli
+      run: uv run pytest tests -v
+
   build-ai-sdk-integration:
     needs: [detect-changes]
     if: >-
@@ -583,6 +739,37 @@ jobs:
       working-directory: ./hindsight-integrations/n8n
       run: npm run build
 
+  test-zapier-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-zapier == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v6
+      with:
+        node-version: '22'
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/zapier
+      run: npm install --no-fund --no-audit
+
+    - name: Validate app definition
+      working-directory: ./hindsight-integrations/zapier
+      run: npm run validate
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/zapier
+      run: npm test
+
   test-hindsight-agent-sdk:
     needs: [detect-changes]
     if: >-
@@ -831,17 +1018,28 @@ jobs:
       with:
         ref: ${{ github.event.pull_request.head.sha || '' }}
 
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
     - name: Set up Python
       uses: actions/setup-python@v6
       with:
-        python-version: '3.11'
+        python-version-file: ".python-version"
 
-    - name: Install pytest
-      run: pip install pytest
+    - name: Build roo-code integration
+      working-directory: ./hindsight-integrations/roo-code
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/roo-code
+      run: uv sync --frozen
 
     - name: Run tests
       working-directory: ./hindsight-integrations/roo-code
-      run: python -m pytest tests/ -v
+      run: uv run pytest tests -v
 
   build-control-plane:
     needs: [detect-changes]
@@ -2862,6 +3060,88 @@ jobs:
       # (requires_real_llm) needs a live Hindsight server and runs separately.
       run: uv run pytest tests -v -m "not requires_real_llm"
 
+  test-composio-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-composio == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build composio integration
+      working-directory: ./hindsight-integrations/composio
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/composio
+      run: uv sync --frozen
+
+    - name: Lint
+      working-directory: ./hindsight-integrations/composio
+      run: uv run ruff check .
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/composio
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
+  test-continue-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-continue == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build continue integration
+      working-directory: ./hindsight-integrations/continue
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/continue
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/continue
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
   test-smolagents-integration:
     needs: [detect-changes]
     if: >-
@@ -2965,6 +3245,84 @@ jobs:
       working-directory: ./hindsight-integrations/flowise
       run: npm test
 
+  test-obsidian-integration:
+    needs: [detect-changes]
+    if: >-
+      github.event_name != 'pull_request_review' &&
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-obsidian == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v6
+      with:
+        node-version: '22'
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/obsidian
+      run: npm install --no-audit --no-fund
+
+    - name: Type check
+      working-directory: ./hindsight-integrations/obsidian
+      run: npx tsc --noEmit
+
+    - name: Build
+      working-directory: ./hindsight-integrations/obsidian
+      run: npm run build
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/obsidian
+      run: npm test
+
+  test-agent-framework-integration:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-agent-framework == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build agent-framework integration
+      working-directory: ./hindsight-integrations/agent-framework
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/agent-framework
+      run: uv sync --frozen
+
+    - name: Lint
+      working-directory: ./hindsight-integrations/agent-framework
+      run: uv run ruff check .
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/agent-framework
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
   test-crewai-integration:
     needs: [detect-changes]
     if: >-
@@ -3233,6 +3591,45 @@ jobs:
       # (requires_real_llm) needs a live Hindsight server and runs separately.
       run: uv run pytest tests -v -m "not requires_real_llm"
 
+  test-haystack-integration:
+    needs: [detect-changes]
+    if: >-
+      github.event_name != 'pull_request_review' &&
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-haystack == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build haystack integration
+      working-directory: ./hindsight-integrations/haystack
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/haystack
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/haystack
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
   test-openai-agents-integration:
     needs: [detect-changes]
     if: >-
@@ -3416,6 +3813,49 @@ jobs:
         echo "=== API Server Logs ==="
         cat /tmp/slim-api-server.log 2>/dev/null || true
 
+  verify-embed-control-center-bundle:
+    # The control center UI (Preact + Tailwind) is built with Vite and its static
+    # output is committed (served as-is by the embed's Python http.server, no Node
+    # at runtime). We can't byte-diff the committed bundle against a fresh build —
+    # Vite's content-hashed asset filenames aren't reproducible across the CI
+    # runner's OS/arch vs the committer's. So instead verify: (1) the committed
+    # bundle is a real, wired Vite build (index.html references JS/CSS that exist),
+    # and (2) the source still builds cleanly.
+    needs: [detect-changes]
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.embed == 'true' ||
+      needs.detect-changes.outputs.ci == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Setup Node
+      uses: actions/setup-node@v6
+      with:
+        node-version: '22'
+
+    # Check the committed bundle BEFORE building (the build overwrites static/).
+    - name: Verify the committed bundle is wired
+      working-directory: ./hindsight-embed/hindsight_embed/control_center
+      run: |
+        test -f static/index.html || { echo "::error::static/index.html missing — run 'npm run build' in control_center/ui and commit static/"; exit 1; }
+        js=$(grep -oE 'assets/[A-Za-z0-9_.-]+\.js' static/index.html | head -1)
+        css=$(grep -oE 'assets/[A-Za-z0-9_.-]+\.css' static/index.html | head -1)
+        { [ -n "$js" ] && [ -f "static/$js" ]; } || { echo "::error::index.html does not reference a committed JS bundle — rebuild the UI and commit static/"; exit 1; }
+        { [ -n "$css" ] && [ -f "static/$css" ]; } || { echo "::error::index.html does not reference a committed CSS bundle — rebuild the UI and commit static/"; exit 1; }
+        echo "committed bundle is wired ✓"
+
+    - name: Verify the source builds cleanly
+      working-directory: ./hindsight-embed/hindsight_embed/control_center/ui
+      run: |
+        npm ci
+        npm run build
+        echo "control center UI builds ✓"
+
   test-embed:
     needs: [detect-changes]
     if: >-
@@ -3631,7 +4071,8 @@ jobs:
         target="$RUNNER_TEMP/install-test"
         PYTHONPATH="$target" python -c "
         from hindsight_embed.daemon_embed_manager import DaemonEmbedManager
-        cmd = DaemonEmbedManager()._find_api_command()
+        # api_version is only used for the uvx fallback; the binary branch ignores it.
+        cmd = DaemonEmbedManager()._find_api_command('0.0.0')
         print('Resolved command:', cmd)
         assert len(cmd) == 1 and cmd[0].endswith('hindsight-api.exe'), (
             f'Expected sibling hindsight-api.exe, got {cmd!r}. '
@@ -3992,6 +4433,60 @@ jobs:
           fi
         done
 
+  # Dead-code detection beyond what ruff's F401/F841 catch (those are already
+  # BLOCKING via the ruff config + the verify-generated-files job).
+  #
+  # - knip (control plane): BLOCKING on unused files / dependencies / unlisted
+  #   dependencies. These are unambiguous — an orphaned file or a dead
+  #   package.json entry — so they fail the build.
+  # - vulture (Python) + knip unused *exports*: ADVISORY only. vulture's
+  #   function/argument heuristics false-positive on FastAPI/SQLAlchemy/Pydantic
+  #   patterns, and the control plane intentionally keeps an unused shadcn/ui
+  #   component surface, so these are surfaced in the step summary, not gated.
+  check-unused-code:
+    needs: [detect-changes]
+    if: >-
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.core == 'true' ||
+      needs.detect-changes.outputs.control-plane == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v6
+      with:
+        node-version: '20'
+        cache: 'npm'
+        cache-dependency-path: package-lock.json
+
+    - name: Install Control Plane dependencies
+      run: npm install --workspace=hindsight-control-plane
+
+    - name: knip — unused files / dependencies (blocking)
+      working-directory: hindsight-control-plane
+      run: npx --yes knip@5 --no-progress --include files,dependencies,unlisted
+
+    - name: Advisory scan — vulture + knip exports
+      continue-on-error: true
+      run: |
+        {
+          echo '## Dead-code scan (advisory)'
+          echo ''
+          echo '```'
+          ./scripts/hooks/check-unused.sh 2>&1 | sed 's/\x1b\[[0-9;]*m//g'
+          echo '```'
+        } | tee -a "$GITHUB_STEP_SUMMARY"
+
   verify-generated-files:
     runs-on: ubuntu-latest
     timeout-minutes: 30
@@ -4173,10 +4668,14 @@ jobs:
       - build-openclaw-integration
       - smoke-openclaw-install
       - test-claude-code-integration
+      - test-cursor-integration
+      - test-cline-integration
       - test-codex-integration
+      - test-cursor-cli-integration
       - build-ai-sdk-integration
       - test-ai-sdk-integration-deno
       - test-opencode-integration
+      - test-omo-integration
       - test-cloudflare-oauth-proxy-integration
       - build-chat-integration
       - test-paperclip-integration
@@ -4205,9 +4704,12 @@ jobs:
       - test-integration
       - test-ag2-integration
       - test-autogen-integration
+      - test-continue-integration
       - test-smolagents-integration
       - test-dify-integration
       - test-flowise-integration
+      - test-obsidian-integration
+      - test-agent-framework-integration
       - test-crewai-integration
       - test-langgraph-integration
       - test-superagent-integration
@@ -4216,9 +4718,11 @@ jobs:
       - test-llamaindex-integration
       - test-openai-agents-integration
       - test-agentcore-integration
+      - test-haystack-integration
       - test-pip-slim
       - test-embed
       - test-embed-windows
+      - verify-embed-control-center-bundle
       - test-hindsight-all
       - test-hindsight-agent-sdk
       - test-claude-agent-sdk-integration

.gitignore

@@ -15,6 +15,8 @@ node_modules/
 
 # Environment variables and local config
 .env
+.env.bak*
+.env.*.bak
 docker-compose.yml
 docker-compose.override.yml
 
@@ -59,4 +61,5 @@ hindsight-integrations/_drafts/
 # Changelog is now tracked in hindsight-docs/src/pages/changelog.md
 # CHANGELOG.md
 
-blog-post*
+blog-post*
+.worktrees/

CLAUDE.md

@@ -216,6 +216,18 @@ migration file dispatches through `run_for_dialect`, which calls either
 ./scripts/hooks/lint.sh
 ```
 
+Dead-code detection runs in CI (the `check-unused-code` job) at two levels:
+- **Blocking:** unused imports (ruff `F401`) and variables (`F841`) — `lint.sh` auto-removes
+  them and `verify-generated-files` fails on any leftover diff; and **knip** for orphaned
+  control-plane files / unused (or unlisted) `package.json` dependencies.
+- **Advisory:** whole unused Python functions (vulture) and unused control-plane *exports*
+  (the shadcn/ui surface is kept on purpose) — surfaced, not gated.
+
+Run both locally with:
+```bash
+./scripts/hooks/check-unused.sh
+```
+
 **After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
 
 **MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
@@ -315,7 +327,10 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
    ```
 
 2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
-   - Add field to the manual `HindsightConfig()` constructor call (search for "CLI override")
+   - No change is needed for ordinary environment-backed config fields. The CLI starts from `_get_raw_config()`,
+     so new `HindsightConfig` fields are carried through automatically.
+   - If the new field should be overridable by a CLI flag, add the argparse option in `_parse_cli_args()` and include
+     that field in the `dataclasses.replace(config, ...)` call near the "CLI override" comment.
 
 3. **Use hierarchical config in MemoryEngine**:
    ```python
@@ -335,6 +350,16 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
    - Add to appropriate section table with Variable, Description, Default
    - Mark if it's hierarchical (can be overridden per-bank)
 
+6. **Env template** (`.env.example`):
+   - Add the variable to the appropriate section, commented if optional, with a
+     short inline comment describing it (mirror the documentation entry).
+   - This file is the single source of truth for the env template:
+     `scripts/dev/setup.sh` copies it to `.env`, and `hindsight-embed` ships a
+     bundled copy (`hindsight-embed/hindsight_embed/env.example`) that seeds
+     embed/profile configs. After editing `.env.example`, re-copy it to the
+     embed package (`cp .env.example hindsight-embed/hindsight_embed/env.example`)
+     or the `test_bundled_template_matches_repo_root` sync test will fail.
+
 #### Hierarchical vs Static Guidelines
 
 **Hierarchical** (per-bank overridable):
@@ -351,7 +376,7 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
 
 ```bash
 cp .env.example .env
-# Edit .env with LLM API key
+# Edit .env with the LLM provider/model and credentials for your setup
 
 # Python deps
 uv sync --directory hindsight-api-slim/
@@ -360,10 +385,10 @@ uv sync --directory hindsight-api-slim/
 npm install
 ```
 
-Required env vars:
+Common LLM settings:
 - `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, minimax, ollama, lmstudio
-- `HINDSIGHT_API_LLM_API_KEY`: Your API key
-- `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., gpt-4o-mini, claude-sonnet-4-20250514)
+- `HINDSIGHT_API_LLM_API_KEY`: API key for providers that require one
+- `HINDSIGHT_API_LLM_MODEL`: Model name (defaults are provider-specific)
 
 Optional (uses local models by default):
 - `HINDSIGHT_API_EMBEDDINGS_PROVIDER`: local (default) or tei

README.md

@@ -7,7 +7,6 @@
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
 [![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![gitcgr](https://gitcgr.com/badge/vectorize-io/hindsight.svg)](https://gitcgr.com/vectorize-io/hindsight)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
 <br/>
@@ -143,6 +142,8 @@ main();
 pip install hindsight-all -U
 ```
 
+On Intel (x86_64) Macs, install `hindsight-all-slim` instead — see [Supported Platforms](#supported-platforms).
+
 ```python
 import os
 from hindsight import HindsightServer, HindsightClient
@@ -300,6 +301,19 @@ client.reflect(bank_id="my-bank", query="What should I know about Alice?")
 [![Star History Chart](https://api.star-history.com/svg?repos=vectorize-io/hindsight&type=date&legend=top-left)](https://www.star-history.com/#vectorize-io/hindsight&type=date&legend=top-left)
 ---
 
+## Supported Platforms
+
+| Platform | Docker | Bare Metal (pip) | Embedded DB (pg0) |
+|----------|--------|------------------|--------------------|
+| **Linux** (x86_64, ARM64) | ✅ | ✅ | ✅ |
+| **macOS** (Apple Silicon / arm64) | ✅ | ✅ | ✅ |
+| **macOS** (Intel / x86_64) | ✅ | ⚠️ | ✅ |
+| **Windows** (x86_64) | ✅ | ✅ | ✅ |
+
+⚠️ Intel Macs: use `hindsight-all-slim` — see the [installation guide](https://hindsight.vectorize.io/developer/installation#supported-platforms) for details.
+
+---
+
 ## Contributing
 
 See [CONTRIBUTING.md](./CONTRIBUTING.md).

deno.lock

@@ -77,7 +77,6 @@
             "npm:@radix-ui/react-dropdown-menu@^2.1.16",
             "npm:@radix-ui/react-label@^2.1.8",
             "npm:@radix-ui/react-popover@^1.1.15",
-            "npm:@radix-ui/react-radio-group@^1.3.8",
             "npm:@radix-ui/react-select@^2.2.6",
             "npm:@radix-ui/react-slider@^1.3.6",
             "npm:@radix-ui/react-slot@^1.2.4",

docker/docker-compose/vchord/docker-compose.yaml

@@ -1,6 +1,6 @@
 name: hindsight
 # Docker Compose file for Hindsight with PostgreSQL and vectorchord
-# docker compose -f docker/docker-compose/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/docker-compose.yaml up -d
+# docker compose -f docker/docker-compose/vchord/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/vchord/docker-compose.yaml up -d
 # Make sure to set the required environment variables before running:
 # - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
 # - Configure LLM provider variables as needed (see below in the hindsight service)

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.8.0
-appVersion: "0.8.0"
+version: 0.8.2
+appVersion: "0.8.2"
 keywords:
   - ai
   - memory

helm/hindsight/README.md

@@ -66,13 +66,13 @@ helm install hindsight ./helm/hindsight -n hindsight --create-namespace -f value
 
 | Parameter | Description | Default |
 |-----------|-------------|---------|
-| `version` | Default image tag for all components | `0.1.0` |
+| `version` | Default image tag for all components | Chart `appVersion` |
 | `api.enabled` | Enable the API component | `true` |
-| `api.image.repository` | API image repository | `hindsight/api` |
+| `api.image.repository` | API image repository | `ghcr.io/vectorize-io/hindsight-api` |
 | `api.image.tag` | API image tag (defaults to `version`) | - |
 | `api.service.port` | API service port | `8888` |
 | `controlPlane.enabled` | Enable the control plane | `true` |
-| `controlPlane.image.repository` | Control plane image repository | `hindsight/control-plane` |
+| `controlPlane.image.repository` | Control plane image repository | `ghcr.io/vectorize-io/hindsight-control-plane` |
 | `controlPlane.image.tag` | Control plane image tag (defaults to `version`) | - |
 | `controlPlane.service.port` | Control plane service port | `3000` |
 | `postgresql.enabled` | Deploy PostgreSQL as subchart | `true` |

helm/hindsight/values.yaml

@@ -13,9 +13,6 @@
 #   - Any other env vars you want to inject
 # existingSecret: "my-hindsight-secret"
 
-# Global settings
-replicaCount: 1
-
 # Image settings for api
 api:
   enabled: true

hindsight-all-npm/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vectorize-io/hindsight-all",
-  "version": "0.8.0",
+  "version": "0.8.2",
   "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",

hindsight-all-slim/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "hindsight-all-slim"
-version = "0.8.0"
+version = "0.8.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim==0.8.0",
+    "hindsight-api-slim==0.8.2",
     "hindsight-client>=0.0.7",
     "hindsight-embed>=0.1.0",
 ]

hindsight-all/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-all"
-version = "0.8.0"
+version = "0.8.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim[all]==0.8.0",
+    "hindsight-api-slim[all]==0.8.2",
     "hindsight-client>=0.0.7",
     "hindsight-embed>=0.1.0",
 ]
@@ -21,7 +21,7 @@ hindsight-embed = { workspace = true }
 
 [project.optional-dependencies]
 local-llm = [
-    "hindsight-api-slim[local-llm]==0.8.0",
+    "hindsight-api-slim[local-llm]==0.8.2",
 ]
 test = [
     "pytest>=7.0.0",

hindsight-api-slim/hindsight_api/__init__.py

@@ -53,4 +53,4 @@ __all__ = [
     "RemoteTEICrossEncoder",
     "LLMConfig",
 ]
-__version__ = "0.8.0"
+__version__ = "0.8.2"

hindsight-api-slim/hindsight_api/_vector_index.py

@@ -54,23 +54,20 @@ _INDEX_TYPE_KEYWORDS = {
 #   pre-dispatcher code (internal benchmarks tuned around our embedding count
 #   and recall floor; see the link_utils / pool init call sites for the
 #   latency-vs-recall framing).
-# - vchord exposes vchordrq.probes (no default; see VectorChord issue #392)
-#   and vchordrq.epsilon (default 1.9). probes = 10 / 30 are starting
-#   defaults pending a workload-specific sweep — vchordrq's recall curve
-#   shape differs from HNSW's, so the pgvector numbers don't translate
-#   directly. Revisit with a per-cluster benchmark once we have production
-#   recall data; until then these are deliberately conservative on the
-#   high-recall path. We leave epsilon at its default; tightening it is a
-#   separate trade-off.
+# - vchord exposes vchordrq.probes, but its shape must match the index's
+#   build.internal.lists hierarchy. VectorChord 1.1 added per-index fallback
+#   parameters for this reason: a session GUC overrides every vchordrq index,
+#   and a single value can be invalid for listless or mixed-layout indexes.
+#   Hindsight's built-in vchord clause does not set lists, so the safe default
+#   is no session-level probe override; deployments that partition vchordrq
+#   indexes should attach probes to the index storage parameters instead.
 # - pgvectorscale / pg_diskann / scann do not expose an equivalent per-statement
 #   knob in the engine today, so the dispatcher returns no statements for them.
 _ANN_TUNING_LOW_LATENCY: dict[str, tuple[tuple[str, str], ...]] = {
     "pgvector": (("hnsw.ef_search", "60"),),
-    "vchord": (("vchordrq.probes", "10"),),
 }
 _ANN_TUNING_HIGH_RECALL: dict[str, tuple[tuple[str, str], ...]] = {
     "pgvector": (("hnsw.ef_search", "200"),),
-    "vchord": (("vchordrq.probes", "30"),),
 }
 
 _EXTENSION_INSTALL_SQL = {

hindsight-api-slim/hindsight_api/admin/cli.py

@@ -49,6 +49,7 @@ BACKUP_TABLES = [
     "entities",
     "chunks",
     "memory_units",
+    "invalidated_memory_units",
     "unit_entities",
     "entity_cooccurrences",
     "memory_links",
@@ -257,12 +258,7 @@ async def _run_migration(
     embedding_dimension: int | None = None,
 ) -> list[str]:
     """Resolve database URL and run migrations for one schema or all discovered schemas."""
-    from ..migrations import (
-        ensure_embedding_dimension,
-        ensure_text_search_extension,
-        ensure_vector_extension,
-        run_migrations,
-    )
+    from ..migrations import run_migrations_for_schemas
 
     is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
@@ -283,32 +279,21 @@ async def _run_migration(
         # Preserve order while removing duplicates.
         schemas = list(dict.fromkeys(schemas))
 
-    for schema in schemas:
-        run_migrations(resolved_url, schema=schema, migration_database_url=config.migration_database_url)
-
-    if embedding_dimension is not None:
-        for schema in schemas:
-            ensure_embedding_dimension(
-                resolved_url,
-                embedding_dimension,
-                schema=schema,
-                vector_extension=config.vector_extension,
-            )
-
-    for schema in schemas:
-        ensure_vector_extension(
-            resolved_url,
-            vector_extension=config.vector_extension,
-            schema=schema,
-        )
-
-    for schema in schemas:
-        ensure_text_search_extension(
-            resolved_url,
-            text_search_extension=config.text_search_extension,
-            pg_search_tokenizer=config.text_search_extension_pg_search_tokenizer,
-            schema=schema,
-        )
+    # Migrate up to `migration_concurrency` schemas at once (each in its own
+    # process); within a schema the work stays sequential. Run off the event
+    # loop so the process pool's blocking joins don't stall it.
+    await asyncio.to_thread(
+        run_migrations_for_schemas,
+        resolved_url,
+        schemas,
+        concurrency=config.migration_concurrency,
+        migration_database_url=config.migration_database_url,
+        embedding_dimension=embedding_dimension,
+        vector_extension=config.vector_extension,
+        text_search_extension=config.text_search_extension,
+        pg_search_tokenizer=config.text_search_extension_pg_search_tokenizer,
+        ensure_extensions=True,
+    )
 
     return schemas
 

hindsight-api-slim/hindsight_api/alembic/versions/2071c7518f88_add_memory_links_bank_id_index.py

@@ -0,0 +1,105 @@
+"""Add a composite index on memory_links(bank_id, link_type) (PostgreSQL).
+
+``bank_id`` was added to ``memory_links`` in ``c5d6e7f8a9b0`` precisely so that
+bank-scoped reads (e.g. the stats endpoint) could filter on the link table
+directly instead of joining ``memory_units`` — that JOIN took 18+ seconds on
+banks with millions of links. The column landed without an index, so every
+``bank_id = $1`` predicate still falls back to a sequential scan over the whole
+table.
+
+This adds the missing btree. It is composite on ``(bank_id, link_type)`` rather
+than ``bank_id`` alone because the hot query is the stats endpoint's
+``SELECT link_type, COUNT(*) ... WHERE bank_id = $1 GROUP BY link_type``: a
+``(bank_id, link_type)`` index serves that filter, grouping and count as an
+index-only scan, never touching the heap, whereas a ``bank_id``-only index would
+still have to read every matching row to recover ``link_type``. ``link_type`` is
+low-cardinality (only ``temporal``/``semantic``/``caused_by`` are written —
+entity edges were dropped in ``e9b2c7d1f3a4``), so the trailing column adds
+little to the index size while removing the heap fetch.
+
+The Oracle baseline (``o1a2b3c4d5e6``) already creates ``idx_ml_bank_id`` on
+``memory_links(bank_id)``; that single-column index already covers Oracle's
+bank-scoped filter, so the Oracle slot here is intentionally absent and only the
+PostgreSQL dialect gets the composite index.
+
+``memory_links`` can hold tens of millions of rows, so the index is built
+CONCURRENTLY to avoid taking a write lock on the table. CONCURRENTLY cannot run
+inside a transaction block, so the statement runs in an ``autocommit_block()``;
+``IF NOT EXISTS`` keeps it idempotent across retries and re-migrated tenant
+schemas. A CONCURRENTLY build interrupted partway (lock conflict, disk
+pressure, signal) leaves the index behind as *invalid*; ``IF NOT EXISTS`` would
+then skip over it forever, so the upgrade first drops any invalid leftover of
+this name before (re)creating it.
+
+Revision ID: 2071c7518f88
+Revises: a1d3f5b7c9e2
+Create Date: 2026-06-16
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+from sqlalchemy import text
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "2071c7518f88"
+down_revision: str | Sequence[str] | None = "a1d3f5b7c9e2"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+_INDEX_NAME = "idx_memory_links_bank_id_link_type"
+
+
+def _get_schema_prefix() -> str:
+    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    bind = op.get_bind()
+    # `or None` collapses an unset option and an explicit empty string into NULL
+    # so the COALESCE below falls back to current_schema() in both cases.
+    target_schema = context.config.get_main_option("target_schema") or None
+    schema = _get_schema_prefix()
+
+    # CREATE INDEX CONCURRENTLY cannot run inside a transaction block; the
+    # autocommit_block runs each statement outside Alembic's migration
+    # transaction.
+    with op.get_context().autocommit_block():
+        # A CONCURRENTLY build that errored on a previous run leaves an INVALID
+        # index of this name behind. `CREATE INDEX ... IF NOT EXISTS` would see
+        # that relation and skip, so bank_id queries would keep seq-scanning.
+        # Drop only the invalid leftover — never a healthy index — so the retry
+        # actually rebuilds a usable one.
+        leftover_invalid = bind.execute(
+            text(
+                "SELECT NOT i.indisvalid "
+                "FROM pg_class c "
+                "JOIN pg_index i ON c.oid = i.indexrelid "
+                "JOIN pg_namespace n ON c.relnamespace = n.oid "
+                "WHERE c.relname = :index_name "
+                "  AND n.nspname = COALESCE(:target_schema, current_schema())"
+            ),
+            {"index_name": _INDEX_NAME, "target_schema": target_schema},
+        ).scalar()
+        if leftover_invalid:
+            op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}{_INDEX_NAME}")
+
+        # IF NOT EXISTS keeps the create idempotent across retries and schemas.
+        op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {_INDEX_NAME} ON {schema}memory_links(bank_id, link_type)")
+
+
+def _pg_downgrade() -> None:
+    schema = _get_schema_prefix()
+    with op.get_context().autocommit_block():
+        op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}{_INDEX_NAME}")
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/a1d3f5b7c9e2_widen_remaining_bank_id_to_text.py

@@ -0,0 +1,85 @@
+"""Repair: widen the remaining live ``bank_id`` columns from VARCHAR(64) to TEXT on PostgreSQL.
+
+Follow-up to ``c3e5a7b9d1f4`` (issue #2106), which widened the two *history*
+tables (``observation_history``, ``mental_model_history``) to ``TEXT`` after the
+narrow ``VARCHAR(64)`` declaration bricked startup. The same VARCHAR(64) / TEXT
+inconsistency still affects the live tables that store a user-supplied
+``bank_id``:
+
+* ``directives``    -- created VARCHAR(64) in ``p1k2l3m4n5o6``
+* ``mental_models`` -- VARCHAR(64) (origin ``pinned_reflections`` in
+                       ``n9i0j1k2l3m4``; recreated in ``h3c4d5e6f7g8``)
+
+``mental_model_versions`` is intentionally *not* widened here: it is created in
+``j5e6f7g8h9i0`` but dropped (``DROP TABLE ... CASCADE``) in ``o0j1k2l3m4n5`` and
+never recreated on the upgrade path, so it does not exist at head. Issuing
+``ALTER TABLE mental_model_versions ...`` would raise ``UndefinedTable`` and --
+because migrations run inside the lifespan-startup transaction -- roll the whole
+migration back, bricking the API. (It is unrelated to the live
+``mental_model_history`` table widened by ``c3e5a7b9d1f4``.)
+
+``banks.bank_id`` is ``TEXT`` (unbounded), so a deployment can create a bank
+whose id exceeds 64 chars -- the 78-char hierarchical org-unit shape reported in
+issue #2106 -- and the bank insert succeeds. The next write that propagates that
+id (``create_directive``, ``create_mental_model`` / consolidation, or
+mental-model versioning) then aborts with::
+
+    psycopg2.errors.StringDataRightTruncation: value too long for type
+    character varying(64)
+
+i.e. a 500 on core write endpoints, instead of the startup brick that
+``c3e5a7b9d1f4`` already repaired.
+
+``ALTER COLUMN ... TYPE TEXT`` is a no-op on a column that is already ``TEXT``,
+so every upgrade path converges on ``TEXT``. These tables are per-tenant (they
+live in each tenant schema, not ``public``), so this runs for every migrated
+schema via the search-path-aware prefix -- the same mechanism as
+``c3e5a7b9d1f4``.
+
+PostgreSQL only: these tables are created by PostgreSQL-only migrations
+(``run_for_dialect(pg=...)``); on Oracle they are absent or already
+``VARCHAR2(256)`` (consistent, never truncates), so the Oracle slot is
+intentionally absent -- mirroring ``c3e5a7b9d1f4``.
+
+Revision ID: a1d3f5b7c9e2
+Revises: c3e5a7b9d1f4
+Create Date: 2026-06-13
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "a1d3f5b7c9e2"
+down_revision: str | Sequence[str] | None = "c3e5a7b9d1f4"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    schema = _get_schema_prefix()
+    op.execute(f"ALTER TABLE {schema}directives ALTER COLUMN bank_id TYPE TEXT")
+    op.execute(f"ALTER TABLE {schema}mental_models ALTER COLUMN bank_id TYPE TEXT")
+
+
+def _pg_downgrade() -> None:
+    # No-op: narrowing back to VARCHAR(64) could truncate real data and would
+    # re-introduce the bug this migration repairs. The column types are owned by
+    # the migrations that created the tables.
+    pass
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/a7b8c9d0e1f2_split_history_into_own_tables.py

@@ -62,7 +62,7 @@ def _pg_upgrade() -> None:
         CREATE TABLE IF NOT EXISTS {schema}mental_model_history (
             id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
             mental_model_id VARCHAR(64) NOT NULL,
-            bank_id VARCHAR(64) NOT NULL,
+            bank_id TEXT NOT NULL,
             content JSONB NOT NULL,
             changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
             FOREIGN KEY (mental_model_id, bank_id)
@@ -83,7 +83,7 @@ def _pg_upgrade() -> None:
         CREATE TABLE IF NOT EXISTS {schema}observation_history (
             id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
             observation_id UUID NOT NULL,
-            bank_id VARCHAR(64) NOT NULL,
+            bank_id TEXT NOT NULL,
             content JSONB NOT NULL,
             changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
             FOREIGN KEY (observation_id)

hindsight-api-slim/hindsight_api/alembic/versions/b2d4f6a8c1e3_repair_maintenance_routines_public.py

@@ -0,0 +1,156 @@
+"""Repair: install maintenance routines on the ``public`` / base-schema run.
+
+The original maintenance-routines migration (``e5f6a7b8c9d0``) only created the
+shared ``public.banks_needing_consolidation()`` and
+``public.schemas_with_expired_rows(...)`` routines when the run had *no*
+``target_schema`` at all. But the single-tenant runtime always migrates an
+explicit schema — which defaults to ``public`` — so on every default
+PostgreSQL deployment the migration was stamped as applied while the functions
+were never created. Background maintenance then logs::
+
+    Retention sweep failed for llm_requests: function public.schemas_with_expired_rows(...) does not exist
+    Consolidation reconcile discovery failed: function public.banks_needing_consolidation() does not exist
+
+See https://github.com/vectorize-io/hindsight/issues/2056.
+
+Because ``e5f6a7b8c9d0`` is already stamped on affected ``0.8.0`` databases,
+editing it would not re-run it there. This forward migration re-installs the
+functions idempotently (``CREATE OR REPLACE``) on the run that targets the
+shared ``public`` schema (base run with no ``target_schema``, or an explicit
+``target_schema=public``), self-healing already-upgraded deployments and
+covering fresh upgrades from earlier versions.
+
+Per-tenant runs against a non-``public`` schema still skip it: re-issuing
+``CREATE OR REPLACE FUNCTION public....`` from each concurrent tenant migration
+aborts with ``tuple concurrently updated`` on the ``pg_proc`` catalog row, and
+the base/public run has already created the functions for every tenant to use.
+Runs that target ``public`` are serialized by the per-schema migration advisory
+lock, so only one wins the create.
+
+PostgreSQL only — the worker poller and these tables are not wired for Oracle,
+so the Oracle slot is intentionally absent (mirrors ``e5f6a7b8c9d0``).
+
+Revision ID: b2d4f6a8c1e3
+Revises: e5f6a7b8c9d0
+Create Date: 2026-06-08
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "b2d4f6a8c1e3"
+down_revision: str | Sequence[str] | None = "e5f6a7b8c9d0"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _should_install_public_routines(target_schema: str | None) -> bool:
+    """True for the run that must (re)create the shared ``public.*`` routines.
+
+    The routines physically live in ``public`` (hard-coded ``public.`` qualifier
+    in the SQL below), so they must be installed exactly once — on the base run
+    (no ``target_schema``) or on the run that explicitly targets ``public``. A
+    run against any other tenant schema skips it to avoid concurrent
+    ``CREATE OR REPLACE`` on the same ``pg_proc`` row.
+    """
+    return not target_schema or target_schema == "public"
+
+
+def _pg_upgrade() -> None:
+    if not _should_install_public_routines(context.config.get_main_option("target_schema")):
+        return
+    # Banks with eligible-but-unscheduled facts and no in-flight consolidation.
+    # Auto-consolidation is filtered here only at the bank level (cheap prune);
+    # the full hierarchical resolution (global -> tenant -> bank, plus
+    # enable_observations) is done by the caller for the small returned set.
+    op.execute(
+        """
+        CREATE OR REPLACE FUNCTION public.banks_needing_consolidation()
+        RETURNS TABLE(schema_name text, bank_id text)
+        LANGUAGE plpgsql STABLE
+        AS $fn$
+        DECLARE
+            sch text;
+        BEGIN
+            FOR sch IN
+                SELECT n.nspname
+                FROM pg_class c
+                JOIN pg_namespace n ON n.oid = c.relnamespace
+                WHERE c.relname = 'memory_units' AND c.relkind = 'r'
+            LOOP
+                RETURN QUERY EXECUTE format($q$
+                    SELECT %1$L::text, m.bank_id
+                    FROM %1$I.memory_units m
+                    JOIN %1$I.banks b ON b.bank_id = m.bank_id
+                    WHERE m.consolidated_at IS NULL
+                      AND m.consolidation_failed_at IS NULL
+                      AND m.fact_type IN ('experience', 'world')
+                      AND COALESCE(b.config -> 'enable_auto_consolidation', 'true'::jsonb) <> 'false'::jsonb
+                      AND NOT EXISTS (
+                          SELECT 1 FROM %1$I.async_operations o
+                          WHERE o.bank_id = m.bank_id
+                            AND o.operation_type = 'consolidation'
+                            AND o.status IN ('pending', 'processing')
+                      )
+                    GROUP BY m.bank_id
+                $q$, sch);
+            END LOOP;
+        END;
+        $fn$;
+        """
+    )
+
+    # Schemas holding at least one row of p_table older than p_days. p_ts_col is
+    # the timestamp column to compare. Returns nothing when p_days <= 0
+    # (retention disabled).
+    op.execute(
+        """
+        CREATE OR REPLACE FUNCTION public.schemas_with_expired_rows(
+            p_table text, p_ts_col text, p_days int
+        )
+        RETURNS SETOF text
+        LANGUAGE plpgsql STABLE
+        AS $fn$
+        DECLARE
+            sch text;
+            has_expired boolean;
+        BEGIN
+            IF p_days IS NULL OR p_days <= 0 THEN
+                RETURN;
+            END IF;
+            FOR sch IN
+                SELECT n.nspname
+                FROM pg_class c
+                JOIN pg_namespace n ON n.oid = c.relnamespace
+                WHERE c.relname = p_table AND c.relkind = 'r'
+            LOOP
+                EXECUTE format(
+                    'SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I < NOW() - make_interval(days => $1))',
+                    sch, p_table, p_ts_col
+                ) INTO has_expired USING p_days;
+                IF has_expired THEN
+                    RETURN NEXT sch;
+                END IF;
+            END LOOP;
+        END;
+        $fn$;
+        """
+    )
+
+
+def _pg_downgrade() -> None:
+    # No-op: ``e5f6a7b8c9d0`` owns the lifecycle of these functions and drops
+    # them on its own downgrade. This migration only ever (re)creates them, so
+    # there is nothing to undo without racing that migration's DROP.
+    pass
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c3e5a7b9d1f4_widen_history_bank_id_to_text.py

@@ -0,0 +1,75 @@
+"""Repair: widen ``*_history.bank_id`` from VARCHAR(64) to TEXT on PostgreSQL.
+
+The original split-history migration (``a7b8c9d0e1f2``) declared
+``observation_history.bank_id`` and ``mental_model_history.bank_id`` as
+``VARCHAR(64)`` on PostgreSQL. But ``memory_units.bank_id`` — the backfill
+source for observations — is ``TEXT`` (unbounded), as are ``banks``,
+``documents`` and ``entities``. Any deployment whose ``bank_id`` exceeds 64
+characters aborts the backfill ``INSERT`` with::
+
+    psycopg2.errors.StringDataRightTruncation: value too long for type
+    character varying(64)
+
+Because the migration runs in ``lifespan`` startup inside a transaction, the
+whole migration rolls back and the API never comes up — unrecoverable from the
+running container. See https://github.com/vectorize-io/hindsight/issues/2106.
+
+``a7b8c9d0e1f2`` itself has been corrected to create the column as ``TEXT``,
+which unblocks deployments that *failed* (the migration rolled back, so it
+re-runs the fixed DDL). This forward migration covers deployments that already
+*succeeded* with the narrow ``VARCHAR(64)`` column — where editing
+``a7b8c9d0e1f2`` has no effect because it will not re-run — by widening the
+column in place. ``ALTER COLUMN ... TYPE TEXT`` is a no-op on a column that is
+already ``TEXT`` (fresh installs and re-run failures), so every upgrade path
+converges on ``TEXT``.
+
+The history tables are per-tenant (they live in each tenant schema, not
+``public``), so this runs for every migrated schema via the search-path-aware
+prefix — unlike the shared-``public`` routines repaired in ``b2d4f6a8c1e3``.
+
+PostgreSQL only. On Oracle both ``memory_units.bank_id`` and the history
+``bank_id`` columns are already ``VARCHAR2(256)`` (consistent, never
+truncates), so the Oracle slot is intentionally absent.
+
+Revision ID: c3e5a7b9d1f4
+Revises: c9a1b2d3e4f5
+Create Date: 2026-06-10
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "c3e5a7b9d1f4"
+down_revision: str | Sequence[str] | None = "c9a1b2d3e4f5"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    schema = _get_schema_prefix()
+    op.execute(f"ALTER TABLE {schema}observation_history ALTER COLUMN bank_id TYPE TEXT")
+    op.execute(f"ALTER TABLE {schema}mental_model_history ALTER COLUMN bank_id TYPE TEXT")
+
+
+def _pg_downgrade() -> None:
+    # No-op: narrowing back to VARCHAR(64) could truncate real data and would
+    # re-introduce the bug this migration repairs. The column type is owned by
+    # ``a7b8c9d0e1f2``'s lifecycle.
+    pass
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c9a1b2d3e4f5_add_invalidated_memory_units.py

@@ -0,0 +1,108 @@
+"""Add invalidated_memory_units table for curation (edit/invalidate).
+
+Curation keeps the recall hot-path (``memory_units``) clean by *moving*
+invalidated facts into a sibling archive table rather than flagging them in
+place. If a row is in ``memory_units`` it is live; if it is in
+``invalidated_memory_units`` it has been retired. Recall/consolidation/graph
+queries never need a state predicate — the rows simply aren't there.
+
+The archive mirrors ``memory_units`` column-for-column — except ``embedding``,
+which it never keeps: the archive is cold storage, never a recall surface, and
+revert recomputes the embedding from the unit's text/dates/entities. Keeping no
+archive vector also means a later embedding-model switch (which re-dimensions
+``memory_units``) can't trip a dimension mismatch on the move (#2209). Plus:
+- ``invalidation_reason``  optional free text recorded on invalidate
+- ``invalidated_at``       when it was retired
+- ``entity_ids``           snapshot of the unit's entity associations, so revert
+                           can restore them (``unit_entities`` is cascade-deleted
+                           when the live row is removed)
+
+This migration also adds ``edited_at`` to ``memory_units``: set whenever a user
+edits a memory's fields (text, context, dates, fact_type, entities) via curation.
+NULL means never manually modified; a non-NULL value answers "has the user ever
+changed this?" with the time of the last edit (distinct from ``updated_at``,
+which background operations also bump). It is added to ``memory_units`` *before*
+the archive is cloned below, so the archive inherits the column and the marker
+travels with a fact when it is invalidated.
+
+Revision ID: c9a1b2d3e4f5
+Revises: b2d4f6a8c1e3
+Create Date: 2026-06-03
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "c9a1b2d3e4f5"
+down_revision: str | Sequence[str] | None = "b2d4f6a8c1e3"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _pg_schema_prefix() -> str:
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    schema = _pg_schema_prefix()
+    # Add edited_at to the live table FIRST so the archive's LIKE clone below
+    # inherits it (keeps the two tables column-for-column identical for round-trip).
+    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS edited_at TIMESTAMPTZ")
+    # LIKE ... INCLUDING DEFAULTS clones every memory_units column (incl.
+    # edited_at) so an invalidated row can move back verbatim. We deliberately
+    # omit indexes/constraints — the archive is cold storage, not a recall
+    # surface; only the lookups below need indexing.
+    op.execute(
+        f"CREATE TABLE IF NOT EXISTS {schema}invalidated_memory_units (LIKE {schema}memory_units INCLUDING DEFAULTS)"
+    )
+    # ...then drop the inherited embedding: the archive never stores one (revert
+    # recomputes it), so it isn't created here only to be dropped again later by
+    # d4f6a8c2e1b3. That migration still runs as a no-op (DROP ... IF EXISTS) on
+    # fresh DBs and does the real drop on DBs created before this column was removed.
+    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS embedding")
+    op.execute(
+        f"ALTER TABLE {schema}invalidated_memory_units "
+        f"ADD COLUMN IF NOT EXISTS invalidation_reason TEXT, "
+        f"ADD COLUMN IF NOT EXISTS invalidated_at TIMESTAMPTZ DEFAULT now(), "
+        f"ADD COLUMN IF NOT EXISTS entity_ids UUID[]"
+    )
+    op.execute(f"CREATE UNIQUE INDEX IF NOT EXISTS idx_invalidated_mu_id ON {schema}invalidated_memory_units (id)")
+    op.execute(
+        f"CREATE INDEX IF NOT EXISTS idx_invalidated_mu_bank "
+        f"ON {schema}invalidated_memory_units (bank_id, invalidated_at)"
+    )
+    # Deleting a document (or bank) should clear its archived facts too, mirroring
+    # the memory_units → documents cascade.
+    op.execute(
+        f"""
+        DO $$ BEGIN
+        IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'invalidated_mu_document_fkey') THEN
+            ALTER TABLE {schema}invalidated_memory_units
+                ADD CONSTRAINT invalidated_mu_document_fkey
+                FOREIGN KEY (document_id, bank_id)
+                REFERENCES {schema}documents(id, bank_id) ON DELETE CASCADE;
+        END IF; END $$;
+        """
+    )
+
+
+def _pg_downgrade() -> None:
+    schema = _pg_schema_prefix()
+    # Drops the archive (and its inherited edited_at) wholesale, then removes
+    # edited_at from the live table.
+    op.execute(f"DROP TABLE IF EXISTS {schema}invalidated_memory_units")
+    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS edited_at")
+
+
+def upgrade() -> None:
+    # PG-only: Oracle gets the table from the baseline snapshot, matching the
+    # convention used by sibling column/index migrations in this tree.
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/d4f6a8c2e1b3_drop_archive_embedding_column.py

@@ -0,0 +1,93 @@
+"""Drop the embedding column from the curation archive (invalidated_memory_units).
+
+The archive is cold storage, never a recall surface, so it has no business
+keeping an embedding. Earlier curation code copied the live row's embedding into
+``invalidated_memory_units`` on invalidate; the engine now leaves it out on
+invalidate and recomputes it on revert, so the column is dead weight.
+
+Dropping it makes "the archive holds no embedding" a schema-enforced invariant
+rather than a convention the move queries have to honour, and removes a latent
+failure mode (#2209): after an embedding-model switch the live tables are
+re-dimensioned but the archive was not, so a stale old-dimension embedding in
+the archive tripped a vector-dimension mismatch on the INSERT … SELECT
+round-trip. With no column at all, there is nothing to mismatch.
+
+The creation sites no longer add the column (the PG ``LIKE`` clone in
+c9a1b2d3e4f5 drops it; the Oracle baseline omits it), so on a fresh database
+this migration is a no-op (DROP ... IF EXISTS / Oracle ORA-00904 swallow). It
+does the real work on databases created before the column was removed there.
+
+DROP COLUMN is a metadata-only operation on both PostgreSQL and Oracle 23ai (no
+table rewrite), so it is cheap even across many tenant schemas. The downgrade
+re-adds an unconstrained vector column (any dimension) — empty, since the
+embeddings are intentionally discarded.
+
+Revision ID: d4f6a8c2e1b3
+Revises: a1d3f5b7c9e2
+Create Date: 2026-06-15
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "d4f6a8c2e1b3"
+down_revision: str | Sequence[str] | None = "a1d3f5b7c9e2"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _pg_schema_prefix() -> str:
+    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _pg_upgrade() -> None:
+    schema = _pg_schema_prefix()
+    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS embedding")
+
+
+def _pg_downgrade() -> None:
+    schema = _pg_schema_prefix()
+    # Unconstrained `vector` (no dimension) so the re-added column accepts any
+    # model's embeddings; it comes back empty regardless.
+    op.execute(f"ALTER TABLE {schema}invalidated_memory_units ADD COLUMN IF NOT EXISTS embedding vector")
+
+
+def _oracle_upgrade() -> None:
+    # Oracle has no `DROP COLUMN IF EXISTS`; swallow ORA-00904 (column does not
+    # exist) so the migration is idempotent and safe on a fresh schema whose
+    # baseline already omits the column.
+    op.execute(
+        """
+        BEGIN
+            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units DROP COLUMN embedding';
+        EXCEPTION WHEN OTHERS THEN
+            IF SQLCODE != -904 THEN RAISE; END IF;
+        END;
+        """
+    )
+
+
+def _oracle_downgrade() -> None:
+    # Swallow ORA-01430 (column already exists) for idempotency.
+    op.execute(
+        """
+        BEGIN
+            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units ADD (embedding VECTOR)';
+        EXCEPTION WHEN OTHERS THEN
+            IF SQLCODE != -1430 THEN RAISE; END IF;
+        END;
+        """
+    )
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/e1f2a3b4c5d6_merge_heads_embedding_drop_and_links_index.py

@@ -0,0 +1,39 @@
+"""Merge two divergent migration heads.
+
+``d4f6a8c2e1b3`` (drop the curation-archive embedding column) and
+``2071c7518f88`` (add the memory_links(bank_id, link_type) index) were authored
+in parallel off the same parent (``a1d3f5b7c9e2``) and merged independently,
+leaving the DAG with two heads. This is a no-op merge that re-unifies them so
+``alembic upgrade head`` is unambiguous again (enforced by
+``tests/test_alembic_dag.py::test_single_head``).
+
+Revision ID: e1f2a3b4c5d6
+Revises: d4f6a8c2e1b3, 2071c7518f88
+Create Date: 2026-06-16
+"""
+
+from collections.abc import Sequence
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "e1f2a3b4c5d6"
+down_revision: str | Sequence[str] | None = ("d4f6a8c2e1b3", "2071c7518f88")
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _pg_upgrade() -> None:
+    # Pure DAG merge — both parents already applied their schema changes.
+    pass
+
+
+def _pg_downgrade() -> None:
+    pass
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/o1a2b3c4d5e6_oracle_baseline.py

@@ -122,6 +122,7 @@ _TABLES: tuple[str, ...] = (
         text_signals      CLOB,
         consolidation_failed_at TIMESTAMP WITH TIME ZONE,
         search_vector     CLOB,
+        edited_at         TIMESTAMP WITH TIME ZONE,
         created_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
         updated_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
         CONSTRAINT pk_memory_units PRIMARY KEY (id),
@@ -138,6 +139,50 @@ _TABLES: tuple[str, ...] = (
     PARTITION BY LIST (bank_id) AUTOMATIC
     (PARTITION p_default VALUES ('__default__'))
     """,
+    # Cold archive for curation: invalidated facts are MOVED here out of
+    # memory_units so the recall hot-path never sees them. Mirrors memory_units
+    # plus invalidation bookkeeping and an entity-id snapshot for lossless revert.
+    # No `embedding` column: the archive is cold storage and revert recomputes the
+    # embedding, so there is no archive vector to fall out of sync with the live
+    # model's dimension on a model switch (#2209).
+    """
+    CREATE TABLE IF NOT EXISTS invalidated_memory_units (
+        id                RAW(16)        NOT NULL,
+        bank_id           VARCHAR2(256)  NOT NULL,
+        document_id       VARCHAR2(512),
+        chunk_id          VARCHAR2(512),
+        text              CLOB           NOT NULL,
+        context           CLOB,
+        event_date        TIMESTAMP WITH TIME ZONE NOT NULL,
+        occurred_start    TIMESTAMP WITH TIME ZONE,
+        occurred_end      TIMESTAMP WITH TIME ZONE,
+        mentioned_at      TIMESTAMP WITH TIME ZONE,
+        fact_type         VARCHAR2(64)   DEFAULT 'world' NOT NULL,
+        confidence_score  BINARY_DOUBLE,
+        access_count      NUMBER(10)     DEFAULT 0 NOT NULL,
+        consolidated_at   TIMESTAMP WITH TIME ZONE,
+        observation_scopes CLOB          CONSTRAINT imu_obs_scopes_json CHECK (observation_scopes IS JSON OR observation_scopes IS NULL),
+        tags              CLOB           DEFAULT '[]' NOT NULL,
+        metadata          CLOB           DEFAULT '{}' NOT NULL
+                                         CONSTRAINT imu_metadata_json CHECK (metadata IS JSON),
+        proof_count       NUMBER(10)     DEFAULT 1,
+        source_memory_ids CLOB,
+        history           CLOB           DEFAULT '[]'
+                                         CONSTRAINT imu_history_json CHECK (history IS JSON OR history IS NULL),
+        text_signals      CLOB,
+        consolidation_failed_at TIMESTAMP WITH TIME ZONE,
+        search_vector     CLOB,
+        edited_at         TIMESTAMP WITH TIME ZONE,
+        created_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+        updated_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
+        invalidation_reason CLOB,
+        invalidated_at    TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP,
+        entity_ids        CLOB           CONSTRAINT imu_entity_ids_json CHECK (entity_ids IS JSON OR entity_ids IS NULL),
+        CONSTRAINT pk_invalidated_memory_units PRIMARY KEY (id),
+        CONSTRAINT fk_imu_document FOREIGN KEY (document_id, bank_id)
+            REFERENCES documents(id, bank_id) ON DELETE CASCADE
+    )
+    """,
     """
     CREATE TABLE IF NOT EXISTS entities (
         id                RAW(16)        DEFAULT SYS_GUID() NOT NULL,

hindsight-api-slim/hindsight_api/alembic/versions/x9s0t1u2v3w4_add_bank_config_column.py

@@ -16,9 +16,7 @@ retention parameters, retrieval settings, etc.) in Python field name format.
 
 from collections.abc import Sequence
 
-import sqlalchemy as sa
 from alembic import context, op
-from sqlalchemy.dialects.postgresql import JSONB
 
 from hindsight_api.alembic._dialect import run_for_dialect
 

hindsight-api-slim/hindsight_api/api/disconnect.py

@@ -0,0 +1,97 @@
+"""Client-disconnect detection that works behind ``BaseHTTPMiddleware``.
+
+``Request.is_disconnected()`` is the obvious way to notice an abandoned HTTP
+request, but it is silently broken once any ``@app.middleware("http")``
+(Starlette ``BaseHTTPMiddleware``) is installed: that middleware runs the route
+in a child task behind anyio memory streams, so the ``http.disconnect`` ASGI
+event never reaches the route's ``Request``. This app has such middlewares, so
+the recall/reflect cancellation in #2122/#2127 never actually fired in
+production — the disconnect was never observed.
+
+This pure-ASGI middleware sits *outside* the ``BaseHTTPMiddleware`` layer, where
+it still owns the real ``receive`` channel. For the recall and reflect routes it
+drains ``receive`` in a background task and trips a :class:`CancellationToken`
+the moment ``http.disconnect`` arrives, stashing the token on the ASGI ``scope``.
+The route copies that token onto its ``RequestContext`` and the engine checks it
+at stage boundaries — so abandoned work stops instead of running to completion.
+
+It only wraps recall/reflect (small JSON bodies); every other request — uploads,
+MCP streams, etc. — passes straight through untouched, so there is no buffering
+or latency cost elsewhere.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import contextlib
+from collections.abc import Awaitable, Callable, MutableMapping
+from typing import Any
+
+from ..cancellation import CancellationToken
+
+# Key under which the per-request CancellationToken is stored on the ASGI scope.
+# A dedicated top-level scope key (not scope["state"]) avoids any interaction
+# with Starlette's per-request state copying.
+SCOPE_CANCELLATION_TOKEN = "hindsight.cancellation_token"
+
+_CLIENT_DISCONNECTED_REASON = "client disconnected"
+
+Scope = MutableMapping[str, Any]
+Receive = Callable[[], Awaitable[MutableMapping[str, Any]]]
+Send = Callable[[MutableMapping[str, Any]], Awaitable[None]]
+
+
+def _should_monitor(path: str) -> bool:
+    """Only the two long-running, abandon-prone read endpoints need monitoring."""
+    return path.endswith("/memories/recall") or path.endswith("/reflect")
+
+
+class ClientDisconnectCancellationMiddleware:
+    """Trip a scope-level CancellationToken when the client disconnects.
+
+    Must be installed *outside* any ``BaseHTTPMiddleware`` so it owns the real
+    ASGI ``receive`` channel.
+    """
+
+    def __init__(self, app: Callable) -> None:
+        self.app = app
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope["type"] != "http" or not _should_monitor(scope.get("path", "")):
+            await self.app(scope, receive, send)
+            return
+
+        token = CancellationToken()
+        scope[SCOPE_CANCELLATION_TOKEN] = token
+
+        # The downstream app still needs to read the request body, so we cannot
+        # simply consume `receive` ourselves. Instead a single pump task drains
+        # the real channel, forwards every message to a queue the app reads from,
+        # and trips the token the instant `http.disconnect` shows up — which the
+        # app would otherwise never pull once it has finished reading the body.
+        queue: asyncio.Queue = asyncio.Queue()
+
+        async def pump() -> None:
+            while True:
+                message = await receive()
+                if message["type"] == "http.disconnect":
+                    token.cancel(_CLIENT_DISCONNECTED_REASON)
+                    await queue.put(message)
+                    return
+                await queue.put(message)
+
+        async def proxied_receive() -> MutableMapping[str, Any]:
+            return await queue.get()
+
+        pump_task = asyncio.create_task(pump())
+        try:
+            await self.app(scope, proxied_receive, send)
+        finally:
+            pump_task.cancel()
+            with contextlib.suppress(asyncio.CancelledError):
+                await pump_task
+
+
+def get_scope_cancellation_token(scope: Scope) -> CancellationToken | None:
+    """Return the CancellationToken the middleware attached, if any."""
+    return scope.get(SCOPE_CANCELLATION_TOKEN)

hindsight-api-slim/hindsight_api/api/http.py

@@ -10,13 +10,16 @@ import json
 import logging
 import re
 import uuid
+from collections.abc import Awaitable
 from contextlib import asynccontextmanager
-from datetime import datetime, timezone
-from typing import Any, Literal
+from datetime import datetime
+from typing import Any, Literal, TypeVar
 
 from fastapi import Depends, FastAPI, File, Form, Header, HTTPException, Query, Request, UploadFile
 from fastapi.middleware.gzip import GZipMiddleware
 
+from hindsight_api.api.disconnect import ClientDisconnectCancellationMiddleware, get_scope_cancellation_token
+from hindsight_api.cancellation import OperationCancelledError
 from hindsight_api.engine.audit import (
     AuditEntry,
     AuditLogger,
@@ -41,13 +44,78 @@ def _parse_metadata(metadata: Any) -> dict[str, Any]:
     return {}
 
 
-from typing import Callable
+from collections.abc import Iterable
+from types import UnionType
+from typing import Callable, Union, get_args, get_origin
 
+from fastapi.routing import APIRoute
 from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
 from hindsight_api import MemoryEngine
 
 
+def _annotation_is_nullable(annotation: Any) -> bool:
+    """True if the annotation is a Union that includes None (i.e. ``X | None``)."""
+    if get_origin(annotation) in (Union, UnionType):
+        return any(arg is type(None) for arg in get_args(annotation))
+    return False
+
+
+def _iter_models(annotation: Any) -> Iterable[type[BaseModel]]:
+    """Yield every Pydantic model referenced by an annotation, recursing through generics."""
+    if isinstance(annotation, type) and issubclass(annotation, BaseModel):
+        yield annotation
+        return
+    for arg in get_args(annotation):
+        yield from _iter_models(arg)
+
+
+def _model_has_required_nullable(model: type[BaseModel], seen: set[type[BaseModel]]) -> bool:
+    """True if the model (or any nested model) declares a required *and* nullable field.
+
+    Such a field is in the OpenAPI ``required`` set but may serialize to null, so dropping
+    it (via ``exclude_none``) would omit a key that strict generated clients expect to be
+    present. Routes whose response model contains one of these must keep emitting nulls to
+    stay wire-compatible with already-generated clients.
+    """
+    if model in seen:
+        return False
+    seen.add(model)
+    for field in model.model_fields.values():
+        annotation = field.annotation
+        if field.is_required() and _annotation_is_nullable(annotation):
+            return True
+        for nested in _iter_models(annotation):
+            if _model_has_required_nullable(nested, seen):
+                return True
+    return False
+
+
+def _response_model_has_required_nullable(response_model: Any) -> bool:
+    seen: set[type[BaseModel]] = set()
+    return any(_model_has_required_nullable(model, seen) for model in _iter_models(response_model))
+
+
+class ExcludeNoneRoute(APIRoute):
+    """Route class that drops null fields from responses, preserving wire compatibility.
+
+    ``response_model_exclude_none`` is enabled automatically for every route whose response
+    model has no required-and-nullable field. Routes that *do* have such a field (where an
+    omitted key would break strict clients) are left untouched and keep emitting nulls.
+    An explicit ``response_model_exclude_none`` passed to the route decorator is respected.
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        response_model = kwargs.get("response_model")
+        if (
+            not kwargs.get("response_model_exclude_none")
+            and response_model is not None
+            and not _response_model_has_required_nullable(response_model)
+        ):
+            kwargs["response_model_exclude_none"] = True
+        super().__init__(*args, **kwargs)
+
+
 def FieldWithDefault(default_factory: Callable, **kwargs) -> Any:
     """
     Field wrapper that ensures default_factory values appear in OpenAPI schema.
@@ -80,9 +148,14 @@ def FieldWithDefault(default_factory: Callable, **kwargs) -> Any:
 
 
 from hindsight_api.config import get_config
-from hindsight_api.engine.memory_engine import Budget, _current_schema, _get_tiktoken_encoding, fq_table
+from hindsight_api.engine.memory_engine import Budget, _current_schema, _get_tiktoken_encoding
 from hindsight_api.engine.providers.none_llm import LLMNotAvailableError
-from hindsight_api.engine.response_models import VALID_RECALL_FACT_TYPES, MemoryFact, TokenUsage
+from hindsight_api.engine.response_models import (
+    VALID_RECALL_FACT_TYPES,
+    DryRunExtractionResult,
+    MemoryFact,
+    TokenUsage,
+)
 from hindsight_api.engine.search.tags import TagGroup, TagsMatch
 from hindsight_api.extensions import HttpExtension, OperationValidationError, load_extension
 from hindsight_api.metrics import create_metrics_collector, get_metrics_collector, initialize_metrics
@@ -90,6 +163,44 @@ from hindsight_api.models import RequestContext
 
 logger = logging.getLogger(__name__)
 
+# 499 is the de facto reverse-proxy status for "client closed request".
+_CLIENT_CLOSED_REQUEST_STATUS_CODE = 499
+
+_T = TypeVar("_T")
+
+
+async def run_cancellable_on_disconnect(
+    http_request: Request,
+    request_context: RequestContext,
+    coro: Awaitable[_T],
+    *,
+    operation: str,
+    bank_id: str,
+) -> _T:
+    """Run an engine coroutine, aborting it with 499 if the client disconnects.
+
+    Shared by the recall and reflect handlers. The actual disconnect detection
+    lives in ``ClientDisconnectCancellationMiddleware`` (a pure-ASGI middleware
+    installed outside the ``BaseHTTPMiddleware`` layer), which attaches a
+    :class:`CancellationToken` to the ASGI scope and trips it on
+    ``http.disconnect``. Here we simply hand that token to the engine via
+    ``RequestContext`` — the engine checks it at stage/iteration boundaries — and
+    translate the resulting ``OperationCancelledError`` into 499 so abandoned
+    work stops instead of running to completion (issue #2122).
+
+    Note: ``Request.is_disconnected()`` is deliberately NOT used — it silently
+    never fires behind ``BaseHTTPMiddleware``, which is why the original #2127
+    implementation did not actually cancel anything in this app.
+    """
+    token = get_scope_cancellation_token(http_request.scope)
+    if token is not None:
+        request_context.cancellation = token
+    try:
+        return await coro
+    except OperationCancelledError as e:
+        logger.info(f"[{operation.upper()} CANCELLED] bank={bank_id} reason={e.reason}")
+        raise HTTPException(status_code=_CLIENT_CLOSED_REQUEST_STATUS_CODE, detail=e.reason) from e
+
 
 class EntityIncludeOptions(BaseModel):
     """Options for including entity observations in recall results."""
@@ -224,7 +335,7 @@ class RecallResult(BaseModel):
 
     id: str
     text: str
-    type: str | None = None  # fact type: world, experience, opinion, observation
+    type: str | None = None  # fact type: world, experience, observation
     entities: list[str] | None = None  # Entity names mentioned in this fact
     context: str | None = None
     occurred_start: str | None = None  # ISO format date when the event started
@@ -504,13 +615,16 @@ class MemoryItem(BaseModel):
             return [v]
         return v
 
-    observation_scopes: Literal["per_tag", "combined", "all_combinations"] | list[list[str]] | None = Field(
+    observation_scopes: Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]] | None = Field(
         default=None,
         title="ObservationScopes",
         description=(
             "How to scope observations during consolidation. "
             "'per_tag' runs one consolidation pass per individual tag, creating separate observations for each tag. "
             "'combined' (default) runs a single pass with all tags together. "
+            "'shared' runs a single pass over one global, untagged scope, so memories consolidate together "
+            "regardless of their tags — useful for deduplicating across volatile per-call provenance tags "
+            "(e.g. per-session ids) while keeping those tags on the source facts. "
             "A list of tag lists runs one pass per inner list, giving full control over which combinations to use."
         ),
     )
@@ -811,7 +925,7 @@ class ReflectFact(BaseModel):
     text: str = Field(
         description="Fact text. When type='observation', this contains markdown-formatted consolidated knowledge"
     )
-    type: str | None = None  # fact type: world, experience, opinion, observation
+    type: str | None = None  # fact type: world, experience, observation
     context: str | None = None
     occurred_start: str | None = None
     occurred_end: str | None = None
@@ -1108,7 +1222,14 @@ class CreateBankRequest(BaseModel):
     )
     retain_chunk_size: int | None = Field(
         default=None,
-        description="Maximum token size for each content chunk during retain.",
+        description="Target maximum characters for each content chunk during retain.",
+    )
+    retain_structured_chunk_size: int | None = Field(
+        default=None,
+        description=(
+            "Maximum characters for a single JSONL line or conversation turn to keep whole during retain. "
+            "Defaults to retain_chunk_size when unset."
+        ),
     )
     enable_observations: bool | None = Field(
         default=None,
@@ -1148,6 +1269,7 @@ class CreateBankRequest(BaseModel):
             "retain_extraction_mode",
             "retain_custom_instructions",
             "retain_chunk_size",
+            "retain_structured_chunk_size",
             "enable_observations",
             "observations_mission",
         ):
@@ -1239,6 +1361,33 @@ class GraphDataResponse(BaseModel):
     limit: int
 
 
+class ObservationScope(BaseModel):
+    """A distinct observation scope: an exact tag set plus its observation count."""
+
+    tags: list[str] = Field(
+        description="The exact tag set defining this scope (normalized order). Empty list is the global/untagged scope."
+    )
+    count: int = Field(description="Number of observations that live under this scope")
+
+
+class ObservationScopesResponse(BaseModel):
+    """Response model for the observation scopes enumeration endpoint."""
+
+    model_config = ConfigDict(
+        json_schema_extra={
+            "example": {
+                "scopes": [
+                    {"tags": ["user:alice"], "count": 12},
+                    {"tags": ["user:alice", "project:apollo"], "count": 4},
+                    {"tags": [], "count": 2},
+                ]
+            }
+        }
+    )
+
+    scopes: list[ObservationScope] = Field(description="Distinct observation scopes, most populous first")
+
+
 class ListMemoryUnitsResponse(BaseModel):
     """Response model for list memory units endpoint."""
 
@@ -1268,6 +1417,32 @@ class ListMemoryUnitsResponse(BaseModel):
     offset: int
 
 
+class DryRunExtractRequest(BaseModel):
+    """Request to run fact extraction ONLY (no resolution/links/embeddings/persistence).
+
+    Every field below the content/context/date is a prompt-affecting override applied just for this
+    call — used to preview what a candidate retain mission (or any extraction setting) would extract,
+    without changing the bank. Unset (null) fields fall back to the bank's resolved config.
+    """
+
+    content: str = Field(description="Text to extract facts from (e.g. a document or a single chunk).")
+    context: str = Field(default="", description="Optional context about the content.")
+    # Named `timestamp` to match the retain item payload (retain maps timestamp -> event_date internally).
+    timestamp: datetime | None = Field(
+        default=None, description="Reference timestamp for resolving relative times (ISO 8601)."
+    )
+    agent_name: str | None = Field(default=None, description="Narrator override (memory owner) primed in the prompt.")
+    # --- prompt-affecting config overrides (null = use the bank's value) ---
+    retain_mission: str | None = None
+    retain_extraction_mode: str | None = None
+    retain_custom_instructions: str | None = None
+    retain_extract_causal_links: bool | None = None
+    retain_chunk_size: int | None = None
+    entity_labels: list | None = None
+    entities_allow_free_form: bool | None = None
+    llm_output_language: str | None = None
+
+
 class ListDocumentsResponse(BaseModel):
     """Response model for list documents endpoint."""
 
@@ -1352,7 +1527,8 @@ class DocumentResponse(BaseModel):
 
     id: str
     bank_id: str
-    original_text: str
+    # None when document text storage is disabled (HINDSIGHT_API_STORE_DOCUMENT_TEXT=false).
+    original_text: str | None
     content_hash: str | None
     created_at: str
     updated_at: str
@@ -1363,6 +1539,12 @@ class DocumentResponse(BaseModel):
     tags: list[str] = FieldWithDefault(list, description="Tags associated with this document")
     document_metadata: dict[str, Any] | None = Field(default=None, description="Document metadata")
     retain_params: dict[str, Any] | None = Field(default=None, description="Parameters used during retain")
+    observation_scopes: str | list[list[str]] | None = Field(
+        default=None,
+        description="The observation_scopes spec configured at retain time (e.g. 'all_combinations', "
+        "'per_tag', or explicit tag-set lists), captured into retain_params. None when none was set "
+        "(default 'combined' scoping) or for documents retained before this was captured.",
+    )
 
 
 class UpdateDocumentRequest(BaseModel):
@@ -1389,6 +1571,83 @@ class UpdateDocumentResponse(BaseModel):
     success: bool = True
 
 
+class UpdateMemoryRequest(BaseModel):
+    """Request model for curating a single memory unit (edit / invalidate / revert).
+
+    Provide ``text`` to correct the fact, and/or ``state`` to invalidate
+    ('invalidated') or revert ('valid') it. ``reason`` is optional free text
+    recorded on the memory. At least one of ``text`` or ``state`` must be set.
+    Only world/experience facts can be curated; observations are derived.
+    """
+
+    model_config = ConfigDict(
+        json_schema_extra={
+            "example": {
+                "state": "invalidated",
+                "reason": "superseded: server decommissioned 2026-06-01",
+            }
+        }
+    )
+
+    text: str | None = Field(
+        default=None,
+        description="New fact text. Re-embeds the memory, drops its derived "
+        "observations and links, and triggers re-consolidation.",
+    )
+    context: str | None = Field(
+        default=None,
+        description="New context for the fact. '' clears it; omit to leave unchanged.",
+    )
+    occurred_start: str | None = Field(
+        default=None,
+        description="New occurred-range start (ISO 8601). '' clears it; omit to leave unchanged.",
+    )
+    occurred_end: str | None = Field(
+        default=None,
+        description="New occurred-range end (ISO 8601). '' clears it; omit to leave unchanged.",
+    )
+    fact_type: str | None = Field(
+        default=None,
+        description="Reclassify the fact: 'world' or 'experience'. Omit to leave unchanged.",
+    )
+    entities: list[str] | None = Field(
+        default=None,
+        description="Replace the fact's entities. Names are resolved/find-or-created "
+        "the same way retain does; '[]' detaches all entities. Omit to leave unchanged.",
+    )
+    state: str | None = Field(
+        default=None,
+        description="Curation state: 'invalidated' to soft-retire the memory "
+        "(excluded from recall/consolidation, links and derived observations "
+        "pruned, moved to the archive) or 'valid' to revert. Reversible.",
+    )
+    reason: str | None = Field(
+        default=None,
+        description="Optional free-text reason recorded when invalidating.",
+    )
+
+    @model_validator(mode="after")
+    def _require_an_edit(self) -> "UpdateMemoryRequest":
+        if all(
+            v is None
+            for v in (
+                self.text,
+                self.context,
+                self.occurred_start,
+                self.occurred_end,
+                self.fact_type,
+                self.entities,
+                self.state,
+            )
+        ):
+            raise ValueError("Provide at least one field to update.")
+        if self.state is not None and self.state not in ("valid", "invalidated"):
+            raise ValueError("state must be 'valid' or 'invalidated'.")
+        if self.fact_type is not None and self.fact_type not in ("world", "experience"):
+            raise ValueError("fact_type must be 'world' or 'experience'.")
+        return self
+
+
 class DeleteDocumentResponse(BaseModel):
     """Response model for delete document endpoint."""
 
@@ -1538,6 +1797,51 @@ class BankStatsResponse(BaseModel):
     total_observations: int = Field(default=0, description="Total number of observations")
 
 
+class LlmOperationHealth(BaseModel):
+    """LLM connectivity status for a single operation. Status only — no provider/model/
+    endpoint/error, so the probe never discloses the LLM configuration."""
+
+    operation: Literal["retain", "consolidation", "reflect"] = Field(
+        # Distinct title so the generated clients don't collide this inline enum with the
+        # async-operation "operation" enum (progenitor names Rust types from the title).
+        title="LlmHealthOperation",
+        description="Operation whose LLM was probed",
+    )
+    ok: bool = Field(description="True only when the probe connected successfully")
+    status: Literal["connected", "not_configured", "auth_failed", "unreachable", "timeout"] = Field(
+        # Distinct title — otherwise this inline enum's default title "Status" collides
+        # with the async-operation status enum and breaks the generated Rust client/CLI.
+        title="LlmHealthStatus",
+        description="'connected'; 'not_configured' (provider is 'none'); 'auth_failed' (rejected — "
+        "usually a wrong/expired API key); 'unreachable' (call failed); 'timeout'",
+    )
+    latency_ms: float | None = Field(default=None, description="Round-trip latency of the probe call")
+
+
+class BankLlmHealthResponse(BaseModel):
+    """Per-bank LLM connectivity probe across retain/consolidation/reflect. Operations
+    that share a configuration are probed once. Discloses status only — never the
+    provider, model, endpoint, API key, or raw error."""
+
+    model_config = ConfigDict(
+        json_schema_extra={
+            "example": {
+                "bank_id": "user123",
+                "operations": [
+                    {"operation": "retain", "ok": True, "status": "connected", "latency_ms": 412.0},
+                    {"operation": "consolidation", "ok": True, "status": "connected", "latency_ms": 412.0},
+                    {"operation": "reflect", "ok": False, "status": "not_configured", "latency_ms": None},
+                ],
+            }
+        }
+    )
+
+    bank_id: str = Field(description="Bank identifier")
+    operations: list[LlmOperationHealth] = Field(
+        description="Connectivity status per operation (retain, consolidation, reflect)"
+    )
+
+
 class MemoryTimeseriesBucket(BaseModel):
     """One bucket in the memory ingestion time-series."""
 
@@ -1830,7 +2134,14 @@ class BankTemplateConfig(BaseModel):
     retain_custom_instructions: str | None = Field(
         default=None, description="Custom extraction prompt (when mode='custom')"
     )
-    retain_chunk_size: int | None = Field(default=None, description="Max token size for each content chunk")
+    retain_chunk_size: int | None = Field(default=None, description="Target max characters for each content chunk")
+    retain_structured_chunk_size: int | None = Field(
+        default=None,
+        description=(
+            "Max characters for a single JSONL line or conversation turn to keep whole; "
+            "defaults to retain_chunk_size when unset"
+        ),
+    )
     enable_observations: bool | None = Field(default=None, description="Toggle observation consolidation")
     observations_mission: str | None = Field(default=None, description="Controls what gets synthesised")
     disposition_skepticism: int | None = Field(default=None, ge=1, le=5, description="Skepticism trait (1-5)")
@@ -1866,6 +2177,16 @@ class BankTemplateConfig(BaseModel):
     max_observations_per_scope: int | None = Field(
         default=None, description="Max observations to retain per consolidation scope"
     )
+    observation_scope_limits: list[dict[str, Any]] | None = Field(
+        default=None,
+        description=(
+            "Per-scope overrides of max_observations_per_scope: "
+            '[{"scope": ["run_*", "shared"], "limit": 1}]. Each scope is a list of '
+            "fnmatch tag-globs; a consolidation scope matches under exact cover "
+            "(every tag matched by a glob and every glob matched by a tag). The first "
+            "matching rule wins; unmatched scopes fall back to max_observations_per_scope."
+        ),
+    )
     reflect_source_facts_max_tokens: int | None = Field(
         default=None, description="Max tokens of source facts per reflect call"
     )
@@ -2413,11 +2734,15 @@ class FeaturesInfo(BaseModel):
     mcp: bool = Field(description="Whether MCP (Model Context Protocol) server is enabled")
     worker: bool = Field(description="Whether the background worker is enabled")
     bank_config_api: bool = Field(description="Whether per-bank configuration API is enabled")
+    bank_llm_health: bool = Field(description="Whether the per-bank LLM connectivity probe is enabled")
     file_upload_api: bool = Field(description="Whether file upload/conversion API is enabled")
     document_export_api: bool = Field(description="Whether the document export endpoint is enabled")
     document_import_api: bool = Field(description="Whether the document import endpoint is enabled")
     audit_log: bool = Field(description="Whether audit logging is enabled")
     llm_trace: bool = Field(description="Whether per-bank LLM request tracing is enabled")
+    store_document_text: bool = Field(
+        description="Whether raw source text is persisted. When false, document/chunk source text is not stored."
+    )
 
 
 class VersionResponse(BaseModel):
@@ -2459,7 +2784,7 @@ class CreateWebhookRequest(BaseModel):
     secret: str | None = Field(default=None, description="HMAC-SHA256 signing secret (optional)")
     event_types: list[str] = Field(
         default=["consolidation.completed"],
-        description="List of event types to deliver. Currently supported: 'consolidation.completed'",
+        description="List of event types to deliver. Supported: 'retain.completed', 'consolidation.completed', 'memory_defense.triggered'.",
     )
     enabled: bool = Field(default=True, description="Whether this webhook is active")
     http_config: WebhookHttpConfig = Field(
@@ -2569,7 +2894,6 @@ def _make_audited_http(audit_logger_getter: Callable[[], AuditLogger | None]):
     from datetime import datetime as _dt
     from datetime import timezone as _tz
     from functools import wraps
-    from typing import Callable as _Callable
 
     def audited(action: str, *, request_param: str | None = "request"):
         """Decorator that wraps an HTTP handler with audit logging.
@@ -2800,6 +3124,10 @@ def create_app(
         root_path=config.base_path,
     )
 
+    # Drop null fields from responses (omit `"x": null`) for routes where it's wire-safe.
+    # Must be set before any route is registered so @app.<method> decorators pick it up.
+    app.router.route_class = ExcludeNoneRoute
+
     # IMPORTANT: Set memory on app.state immediately, don't wait for lifespan
     # This is required for mounted sub-applications where lifespan may not fire
     app.state.memory = memory
@@ -2913,8 +3241,6 @@ def create_app(
         # Replace UUIDs and numeric IDs with placeholders
         import re
 
-        from starlette.requests import Request
-
         path = request.url.path
         # Replace UUIDs
         path = re.sub(r"/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", "/{id}", path)
@@ -2944,6 +3270,13 @@ def create_app(
             app.include_router(root_router)
             logging.info("HTTP extension root router mounted")
 
+    # Client-disconnect cancellation for recall/reflect. Added LAST so it sits
+    # OUTSIDE the @app.middleware("http") (BaseHTTPMiddleware) layers above —
+    # that placement is mandatory: BaseHTTPMiddleware breaks
+    # Request.is_disconnected(), so the only way to observe an abandoned request
+    # is to own the raw ASGI receive channel from outside it (issue #2122).
+    app.add_middleware(ClientDisconnectCancellationMiddleware)
+
     return app
 
 
@@ -3080,11 +3413,13 @@ def _register_routes(app: FastAPI):
                 mcp=config.mcp_enabled,
                 worker=config.worker_enabled,
                 bank_config_api=config.enable_bank_config_api,
+                bank_llm_health=config.enable_bank_llm_health,
                 file_upload_api=config.enable_file_upload_api,
                 document_export_api=config.enable_document_export_api,
                 document_import_api=config.enable_document_import_api,
                 audit_log=config.audit_log_enabled,
                 llm_trace=config.llm_trace_enabled,
+                store_document_text=config.store_document_text,
             ),
         )
 
@@ -3106,7 +3441,7 @@ def _register_routes(app: FastAPI):
         "/v1/default/banks/{bank_id}/graph",
         response_model=GraphDataResponse,
         summary="Get memory graph data",
-        description="Retrieve graph data for visualization, optionally filtered by type (world/experience/opinion).",
+        description="Retrieve graph data for visualization, optionally filtered by type (world/experience/observation).",
         operation_id="get_graph",
         tags=["Memory"],
     )
@@ -3159,6 +3494,8 @@ def _register_routes(app: FastAPI):
         type: str | None = None,
         q: str | None = None,
         consolidation_state: str | None = None,
+        state: str | None = None,
+        document_id: str | None = None,
         limit: int = 100,
         offset: int = 0,
         request_context: RequestContext = Depends(get_request_context),
@@ -3171,7 +3508,7 @@ def _register_routes(app: FastAPI):
 
         Args:
             bank_id: Memory Bank ID (from path)
-            type: Filter by fact type (world, experience, opinion)
+            type: Filter by fact type (world, experience, observation)
             q: Search query for full-text search (searches text and context)
             consolidation_state: Filter by consolidation state for source memories
                 (world/experience). One of 'failed', 'pending', or 'done'.
@@ -3184,6 +3521,8 @@ def _register_routes(app: FastAPI):
                 fact_type=type,
                 search_query=q,
                 consolidation_state=consolidation_state,
+                state=state,
+                document_id=document_id,
                 limit=limit,
                 offset=offset,
                 request_context=request_context,
@@ -3202,6 +3541,75 @@ def _register_routes(app: FastAPI):
             logger.error(f"Error in /v1/default/banks/{bank_id}/memories/list: {error_detail}")
             raise HTTPException(status_code=500, detail=str(e))
 
+    async def _require_dry_run_enabled() -> None:
+        """Feature-flag gate for dry-run extraction.
+
+        Declared as a dependency BEFORE ``precheck_for("dry_run_extract")`` so a
+        disabled route returns 404 regardless of tenant/billing state — FastAPI
+        resolves path-operation dependencies in signature order, so this runs
+        first and preserves the original "disabled → 404" contract.
+        """
+        if not get_config().enable_dry_run_extract:
+            raise HTTPException(
+                status_code=404,
+                detail="Dry-run extraction is disabled. Set HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT=true to re-enable.",
+            )
+
+    @app.post(
+        "/v1/default/banks/{bank_id}/memories/dry-run-extract",
+        response_model=DryRunExtractionResult,
+        summary="Dry-run fact extraction (preview, no persistence)",
+        description=(
+            "Preview what the retain step would extract from text WITHOUT changing the bank — no "
+            "entity resolution, links, embeddings, or persistence. Returns the candidate facts and "
+            "the LLM token usage. Every prompt-affecting setting (retain mission, extraction mode, "
+            "chunk size, …) is overridable in the body to A/B a candidate config against the bank's "
+            "current one. This is a read-only tool: nothing is stored."
+        ),
+        operation_id="dry_run_extract_memories",
+        tags=["Memory"],
+    )
+    async def api_dry_run_extract(
+        bank_id: str,
+        body: DryRunExtractRequest,
+        request_context: RequestContext = Depends(get_request_context),
+        _enabled: None = Depends(_require_dry_run_enabled),
+        _precheck: None = Depends(precheck_for("dry_run_extract")),
+    ):
+        try:
+            override_fields = (
+                "retain_mission",
+                "retain_extraction_mode",
+                "retain_custom_instructions",
+                "retain_extract_causal_links",
+                "retain_chunk_size",
+                "entity_labels",
+                "entities_allow_free_form",
+                "llm_output_language",
+            )
+            overrides = {f: getattr(body, f) for f in override_fields if getattr(body, f) is not None}
+            return await app.state.memory.extract_dry_run(
+                bank_id,
+                body.content,
+                context=body.context or "",
+                event_date=body.timestamp,
+                overrides=overrides,
+                agent_name=body.agent_name,
+                request_context=request_context,
+            )
+        except ValueError as e:
+            raise HTTPException(status_code=400, detail=str(e))
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
+            logger.error(f"Error in /v1/default/banks/{bank_id}/memories/dry-run-extract: {error_detail}")
+            raise HTTPException(status_code=500, detail=str(e))
+
     @app.get(
         "/v1/default/banks/{bank_id}/memories/{memory_id}",
         summary="Get memory unit",
@@ -3237,6 +3645,53 @@ def _register_routes(app: FastAPI):
             logger.error(f"Error in /v1/default/banks/{bank_id}/memories/{memory_id}: {error_detail}")
             raise HTTPException(status_code=500, detail=str(e))
 
+    @app.patch(
+        "/v1/default/banks/{bank_id}/memories/{memory_id}",
+        summary="Curate memory unit",
+        description="Edit a memory's text and/or change its curation state "
+        "(invalidate / revert). Invalidated memories are excluded from recall, "
+        "consolidation, and graph maintenance but kept for audit (reversible). "
+        "Only world/experience facts can be curated; observations are derived.",
+        operation_id="update_memory",
+        tags=["Memory"],
+    )
+    async def api_update_memory(
+        bank_id: str,
+        memory_id: str,
+        request: UpdateMemoryRequest,
+        request_context: RequestContext = Depends(get_request_context),
+    ):
+        """Curate a single memory unit (edit text / invalidate / revert)."""
+        try:
+            data = await app.state.memory.update_memory_unit(
+                bank_id=bank_id,
+                memory_id=memory_id,
+                text=request.text,
+                context=request.context,
+                occurred_start=request.occurred_start,
+                occurred_end=request.occurred_end,
+                new_fact_type=request.fact_type,
+                entities=request.entities,
+                state=request.state,
+                reason=request.reason,
+                request_context=request_context,
+            )
+            if data is None:
+                raise HTTPException(status_code=404, detail=f"Memory unit '{memory_id}' not found")
+            return data
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except ValueError as e:
+            raise HTTPException(status_code=400, detail=str(e))
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
+            logger.error(f"Error in PATCH /v1/default/banks/{bank_id}/memories/{memory_id}: {error_detail}")
+            raise HTTPException(status_code=500, detail=str(e))
+
     @app.get(
         "/v1/default/banks/{bank_id}/memories/{memory_id}/history",
         summary="Get observation history",
@@ -3285,6 +3740,7 @@ def _register_routes(app: FastAPI):
     async def api_recall(
         bank_id: str,
         request: RecallRequest,
+        http_request: Request,
         request_context: RequestContext = Depends(get_request_context),
         _precheck: None = Depends(precheck_for("recall")),
     ):
@@ -3340,25 +3796,34 @@ def _register_routes(app: FastAPI):
                 "recall", bank_id=bank_id, source="api", budget=request.budget.value, max_tokens=request.max_tokens
             ):
                 recall_start = time.time()
-                core_result = await app.state.memory.recall_async(
+                # Cancel the recall if the client disconnects: the engine checks
+                # request_context at each stage boundary and aborts abandoned
+                # work rather than running it to completion (issue #2122).
+                core_result = await run_cancellable_on_disconnect(
+                    http_request,
+                    request_context,
+                    app.state.memory.recall_async(
+                        bank_id=bank_id,
+                        query=request.query,
+                        budget=request.budget,
+                        max_tokens=request.max_tokens,
+                        enable_trace=request.trace,
+                        fact_type=fact_types,
+                        question_date=question_date,
+                        include_entities=include_entities,
+                        max_entity_tokens=max_entity_tokens,
+                        include_chunks=include_chunks,
+                        max_chunk_tokens=max_chunk_tokens,
+                        include_source_facts=include_source_facts,
+                        max_source_facts_tokens=max_source_facts_tokens,
+                        max_source_facts_tokens_per_observation=max_source_facts_tokens_per_observation,
+                        request_context=request_context,
+                        tags=request.tags,
+                        tags_match=request.tags_match,
+                        tag_groups=request.tag_groups,
+                    ),
+                    operation="recall",
                     bank_id=bank_id,
-                    query=request.query,
-                    budget=request.budget,
-                    max_tokens=request.max_tokens,
-                    enable_trace=request.trace,
-                    fact_type=fact_types,
-                    question_date=question_date,
-                    include_entities=include_entities,
-                    max_entity_tokens=max_entity_tokens,
-                    include_chunks=include_chunks,
-                    max_chunk_tokens=max_chunk_tokens,
-                    include_source_facts=include_source_facts,
-                    max_source_facts_tokens=max_source_facts_tokens,
-                    max_source_facts_tokens_per_observation=max_source_facts_tokens_per_observation,
-                    request_context=request_context,
-                    tags=request.tags,
-                    tags_match=request.tags_match,
-                    tag_groups=request.tag_groups,
                 )
 
             # Convert core MemoryFact objects to API RecallResult objects (excluding internal metrics)
@@ -3462,11 +3927,11 @@ def _register_routes(app: FastAPI):
         "/v1/default/banks/{bank_id}/reflect",
         response_model=ReflectResponse,
         summary="Reflect and generate answer",
-        description="Reflect and formulate an answer using bank identity, world facts, and opinions.\n\n"
+        description="Reflect and formulate an answer using bank identity, world facts, observations, and mental models.\n\n"
         "This endpoint:\n"
         "1. Retrieves experience (conversations and events)\n"
         "2. Retrieves world facts relevant to the query\n"
-        "3. Retrieves existing opinions (bank's perspectives)\n"
+        "3. Retrieves observations and mental models (bank's synthesized perspectives)\n"
         "4. Uses LLM to formulate a contextual answer\n"
         "5. Returns plain text answer and the facts used",
         operation_id="reflect",
@@ -3476,6 +3941,7 @@ def _register_routes(app: FastAPI):
     async def api_reflect(
         bank_id: str,
         request: ReflectRequest,
+        http_request: Request,
         request_context: RequestContext = Depends(get_request_context),
         _precheck: None = Depends(precheck_for("reflect")),
     ):
@@ -3489,20 +3955,30 @@ def _register_routes(app: FastAPI):
 
             # Use the memory system's reflect_async method (record metrics)
             with metrics.record_operation("reflect", bank_id=bank_id, source="api", budget=request.budget.value):
-                core_result = await app.state.memory.reflect_async(
+                # Cancel the reflect if the client disconnects: the agent loop
+                # checks request_context between iterations and the nested recall
+                # checks at its stage boundaries, so abandoned work stops instead
+                # of running to completion (issue #2122).
+                core_result = await run_cancellable_on_disconnect(
+                    http_request,
+                    request_context,
+                    app.state.memory.reflect_async(
+                        bank_id=bank_id,
+                        query=query,
+                        budget=request.budget,
+                        context=None,  # Deprecated, now concatenated with query
+                        max_tokens=request.max_tokens,
+                        response_schema=request.response_schema,
+                        request_context=request_context,
+                        tags=request.tags,
+                        tags_match=request.tags_match,
+                        tag_groups=request.tag_groups,
+                        fact_types=request.fact_types,
+                        exclude_mental_models=request.exclude_mental_models,
+                        exclude_mental_model_ids=request.exclude_mental_model_ids,
+                    ),
+                    operation="reflect",
                     bank_id=bank_id,
-                    query=query,
-                    budget=request.budget,
-                    context=None,  # Deprecated, now concatenated with query
-                    max_tokens=request.max_tokens,
-                    response_schema=request.response_schema,
-                    request_context=request_context,
-                    tags=request.tags,
-                    tags_match=request.tags_match,
-                    tag_groups=request.tag_groups,
-                    fact_types=request.fact_types,
-                    exclude_mental_models=request.exclude_mental_models,
-                    exclude_mental_model_ids=request.exclude_mental_model_ids,
                 )
 
             # Build based_on (memories + mental_models + directives) if facts are requested
@@ -3668,6 +4144,45 @@ def _register_routes(app: FastAPI):
             logger.error(f"Error in /v1/default/banks/{bank_id}/stats: {error_detail}")
             raise HTTPException(status_code=500, detail=str(e))
 
+    @app.post(
+        "/v1/default/banks/{bank_id}/health/llm",
+        response_model=BankLlmHealthResponse,
+        summary="Test the bank's LLM connectivity",
+        description="Probe the LLMs this bank would use for retain / consolidation / reflect with one minimal call "
+        "each (configs shared across operations are probed once), so you can discover 'not configured / unreachable' "
+        "instead of a silent stall. Deliberate action (makes a real provider call); not for polling. Returns status "
+        "only — never the provider, model, endpoint, API key, or raw error. Disable with "
+        "HINDSIGHT_API_ENABLE_BANK_LLM_HEALTH=false.",
+        operation_id="test_bank_llm",
+        tags=["Banks"],
+    )
+    async def api_bank_llm_health(bank_id: str, request_context: RequestContext = Depends(get_request_context)):
+        """Probe per-bank LLM connectivity."""
+        if not get_config().enable_bank_llm_health:
+            raise HTTPException(
+                status_code=404,
+                detail="Bank LLM health check is disabled. Set HINDSIGHT_API_ENABLE_BANK_LLM_HEALTH=true to enable.",
+            )
+        try:
+            result = await app.state.memory.check_bank_llm(bank_id, request_context=request_context)
+            return BankLlmHealthResponse(
+                bank_id=result.bank_id,
+                operations=[
+                    LlmOperationHealth(operation=op.operation, ok=op.ok, status=op.status, latency_ms=op.latency_ms)
+                    for op in result.operations
+                ],
+            )
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
+            logger.error(f"Error in POST /v1/default/banks/{bank_id}/health/llm: {error_detail}")
+            raise HTTPException(status_code=500, detail=str(e))
+
     @app.get(
         "/v1/default/banks/{bank_id}/stats/memories-timeseries",
         response_model=MemoriesTimeseriesResponse,
@@ -5486,6 +6001,35 @@ def _register_routes(app: FastAPI):
             logger.error(f"Error in DELETE /v1/default/banks/{bank_id}/observations: {error_detail}")
             raise HTTPException(status_code=500, detail=str(e))
 
+    @app.get(
+        "/v1/default/banks/{bank_id}/observations/scopes",
+        response_model=ObservationScopesResponse,
+        summary="List observation scopes",
+        description=(
+            "Enumerate the distinct scopes across a bank's observations. Each observation lives "
+            "under a scope: the exact set of tags it was consolidated with. Returns every distinct "
+            "scope (tag order normalized) with the number of observations in it; the empty tag list "
+            "is the global/untagged scope. Use a returned scope with the graph endpoint "
+            "(tags=<scope> & tags_match=exact) to filter observations to exactly that scope."
+        ),
+        operation_id="list_observation_scopes",
+        tags=["Memory"],
+    )
+    async def api_list_observation_scopes(bank_id: str, request_context: RequestContext = Depends(get_request_context)):
+        """List the distinct observation scopes (exact tag sets) for a bank."""
+        try:
+            return await app.state.memory.list_observation_scopes(bank_id, request_context=request_context)
+        except OperationValidationError as e:
+            raise HTTPException(status_code=e.status_code, detail=e.reason)
+        except (AuthenticationError, HTTPException):
+            raise
+        except Exception as e:
+            import traceback
+
+            error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
+            logger.error(f"Error in GET /v1/default/banks/{bank_id}/observations/scopes: {error_detail}")
+            raise HTTPException(status_code=500, detail=str(e))
+
     @app.post(
         "/v1/default/banks/{bank_id}/consolidation/recover",
         response_model=RecoverConsolidationResponse,
@@ -5627,6 +6171,15 @@ def _register_routes(app: FastAPI):
                     app.state.memory._operation_validator.validate_bank_write(ctx)
                 )
 
+            # Validate Memory Defense policy shape before persisting.
+            if "memory_defense" in request.updates and request.updates["memory_defense"] is not None:
+                from hindsight_api.extensions.memory_defense import parse_policy
+
+                try:
+                    parse_policy(request.updates["memory_defense"])
+                except ValueError as exc:
+                    raise HTTPException(status_code=422, detail=f"invalid memory_defense policy: {exc}")
+
             # Update config via config resolver (validates configurable fields and permissions)
             await app.state.memory._config_resolver.update_bank_config(bank_id, request.updates, request_context)
 
@@ -6155,7 +6708,21 @@ def _register_routes(app: FastAPI):
             raise HTTPException(status_code=e.status_code, detail=e.reason)
         except (AuthenticationError, HTTPException):
             raise
+        except ValueError as e:
+            # Invalid request parameters (e.g. duplicate document_ids, or
+            # update_mode='append' when document text storage is disabled) are
+            # client errors, not server faults.
+            raise HTTPException(status_code=400, detail=str(e))
         except Exception as e:
+            from dataclasses import asdict
+
+            from hindsight_api.engine.retain.orchestrator import MemoryDefenseAllBlockedError
+
+            if isinstance(e, MemoryDefenseAllBlockedError):
+                raise HTTPException(
+                    status_code=422,
+                    detail={"violations": [asdict(v) for v in e.violations]},
+                )
             import traceback
 
             # Create a summary of the input for debugging
@@ -6274,7 +6841,6 @@ def _register_routes(app: FastAPI):
                 _validate_parsers(_resolve_parser(request_data.parser), "request-level parser")
 
             # Prepare file items and calculate total batch size
-            import io
 
             file_items = []
             total_batch_size = 0
@@ -6353,14 +6919,14 @@ def _register_routes(app: FastAPI):
         "/v1/default/banks/{bank_id}/memories",
         response_model=DeleteResponse,
         summary="Clear memory bank memories",
-        description="Delete memory units for a memory bank. Optionally filter by type (world, experience, opinion) to delete only specific types. This is a destructive operation that cannot be undone. The bank profile (disposition and background) will be preserved.",
+        description="Delete memory units for a memory bank. Optionally filter by type (world, experience, observation) to delete only specific types. This is a destructive operation that cannot be undone. The bank profile (disposition and background) will be preserved.",
         operation_id="clear_bank_memories",
         tags=["Memory"],
     )
     @audited("clear_memories", request_param=None)
     async def api_clear_bank_memories(
         bank_id: str,
-        type: str | None = Query(None, description="Optional fact type filter (world, experience, opinion)"),
+        type: str | None = Query(None, description="Optional fact type filter (world, experience, observation)"),
         request_context: RequestContext = Depends(get_request_context),
     ):
         """Clear memories for a memory bank, optionally filtered by type."""

hindsight-api-slim/hindsight_api/api/mcp.py

@@ -113,6 +113,8 @@ def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
             "delete_directive",
             "list_memories",
             "get_memory",
+            "update_memory",
+            "invalidate_memory",
             "list_documents",
             "get_document",
             "delete_document",

hindsight-api-slim/hindsight_api/cancellation.py

@@ -0,0 +1,85 @@
+"""Cooperative cancellation for long-running engine operations.
+
+Recall runs as a staged pipeline whose heavy stages — graph expansion and
+cross-encoder reranking — execute in worker threads (``run_in_executor``) that
+asyncio task cancellation cannot interrupt once they have started. Cancelling
+the awaiting task only unblocks the ``await``; the thread keeps burning CPU to
+completion. So rather than rely on task cancellation, callers thread a
+``CancellationToken`` through ``RequestContext`` and the engine checks it at
+stage boundaries (``raise_if_cancelled``), bailing out *before* dispatching the
+next expensive stage.
+
+This is cooperative by design: it cannot stop a computation already inside a
+worker thread, but it does stop an abandoned recall from progressing into — or
+past — that work, which is what starves the instance in issue #2122. The token
+lives on ``RequestContext``, so any operation that receives one (recall today;
+reflect/consolidation/MCP later) can adopt the same checkpoints, and any driver
+(client disconnect today; a deadline tomorrow) can fire it.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+
+class OperationCancelledError(Exception):
+    """Raised at a checkpoint when the operation has been cancelled.
+
+    Carries the ``reason`` set by whoever cancelled (e.g. "client disconnected")
+    so the HTTP layer can translate it into the appropriate status code instead
+    of a generic 500.
+
+    NOTE: this is a plain ``Exception`` on purpose, NOT ``BaseException``. The
+    recall/reflect pipelines have broad ``except Exception`` handlers that would
+    otherwise swallow it — those handlers re-raise ``OperationCancelledError``
+    explicitly (see ``_search_with_retries``) so cancellation propagates to the
+    HTTP layer. A ``BaseException`` would dodge those handlers but also slip past
+    legitimate ``isinstance(result, Exception)`` checks (e.g. the reflect agent's
+    ``asyncio.gather(..., return_exceptions=True)`` tool-result handling), which
+    expect every non-tuple result to be an ``Exception``.
+    """
+
+    def __init__(self, reason: str = "operation cancelled") -> None:
+        super().__init__(reason)
+        self.reason = reason
+
+
+class CancellationToken:
+    """A one-shot, cooperative cancellation signal.
+
+    Cheap to poll (``raise_if_cancelled``) at stage boundaries and awaitable
+    (``wait``) so a driver task can block until cancellation. Safe to share
+    across an engine call tree; polling is a no-op until something cancels, and
+    cancellation is idempotent (the first reason wins).
+    """
+
+    __slots__ = ("_event", "_reason")
+
+    def __init__(self) -> None:
+        self._event = asyncio.Event()
+        self._reason = "operation cancelled"
+
+    def cancel(self, reason: str = "operation cancelled") -> None:
+        """Signal cancellation. Idempotent; the first reason recorded wins."""
+        if not self._event.is_set():
+            self._reason = reason
+            self._event.set()
+
+    @property
+    def cancelled(self) -> bool:
+        """Whether cancellation has been signalled."""
+        return self._event.is_set()
+
+    @property
+    def reason(self) -> str:
+        """The reason recorded by the first ``cancel`` call."""
+        return self._reason
+
+    def raise_if_cancelled(self) -> None:
+        """Raise ``OperationCancelledError`` if cancellation has been signalled."""
+        if self._event.is_set():
+            raise OperationCancelledError(self._reason)
+
+    async def wait(self) -> None:
+        """Block until cancellation is signalled."""
+        await self._event.wait()

hindsight-api-slim/hindsight_api/config.py

@@ -141,9 +141,11 @@ ENV_LLM_TIMEOUT = "HINDSIGHT_API_LLM_TIMEOUT"
 ENV_LLM_REASONING_EFFORT = "HINDSIGHT_API_LLM_REASONING_EFFORT"
 ENV_LLM_GROQ_SERVICE_TIER = "HINDSIGHT_API_LLM_GROQ_SERVICE_TIER"
 ENV_LLM_OPENAI_SERVICE_TIER = "HINDSIGHT_API_LLM_OPENAI_SERVICE_TIER"
+ENV_LLM_BEDROCK_SERVICE_TIER = "HINDSIGHT_API_LLM_BEDROCK_SERVICE_TIER"
 ENV_LLM_EXTRA_BODY = "HINDSIGHT_API_LLM_EXTRA_BODY"
 ENV_LLM_DEFAULT_HEADERS = "HINDSIGHT_API_LLM_DEFAULT_HEADERS"
 ENV_LLM_STRICT_SCHEMA = "HINDSIGHT_API_LLM_STRICT_SCHEMA"
+ENV_LLM_SEND_BANK_AS_USER = "HINDSIGHT_API_LLM_SEND_BANK_AS_USER"
 
 # LiteLLM Router chain — provider-specific config consumed by the "litellmrouter"
 # provider. Each entry is a deployment; the Router tries them in declared order and
@@ -156,6 +158,7 @@ ENV_LLM_LITELLMROUTER_CONFIG = "HINDSIGHT_API_LLM_LITELLMROUTER_CONFIG"
 # Defaults for service tiers
 DEFAULT_LLM_GROQ_SERVICE_TIER = "auto"  # "on_demand", "flex", or "auto"
 DEFAULT_LLM_OPENAI_SERVICE_TIER = None  # None (default) or "flex" (50% cheaper)
+DEFAULT_LLM_BEDROCK_SERVICE_TIER = None  # None (default), "flex", "priority", or "reserved"
 DEFAULT_LLM_EXTRA_BODY = None  # None = no extra body params; JSON dict merged into OpenAI extra_body
 DEFAULT_LLM_DEFAULT_HEADERS = (
     None  # None = no extra headers; JSON dict passed as default_headers to provider SDK clients
@@ -252,6 +255,7 @@ ENV_EMBEDDINGS_OPENROUTER_API_KEY = "HINDSIGHT_API_EMBEDDINGS_OPENROUTER_API_KEY
 ENV_EMBEDDINGS_OPENROUTER_MODEL = "HINDSIGHT_API_EMBEDDINGS_OPENROUTER_MODEL"
 ENV_RERANKER_OPENROUTER_API_KEY = "HINDSIGHT_API_RERANKER_OPENROUTER_API_KEY"
 ENV_RERANKER_OPENROUTER_MODEL = "HINDSIGHT_API_RERANKER_OPENROUTER_MODEL"
+ENV_RERANKER_OPENROUTER_BASE_URL = "HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL"
 
 # ZeroEntropy configuration (embeddings)
 ENV_EMBEDDINGS_ZEROENTROPY_API_KEY = "HINDSIGHT_API_EMBEDDINGS_ZEROENTROPY_API_KEY"
@@ -351,6 +355,8 @@ ENV_MCP_ENABLED = "HINDSIGHT_API_MCP_ENABLED"
 ENV_MCP_ENABLED_TOOLS = "HINDSIGHT_API_MCP_ENABLED_TOOLS"
 ENV_MCP_STATELESS = "HINDSIGHT_API_MCP_STATELESS"
 ENV_ENABLE_BANK_CONFIG_API = "HINDSIGHT_API_ENABLE_BANK_CONFIG_API"
+ENV_ENABLE_BANK_LLM_HEALTH = "HINDSIGHT_API_ENABLE_BANK_LLM_HEALTH"
+ENV_ENABLE_DRY_RUN_EXTRACT = "HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT"
 ENV_DEFAULT_BANK_TEMPLATE = "HINDSIGHT_API_DEFAULT_BANK_TEMPLATE"
 ENV_GRAPH_RETRIEVER = "HINDSIGHT_API_GRAPH_RETRIEVER"
 ENV_RECALL_MAX_CONCURRENT = "HINDSIGHT_API_RECALL_MAX_CONCURRENT"
@@ -391,6 +397,7 @@ ENV_LLM_PROMPT_CACHE_ENABLED = "HINDSIGHT_API_LLM_PROMPT_CACHE_ENABLED"
 # Retain settings
 ENV_RETAIN_MAX_COMPLETION_TOKENS = "HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS"
 ENV_RETAIN_CHUNK_SIZE = "HINDSIGHT_API_RETAIN_CHUNK_SIZE"
+ENV_RETAIN_STRUCTURED_CHUNK_SIZE = "HINDSIGHT_API_RETAIN_STRUCTURED_CHUNK_SIZE"
 ENV_RETAIN_EXTRACT_CAUSAL_LINKS = "HINDSIGHT_API_RETAIN_EXTRACT_CAUSAL_LINKS"
 ENV_RETAIN_EXTRACTION_MODE = "HINDSIGHT_API_RETAIN_EXTRACTION_MODE"
 ENV_RETAIN_MISSION = "HINDSIGHT_API_RETAIN_MISSION"
@@ -424,6 +431,7 @@ ENV_FILE_CONVERSION_MAX_BATCH_SIZE_MB = "HINDSIGHT_API_FILE_CONVERSION_MAX_BATCH
 ENV_FILE_CONVERSION_MAX_BATCH_SIZE = "HINDSIGHT_API_FILE_CONVERSION_MAX_BATCH_SIZE"
 ENV_ENABLE_FILE_UPLOAD_API = "HINDSIGHT_API_ENABLE_FILE_UPLOAD_API"
 ENV_FILE_DELETE_AFTER_RETAIN = "HINDSIGHT_API_FILE_DELETE_AFTER_RETAIN"
+ENV_STORE_DOCUMENT_TEXT = "HINDSIGHT_API_STORE_DOCUMENT_TEXT"
 
 # Document transfer (export/import documents between banks without re-running the LLM)
 ENV_ENABLE_DOCUMENT_EXPORT_API = "HINDSIGHT_API_ENABLE_DOCUMENT_EXPORT_API"
@@ -447,6 +455,7 @@ ENV_CONSOLIDATION_RECALL_BUDGET = "HINDSIGHT_API_CONSOLIDATION_RECALL_BUDGET"
 ENV_CONSOLIDATION_MAX_ATTEMPTS = "HINDSIGHT_API_CONSOLIDATION_MAX_ATTEMPTS"
 ENV_OBSERVATIONS_MISSION = "HINDSIGHT_API_OBSERVATIONS_MISSION"
 ENV_MAX_OBSERVATIONS_PER_SCOPE = "HINDSIGHT_API_MAX_OBSERVATIONS_PER_SCOPE"
+ENV_OBSERVATION_SCOPE_LIMITS = "HINDSIGHT_API_OBSERVATION_SCOPE_LIMITS"
 ENV_ENABLE_OBSERVATION_HISTORY = "HINDSIGHT_API_ENABLE_OBSERVATION_HISTORY"
 ENV_OBSERVATION_HISTORY_MAX_ENTRIES = "HINDSIGHT_API_OBSERVATION_HISTORY_MAX_ENTRIES"
 ENV_ENABLE_MENTAL_MODEL_HISTORY = "HINDSIGHT_API_ENABLE_MENTAL_MODEL_HISTORY"
@@ -472,6 +481,7 @@ ENV_LAZY_RERANKER = "HINDSIGHT_API_LAZY_RERANKER"
 
 # Database migrations
 ENV_RUN_MIGRATIONS_ON_STARTUP = "HINDSIGHT_API_RUN_MIGRATIONS_ON_STARTUP"
+ENV_MIGRATION_CONCURRENCY = "HINDSIGHT_API_MIGRATION_CONCURRENCY"
 
 # Database connection pool
 ENV_DB_POOL_MIN_SIZE = "HINDSIGHT_API_DB_POOL_MIN_SIZE"
@@ -592,6 +602,7 @@ PROVIDER_DEFAULT_MODELS = {
     "volcano": "doubao-pro-32k",
     "openrouter": "qwen/qwen3.5-9b",
     "fireworks": "accounts/fireworks/models/llama-v3p1-8b-instruct",
+    "nous": "deepseek/deepseek-v4-flash",
 }
 DEFAULT_LLM_MODEL = "gpt-4o-mini"  # Fallback if provider not in table
 # Built-in llama.cpp defaults
@@ -615,6 +626,7 @@ DEFAULT_LLM_INITIAL_BACKOFF = 1.0  # Initial backoff in seconds for retry expone
 DEFAULT_LLM_MAX_BACKOFF = 60.0  # Max backoff cap in seconds for retry exponential backoff
 DEFAULT_LLM_TIMEOUT = 120.0  # seconds
 DEFAULT_LLM_REASONING_EFFORT = "low"
+DEFAULT_LLM_SEND_BANK_AS_USER = False  # Opt-in: tag provider calls with user=<bank_id>
 
 # Vertex AI defaults
 DEFAULT_LLM_VERTEXAI_PROJECT_ID = None  # Required for Vertex AI
@@ -735,6 +747,7 @@ DEFAULT_RERANKER_COHERE_MODEL = "rerank-english-v3.0"
 # OpenRouter defaults
 DEFAULT_EMBEDDINGS_OPENROUTER_MODEL = "perplexity/pplx-embed-v1-0.6b"
 DEFAULT_RERANKER_OPENROUTER_MODEL = "cohere/rerank-v3.5"
+DEFAULT_RERANKER_OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1/rerank"
 
 # ZeroEntropy defaults
 DEFAULT_EMBEDDINGS_ZEROENTROPY_MODEL = "zembed-1"
@@ -792,6 +805,13 @@ DEFAULT_MCP_ENABLED = True
 DEFAULT_MCP_ENABLED_TOOLS: list[str] | None = None  # None = all tools enabled
 DEFAULT_MCP_STATELESS = False  # False = stateful (supports SSE/GET); True = stateless (POST-only)
 DEFAULT_ENABLE_BANK_CONFIG_API = True
+# Dry-run extraction is a preview tool that makes a real LLM call but stores nothing. Enabled by
+# default; set HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT=false to remove the endpoint (e.g. to cap
+# provider cost/abuse on untrusted deployments).
+DEFAULT_ENABLE_DRY_RUN_EXTRACT = True
+# The per-bank LLM connectivity probe makes a real provider call, so it's OFF by
+# default (cost/abuse concerns) and must be explicitly enabled to expose the endpoint.
+DEFAULT_ENABLE_BANK_LLM_HEALTH = False
 DEFAULT_DEFAULT_BANK_TEMPLATE: dict | None = None  # BankTemplateManifest dict applied to newly-created banks
 DEFAULT_GRAPH_RETRIEVER = "link_expansion"
 DEFAULT_RECALL_MAX_CONCURRENT = 32  # Max concurrent recall operations per worker
@@ -831,6 +851,7 @@ DEFAULT_FILE_CONVERSION_MAX_BATCH_SIZE_MB = 100  # Max total batch size in MB (a
 DEFAULT_FILE_CONVERSION_MAX_BATCH_SIZE = 10  # Max files per batch upload
 DEFAULT_ENABLE_FILE_UPLOAD_API = True  # Enable file upload endpoint
 DEFAULT_FILE_DELETE_AFTER_RETAIN = True  # Delete file bytes after retain (saves storage)
+DEFAULT_STORE_DOCUMENT_TEXT = True  # Persist raw source text in documents.original_text / chunks.chunk_text
 
 # Document transfer defaults (export/import enabled by default; gated independently)
 DEFAULT_ENABLE_DOCUMENT_EXPORT_API = True
@@ -880,9 +901,16 @@ DEFAULT_CONSOLIDATION_SOURCE_FACTS_MAX_TOKENS_PER_OBSERVATION = (
 )
 DEFAULT_OBSERVATIONS_MISSION = None  # Declarative spec of what observations are for this bank
 DEFAULT_MAX_OBSERVATIONS_PER_SCOPE = -1  # Max observations per tag scope (-1 = unlimited)
+# Per-scope overrides of the cap above: list of {"scope": [tag-globs], "limit": int}.
+# First rule whose pattern exact-covers a scope's tags wins; else the default above.
+DEFAULT_OBSERVATION_SCOPE_LIMITS: list | None = None
 
 # Database migrations
 DEFAULT_RUN_MIGRATIONS_ON_STARTUP = True
+# Number of tenant schemas to migrate concurrently. Each schema runs in its own
+# process (Alembic's command.upgrade() is not thread-safe); within a schema the
+# work is always sequential. 1 = fully sequential (the safe default).
+DEFAULT_MIGRATION_CONCURRENCY = 1
 
 # Database connection pool
 DEFAULT_DB_POOL_MIN_SIZE = 5
@@ -1060,6 +1088,63 @@ def _parse_optional_positive_int(name: str, raw: str | None) -> int | None:
     return _parse_positive_int(name, raw, 1)
 
 
+def _validate_retain_chunking_int(name: str, value: Any) -> int:
+    if isinstance(value, bool) or not isinstance(value, int):
+        raise ValueError(f"{name} must be an integer, got {value!r}")
+    if value < 1:
+        raise ValueError(f"{name} must be >= 1, got {value}")
+    return value
+
+
+def validate_retain_chunking_config(
+    retain_chunk_size: Any,
+    retain_structured_chunk_size: Any,
+    *,
+    retain_chunk_size_name: str = "retain_chunk_size",
+    retain_structured_chunk_size_name: str = "retain_structured_chunk_size",
+) -> None:
+    """Validate retain chunking size fields.
+
+    Defaults emit field-style names ("retain_chunk_size") so API/PATCH callers
+    don't have to override them. The startup validator (HindsightConfig.validate)
+    overrides to env-style names ("HINDSIGHT_API_RETAIN_CHUNK_SIZE") for env
+    misconfig errors.
+    """
+    _validate_retain_chunking_int(retain_chunk_size_name, retain_chunk_size)
+    if retain_structured_chunk_size is None:
+        return
+    _validate_retain_chunking_int(
+        retain_structured_chunk_size_name,
+        retain_structured_chunk_size,
+    )
+
+
+def validate_retain_completion_token_budget(
+    *,
+    llm_provider: str,
+    retain_max_completion_tokens: int,
+    retain_chunk_size: int,
+    retain_llm_model: str | None = None,
+    llm_model: str | None = None,
+    retain_llm_provider: str | None = None,
+    retain_max_completion_tokens_name: str = "retain_max_completion_tokens",
+    retain_chunk_size_name: str = "retain_chunk_size",
+) -> None:
+    """Validate that retain LLM output capacity exceeds the configured chunk size."""
+    if llm_provider == "none" or retain_max_completion_tokens > retain_chunk_size:
+        return
+    raise ValueError(
+        f"Invalid configuration: {retain_max_completion_tokens_name} "
+        f"({retain_max_completion_tokens}) must be greater than "
+        f"{retain_chunk_size_name} ({retain_chunk_size}). "
+        f"\n\nYou have two options to fix this:"
+        f"\n  1. Increase {retain_max_completion_tokens_name} to a value > {retain_chunk_size}"
+        f"\n  2. Use a model that supports at least {retain_max_completion_tokens} output tokens"
+        f"\n     (current model: {retain_llm_model or llm_model}, "
+        f"provider: {retain_llm_provider or llm_provider})"
+    )
+
+
 def _parse_optional_choice(name: str, raw: str | None, allowed: frozenset[str]) -> str | None:
     """Parse an optional string env var constrained to a small allowlist."""
     if raw is None or raw == "":
@@ -1212,6 +1297,7 @@ class HindsightConfig:
     llm_reasoning_effort: str
     llm_groq_service_tier: str  # Groq: "on_demand", "flex", or "auto"
     llm_openai_service_tier: str | None  # OpenAI: None (default) or "flex" (50% cheaper)
+    llm_bedrock_service_tier: str | None  # Bedrock: None (default), "flex", "priority", or "reserved"
     llm_extra_body: (
         dict | None
     )  # Extra body params merged into OpenAI-compatible API calls (e.g. {"chat_template_kwargs": {"enable_thinking": true}})
@@ -1219,6 +1305,11 @@ class HindsightConfig:
         dict | None
     )  # Custom headers passed as default_headers to provider SDK clients (e.g. {"X-Component-Id": "hindsight"} for proxies / request tracing)
     llm_strict_schema: bool  # Grammar-enforce structured output via the provider's strongest schema mode (see DEFAULT_LLM_STRICT_SCHEMA)
+    # Tags outbound OpenAI-compatible LLM + embedding calls with `user=<bank_id>` for
+    # per-bank cost attribution. Downstream cost gateways (OpenRouter usage accounting,
+    # LiteLLM, Helicone) key attribution on the OpenAI `user` field. Opt-in; never
+    # overrides a `user` the caller already set.
+    llm_send_bank_as_user: bool
 
     # LiteLLM Router chain (provider-specific; consumed by the "litellmrouter" provider).
     # List of deployment dicts evaluated in order with fallback on transient errors.
@@ -1350,6 +1441,7 @@ class HindsightConfig:
     reranker_cohere_timeout: float
     reranker_openrouter_api_key: str | None
     reranker_openrouter_model: str
+    reranker_openrouter_base_url: str
     reranker_openrouter_timeout: float
     reranker_litellm_api_base: str
     reranker_litellm_api_key: str | None
@@ -1387,6 +1479,8 @@ class HindsightConfig:
     mcp_enabled_tools: list[str] | None  # None = all tools; explicit list = allowlist
     mcp_stateless: bool  # True = stateless HTTP (POST-only); False = stateful (supports GET/SSE)
     enable_bank_config_api: bool
+    enable_bank_llm_health: bool
+    enable_dry_run_extract: bool
     # Default bank template (static, server-level only). When set, the manifest is applied
     # to every newly-created bank, overriding the env/config defaults for any fields it sets.
     default_bank_template: dict | None
@@ -1405,6 +1499,7 @@ class HindsightConfig:
     # Retain settings
     retain_max_completion_tokens: int
     retain_chunk_size: int
+    retain_structured_chunk_size: int | None
     retain_extract_causal_links: bool
     retain_extraction_mode: str
     retain_mission: str | None
@@ -1439,6 +1534,7 @@ class HindsightConfig:
     file_conversion_max_batch_size: int  # Max files per request
     enable_file_upload_api: bool
     file_delete_after_retain: bool
+    store_document_text: bool  # When False, store NULL original_text / empty chunk_text
     enable_document_export_api: bool
     enable_document_import_api: bool
 
@@ -1462,6 +1558,10 @@ class HindsightConfig:
     consolidation_max_attempts: int
     observations_mission: str | None
     max_observations_per_scope: int
+    # Per-scope observation caps overriding max_observations_per_scope.
+    # Raw JSON shape: [{"scope": ["run_*", "shared"], "limit": 1}, ...]
+    # (validated/applied in engine.consolidation.consolidator._effective_scope_limit)
+    observation_scope_limits: list | None
 
     # Entity labels (controlled vocabulary of key:value classification labels extracted at retain time)
     # List of label group dicts: [{key, description, type, optional, values: [{value, description}]}]
@@ -1470,6 +1570,10 @@ class HindsightConfig:
     # When False: only label entities are extracted (or no entities at all if no labels configured)
     entities_allow_free_form: bool
 
+    # Memory Defense policy (dict matching DefensePolicy schema — validated on write)
+    # None = Memory Defense disabled / not configured for this bank
+    memory_defense: dict | None
+
     # Reflect agent settings
     reflect_mission: str | None
     reflect_source_facts_max_tokens: int
@@ -1504,6 +1608,7 @@ class HindsightConfig:
 
     # Database migrations
     run_migrations_on_startup: bool
+    migration_concurrency: int
 
     # Database connection pool
     db_pool_min_size: int
@@ -1594,6 +1699,7 @@ class HindsightConfig:
         "embeddings_tei_base_url",
         "reranker_tei_base_url",
         "reranker_cohere_base_url",
+        "reranker_openrouter_base_url",
         "embeddings_zeroentropy_base_url",
         "reranker_zeroentropy_base_url",
         "reranker_siliconflow_base_url",
@@ -1622,6 +1728,7 @@ class HindsightConfig:
         "mcp_enabled_tools",
         # Retention settings (behavioral)
         "retain_chunk_size",
+        "retain_structured_chunk_size",
         "retain_extraction_mode",
         "retain_mission",
         "retain_custom_instructions",
@@ -1641,6 +1748,7 @@ class HindsightConfig:
         "consolidation_source_facts_max_tokens_per_observation",
         "observations_mission",
         "max_observations_per_scope",
+        "observation_scope_limits",
         # Reflect settings
         "reflect_mission",
         "reflect_source_facts_max_tokens",
@@ -1664,6 +1772,8 @@ class HindsightConfig:
         "disposition_empathy",
         # Gemini safety settings (controls content filtering for Gemini/VertexAI providers)
         "llm_gemini_safety_settings",
+        # Memory Defense policy (validated against DefensePolicy schema on write)
+        "memory_defense",
     }
 
     @property
@@ -1759,6 +1869,16 @@ class HindsightConfig:
                 f"Invalid semantic_min_similarity: {self.semantic_min_similarity}. Must be between 0.0 and 1.0"
             )
 
+        # Validate bedrock_service_tier
+        valid_bedrock_tiers = (None, "flex", "priority", "reserved")
+        if self.llm_bedrock_service_tier not in valid_bedrock_tiers:
+            raise ValueError(
+                f"Invalid HINDSIGHT_API_LLM_BEDROCK_SERVICE_TIER: "
+                f"{self.llm_bedrock_service_tier!r}. Must be one of: "
+                f"{', '.join(t for t in valid_bedrock_tiers if t is not None)}. "
+                f"Note: 'standard' is not a valid Bedrock service tier -- use unset for default tier."
+            )
+
         # When LLM provider is "none", force chunks-only mode and disable LLM-dependent features
         if self.llm_provider == "none":
             self.retain_extraction_mode = "chunks"
@@ -1768,20 +1888,23 @@ class HindsightConfig:
                 "disabling observations/consolidation. Reflect will return HTTP 400."
             )
 
-        # RETAIN_MAX_COMPLETION_TOKENS must be greater than RETAIN_CHUNK_SIZE
-        # to ensure the LLM has enough output capacity to extract facts from chunks
-        # (not applicable when provider is "none" since no LLM calls are made)
-        if self.llm_provider != "none" and self.retain_max_completion_tokens <= self.retain_chunk_size:
-            raise ValueError(
-                f"Invalid configuration: HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS "
-                f"({self.retain_max_completion_tokens}) must be greater than "
-                f"HINDSIGHT_API_RETAIN_CHUNK_SIZE ({self.retain_chunk_size}). "
-                f"\n\nYou have two options to fix this:"
-                f"\n  1. Increase HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS to a value > {self.retain_chunk_size}"
-                f"\n  2. Use a model that supports at least {self.retain_max_completion_tokens} output tokens"
-                f"\n     (current model: {self.retain_llm_model or self.llm_model}, "
-                f"provider: {self.retain_llm_provider or self.llm_provider})"
-            )
+        validate_retain_chunking_config(
+            self.retain_chunk_size,
+            self.retain_structured_chunk_size,
+            retain_chunk_size_name="HINDSIGHT_API_RETAIN_CHUNK_SIZE",
+            retain_structured_chunk_size_name="HINDSIGHT_API_RETAIN_STRUCTURED_CHUNK_SIZE",
+        )
+
+        validate_retain_completion_token_budget(
+            llm_provider=self.llm_provider,
+            retain_max_completion_tokens=self.retain_max_completion_tokens,
+            retain_chunk_size=self.retain_chunk_size,
+            retain_llm_model=self.retain_llm_model,
+            llm_model=self.llm_model,
+            retain_llm_provider=self.retain_llm_provider,
+            retain_max_completion_tokens_name="HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS",
+            retain_chunk_size_name="HINDSIGHT_API_RETAIN_CHUNK_SIZE",
+        )
 
         # Warn if local ML dependencies are missing when configured.
         # Don't hard-fail here — the actual ImportError fires at model init time
@@ -1872,9 +1995,12 @@ class HindsightConfig:
             llm_reasoning_effort=os.getenv(ENV_LLM_REASONING_EFFORT, DEFAULT_LLM_REASONING_EFFORT),
             llm_groq_service_tier=os.getenv(ENV_LLM_GROQ_SERVICE_TIER, DEFAULT_LLM_GROQ_SERVICE_TIER),
             llm_openai_service_tier=os.getenv(ENV_LLM_OPENAI_SERVICE_TIER, DEFAULT_LLM_OPENAI_SERVICE_TIER),
+            llm_bedrock_service_tier=os.getenv(ENV_LLM_BEDROCK_SERVICE_TIER) or None,
             llm_extra_body=json.loads(os.getenv(ENV_LLM_EXTRA_BODY, "null")),
             llm_default_headers=json.loads(os.getenv(ENV_LLM_DEFAULT_HEADERS, "null")),
             llm_strict_schema=os.getenv(ENV_LLM_STRICT_SCHEMA, str(DEFAULT_LLM_STRICT_SCHEMA)).lower() in ("true", "1"),
+            llm_send_bank_as_user=os.getenv(ENV_LLM_SEND_BANK_AS_USER, str(DEFAULT_LLM_SEND_BANK_AS_USER)).lower()
+            in ("true", "1"),
             llm_litellmrouter_config=_parse_llm_router_config(ENV_LLM_LITELLMROUTER_CONFIG),
             # Vertex AI
             llm_vertexai_project_id=os.getenv(ENV_LLM_VERTEXAI_PROJECT_ID) or DEFAULT_LLM_VERTEXAI_PROJECT_ID,
@@ -2154,6 +2280,9 @@ class HindsightConfig:
             or os.getenv(ENV_OPENROUTER_API_KEY)
             or os.getenv(ENV_LLM_API_KEY),
             reranker_openrouter_model=os.getenv(ENV_RERANKER_OPENROUTER_MODEL, DEFAULT_RERANKER_OPENROUTER_MODEL),
+            reranker_openrouter_base_url=os.getenv(
+                ENV_RERANKER_OPENROUTER_BASE_URL, DEFAULT_RERANKER_OPENROUTER_BASE_URL
+            ),
             reranker_openrouter_timeout=float(
                 os.getenv(ENV_RERANKER_OPENROUTER_TIMEOUT, str(DEFAULT_RERANKER_OPENROUTER_TIMEOUT))
             ),
@@ -2216,8 +2345,12 @@ class HindsightConfig:
             if os.getenv(ENV_MCP_ENABLED_TOOLS)
             else DEFAULT_MCP_ENABLED_TOOLS,
             mcp_stateless=os.getenv(ENV_MCP_STATELESS, str(DEFAULT_MCP_STATELESS)).lower() == "true",
+            enable_bank_llm_health=os.getenv(ENV_ENABLE_BANK_LLM_HEALTH, str(DEFAULT_ENABLE_BANK_LLM_HEALTH)).lower()
+            == "true",
             enable_bank_config_api=os.getenv(ENV_ENABLE_BANK_CONFIG_API, str(DEFAULT_ENABLE_BANK_CONFIG_API)).lower()
             == "true",
+            enable_dry_run_extract=os.getenv(ENV_ENABLE_DRY_RUN_EXTRACT, str(DEFAULT_ENABLE_DRY_RUN_EXTRACT)).lower()
+            == "true",
             default_bank_template=_parse_default_bank_template(os.getenv(ENV_DEFAULT_BANK_TEMPLATE)),
             # Recall
             graph_retriever=os.getenv(ENV_GRAPH_RETRIEVER, DEFAULT_GRAPH_RETRIEVER),
@@ -2247,6 +2380,10 @@ class HindsightConfig:
                 os.getenv(ENV_RETAIN_MAX_COMPLETION_TOKENS, str(DEFAULT_RETAIN_MAX_COMPLETION_TOKENS))
             ),
             retain_chunk_size=int(os.getenv(ENV_RETAIN_CHUNK_SIZE, str(DEFAULT_RETAIN_CHUNK_SIZE))),
+            retain_structured_chunk_size=_parse_optional_positive_int(
+                ENV_RETAIN_STRUCTURED_CHUNK_SIZE,
+                os.getenv(ENV_RETAIN_STRUCTURED_CHUNK_SIZE),
+            ),
             retain_extract_causal_links=os.getenv(
                 ENV_RETAIN_EXTRACT_CAUSAL_LINKS, str(DEFAULT_RETAIN_EXTRACT_CAUSAL_LINKS)
             ).lower()
@@ -2302,6 +2439,7 @@ class HindsightConfig:
                 ENV_FILE_DELETE_AFTER_RETAIN, str(DEFAULT_FILE_DELETE_AFTER_RETAIN)
             ).lower()
             == "true",
+            store_document_text=os.getenv(ENV_STORE_DOCUMENT_TEXT, str(DEFAULT_STORE_DOCUMENT_TEXT)).lower() == "true",
             enable_document_export_api=os.getenv(
                 ENV_ENABLE_DOCUMENT_EXPORT_API, str(DEFAULT_ENABLE_DOCUMENT_EXPORT_API)
             ).lower()
@@ -2385,10 +2523,14 @@ class HindsightConfig:
             max_observations_per_scope=int(
                 os.getenv(ENV_MAX_OBSERVATIONS_PER_SCOPE, str(DEFAULT_MAX_OBSERVATIONS_PER_SCOPE))
             ),
+            observation_scope_limits=json.loads(os.getenv(ENV_OBSERVATION_SCOPE_LIMITS, "null"))
+            or DEFAULT_OBSERVATION_SCOPE_LIMITS,
             entity_labels=None,
             entities_allow_free_form=True,
+            memory_defense=None,
             # Database migrations
             run_migrations_on_startup=os.getenv(ENV_RUN_MIGRATIONS_ON_STARTUP, "true").lower() == "true",
+            migration_concurrency=int(os.getenv(ENV_MIGRATION_CONCURRENCY, str(DEFAULT_MIGRATION_CONCURRENCY))),
             # Database connection pool
             db_pool_min_size=int(os.getenv(ENV_DB_POOL_MIN_SIZE, str(DEFAULT_DB_POOL_MIN_SIZE))),
             db_pool_max_size=int(os.getenv(ENV_DB_POOL_MAX_SIZE, str(DEFAULT_DB_POOL_MAX_SIZE))),

hindsight-api-slim/hindsight_api/config_resolver.py

@@ -18,6 +18,8 @@ from hindsight_api.config import (
     HindsightConfig,
     _get_raw_config,
     normalize_config_dict,
+    validate_retain_chunking_config,
+    validate_retain_completion_token_budget,
 )
 from hindsight_api.engine.memory_engine import fq_table
 from hindsight_api.extensions.tenant import TenantExtension
@@ -29,6 +31,35 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
+def _validate_retain_strategy_chunking(base_config: HindsightConfig, strategies: Any) -> None:
+    """Validate retain strategy chunking with the same semantics as apply_strategy()."""
+    if not isinstance(strategies, dict):
+        return
+    configurable = HindsightConfig.get_configurable_fields()
+    for strategy_name, overrides in strategies.items():
+        if not isinstance(overrides, dict):
+            raise ValueError(f"Invalid retain strategy {strategy_name!r}: must be an object")
+        filtered = {k: v for k, v in overrides.items() if k in configurable}
+        if not filtered:
+            continue
+        try:
+            resolved = replace(base_config, **filtered)
+            validate_retain_chunking_config(
+                resolved.retain_chunk_size,
+                resolved.retain_structured_chunk_size,
+            )
+            validate_retain_completion_token_budget(
+                llm_provider=resolved.llm_provider,
+                retain_max_completion_tokens=resolved.retain_max_completion_tokens,
+                retain_chunk_size=resolved.retain_chunk_size,
+                retain_llm_model=resolved.retain_llm_model,
+                llm_model=resolved.llm_model,
+                retain_llm_provider=resolved.retain_llm_provider,
+            )
+        except ValueError as e:
+            raise ValueError(f"Invalid retain strategy {strategy_name!r}: {e}") from e
+
+
 class ConfigResolver:
     """Resolves hierarchical configuration with tenant/bank overrides."""
 
@@ -46,6 +77,26 @@ class ConfigResolver:
         self._configurable_fields = HindsightConfig.get_configurable_fields()
         self._credential_fields = HindsightConfig.get_credential_fields()
 
+    async def _resolve_parent_config_dict(self, bank_id: str, context: RequestContext | None = None) -> dict[str, Any]:
+        """Resolve global + tenant config before bank-level overrides."""
+        config_dict = asdict(self._global_config)
+
+        if self.tenant_extension and context:
+            try:
+                tenant_overrides = await self.tenant_extension.get_tenant_config(context)
+                if tenant_overrides:
+                    # Normalize keys and filter to configurable fields only
+                    normalized_tenant = normalize_config_dict(tenant_overrides)
+                    configurable_tenant = {k: v for k, v in normalized_tenant.items() if k in self._configurable_fields}
+                    config_dict.update(configurable_tenant)
+                    logger.debug(
+                        f"Applied tenant config overrides for bank {bank_id}: {list(configurable_tenant.keys())}"
+                    )
+            except Exception as e:
+                logger.warning(f"Failed to load tenant config for bank {bank_id}: {e}")
+
+        return config_dict
+
     async def resolve_full_config(self, bank_id: str, context: RequestContext | None = None) -> HindsightConfig:
         """
         Resolve full HindsightConfig for a bank with hierarchical overrides applied.
@@ -65,23 +116,7 @@ class ConfigResolver:
         Returns:
             Complete HindsightConfig with hierarchical overrides applied
         """
-        # Start with global config (all fields)
-        config_dict = asdict(self._global_config)
-
-        # Load tenant config overrides (if tenant extension available)
-        if self.tenant_extension and context:
-            try:
-                tenant_overrides = await self.tenant_extension.get_tenant_config(context)
-                if tenant_overrides:
-                    # Normalize keys and filter to configurable fields only
-                    normalized_tenant = normalize_config_dict(tenant_overrides)
-                    configurable_tenant = {k: v for k, v in normalized_tenant.items() if k in self._configurable_fields}
-                    config_dict.update(configurable_tenant)
-                    logger.debug(
-                        f"Applied tenant config overrides for bank {bank_id}: {list(configurable_tenant.keys())}"
-                    )
-            except Exception as e:
-                logger.warning(f"Failed to load tenant config for bank {bank_id}: {e}")
+        config_dict = await self._resolve_parent_config_dict(bank_id, context)
 
         # Load bank config overrides
         bank_overrides = await self._load_bank_config(bank_id)
@@ -92,6 +127,10 @@ class ConfigResolver:
         # Return full config object (dataclass doesn't have __init__ that accepts kwargs, so we update the object)
         # Create a new config instance by copying the global config and updating fields
         resolved_config = HindsightConfig(**config_dict)
+        validate_retain_chunking_config(
+            resolved_config.retain_chunk_size,
+            resolved_config.retain_structured_chunk_size,
+        )
         return resolved_config
 
     async def get_bank_config(self, bank_id: str, context: RequestContext | None = None) -> dict[str, Any]:
@@ -266,6 +305,29 @@ class ConfigResolver:
         # Validate recall budget fields
         _validate_recall_budget_updates(normalized_updates)
 
+        chunking_fields_updated = (
+            "retain_chunk_size" in normalized_updates
+            or "retain_structured_chunk_size" in normalized_updates
+            or "retain_strategies" in normalized_updates
+        )
+        if chunking_fields_updated:
+            config_dict = await self._resolve_parent_config_dict(bank_id, context)
+            active_bank_overrides = await self._load_bank_config(bank_id)
+            for key, value in normalized_updates.items():
+                if key not in self._configurable_fields:
+                    continue
+                if value is None:
+                    active_bank_overrides.pop(key, None)
+                else:
+                    active_bank_overrides[key] = value
+            config_dict.update(active_bank_overrides)
+            base_config = HindsightConfig(**config_dict)
+            validate_retain_chunking_config(
+                base_config.retain_chunk_size,
+                base_config.retain_structured_chunk_size,
+            )
+            _validate_retain_strategy_chunking(base_config, base_config.retain_strategies)
+
         # Persist the override. Banks are created lazily (on first retain), so a
         # PATCH that precedes any ingestion would otherwise UPDATE zero rows and
         # silently no-op while returning 200. Ensure the bank row exists first
@@ -364,7 +426,8 @@ def apply_strategy(config: HindsightConfig, strategy_name: str) -> HindsightConf
     A strategy is a named set of hierarchical field overrides stored in
     config.retain_strategies. Any field in _HIERARCHICAL_FIELDS can be
     overridden, including retain_extraction_mode, retain_chunk_size,
-    entity_labels, entities_allow_free_form, etc.
+    retain_structured_chunk_size, entity_labels,
+    entities_allow_free_form, etc.
 
     Unknown strategy names log a warning and return config unchanged.
     Unknown or non-hierarchical fields in the strategy are silently ignored.
@@ -386,4 +449,17 @@ def apply_strategy(config: HindsightConfig, strategy_name: str) -> HindsightConf
         return config
 
     logger.debug(f"Applying retain strategy '{strategy_name}': {list(filtered.keys())}")
-    return replace(config, **filtered)
+    resolved = replace(config, **filtered)
+    validate_retain_chunking_config(
+        resolved.retain_chunk_size,
+        resolved.retain_structured_chunk_size,
+    )
+    validate_retain_completion_token_budget(
+        llm_provider=resolved.llm_provider,
+        retain_max_completion_tokens=resolved.retain_max_completion_tokens,
+        retain_chunk_size=resolved.retain_chunk_size,
+        retain_llm_model=resolved.retain_llm_model,
+        llm_model=resolved.llm_model,
+        retain_llm_provider=resolved.retain_llm_provider,
+    )
+    return resolved

hindsight-api-slim/hindsight_api/engine/bank_attribution.py

@@ -0,0 +1,34 @@
+"""Per-bank provider cost attribution via the OpenAI ``user`` field.
+
+Shared by the OpenAI-compatible LLM path and the OpenAI embeddings path so both
+tag outbound requests identically. Opt-in via ``HINDSIGHT_API_LLM_SEND_BANK_AS_USER``;
+downstream cost gateways (OpenRouter usage accounting, LiteLLM, Helicone) key spend
+on the OpenAI ``user`` field.
+
+Note: when enabled, the bank id is transmitted to the upstream provider as the
+end-user identifier. Banks that are themselves end-user identifiers are therefore
+forwarded to the provider — which is exactly what the OpenAI ``user`` field is for,
+but operators should opt in with that in mind.
+"""
+
+from typing import Any
+
+
+def apply_bank_attribution(request: dict[str, Any]) -> None:
+    """Tag ``request`` with ``user=<bank_id>`` for per-bank cost attribution.
+
+    Mutates ``request`` in place. No-op when the flag is off, no bank is in context,
+    or the caller already set ``user`` — we never override an explicit value.
+    """
+    if "user" in request:
+        return
+    # Lazy imports: memory_engine imports the embeddings/provider modules that call
+    # this, so a top-level import of memory_engine here would be circular.
+    from ..config import get_config
+    from .memory_engine import get_current_bank_id
+
+    if not get_config().llm_send_bank_as_user:
+        return
+    bank_id = get_current_bank_id()
+    if bank_id:
+        request["user"] = bank_id

hindsight-api-slim/hindsight_api/engine/consolidation/consolidator.py

@@ -24,6 +24,7 @@ from collections import defaultdict
 from contextlib import AsyncExitStack
 from dataclasses import asdict, dataclass, field
 from datetime import datetime, timezone
+from fnmatch import fnmatchcase
 from itertools import combinations
 from typing import TYPE_CHECKING, Any, Literal
 
@@ -334,7 +335,15 @@ def _resolve_obs_tags_list(memory: dict[str, Any]) -> list[list[str]] | None:
 
     Returns ``None`` for the default ``combined``-mode single pass (caller uses
     the memory's own tags). Returns a list[list[str]] when the memory requested
-    multi-pass scoping (``per_tag``, ``all_combinations``, or an explicit list).
+    multi-pass scoping (``per_tag``, ``all_combinations``, ``shared``, or an
+    explicit list).
+
+    ``shared`` resolves to ``[[]]`` — a single pass over the empty (untagged)
+    scope. The created observation carries no tags and recall/dedup match it with
+    ``tags_match="any"``, so every memory consolidates into one shared observation
+    regardless of its own tags. Use it to deduplicate across volatile per-call
+    provenance tags (e.g. per-session ids) without dropping those tags from the
+    source facts.
     """
     parsed = _parse_observation_scopes(memory)
     tags = list(memory.get("tags") or [])
@@ -345,6 +354,8 @@ def _resolve_obs_tags_list(memory: dict[str, Any]) -> list[list[str]] | None:
         if not tags:
             return None
         return [list(c) for r in range(1, len(tags) + 1) for c in combinations(tags, r)]
+    if parsed == "shared":
+        return [[]]
     if parsed == "combined" or parsed is None:
         return None
     return parsed  # explicit list[list[str]]
@@ -361,6 +372,7 @@ def _resolve_write_scopes(memory: dict[str, Any]) -> list[frozenset[str]]:
     - ``combined`` / ``None``    -> ``[frozenset(memory.tags)]``
     - ``per_tag``                -> ``[frozenset({t}) for t in memory.tags]``
     - ``all_combinations``       -> one frozenset per nonempty subset of tags
+    - ``shared``                 -> ``[frozenset()]`` (the single untagged scope)
     - explicit ``list[list[str]]`` -> one frozenset per declared scope
 
     Empty-tag memories collapse to a single ``frozenset()`` in all modes so they
@@ -375,6 +387,8 @@ def _resolve_write_scopes(memory: dict[str, Any]) -> list[frozenset[str]]:
         if not tags:
             return [frozenset()]
         return [frozenset(c) for r in range(1, len(tags) + 1) for c in combinations(tags, r)]
+    if parsed == "shared":
+        return [frozenset()]
     if parsed == "combined" or parsed is None:
         return [frozenset(tags)]
     return [frozenset(s) for s in parsed]  # explicit list[list[str]]
@@ -523,6 +537,86 @@ async def _count_observations_for_scope(
     )
 
 
+@dataclass(frozen=True)
+class _ScopeLimitRule:
+    """One ``observation_scope_limits`` rule: a scope pattern -> an observation cap.
+
+    ``globs`` is a tuple of fnmatch tag-globs describing one consolidation scope.
+    A concrete scope (the set of ``fact_tags`` for a consolidation pass) matches
+    under *exact cover*: every tag is matched by some glob AND every glob matches
+    some tag. So ``["shared"]`` matches the scope ``{shared}`` but not
+    ``{run_1, shared}``, and ``["run_*", "shared"]`` matches ``{run_1, shared}``
+    but not ``{shared}``.
+
+    ``limit`` is the cap applied to matching scopes (-1 = unlimited, 0 = no new
+    observations, >0 = hard cap), mirroring ``max_observations_per_scope``.
+    """
+
+    globs: tuple[str, ...]
+    limit: int
+
+
+def _parse_scope_limit_rules(raw: Any) -> list[_ScopeLimitRule]:
+    """Parse the raw ``observation_scope_limits`` config into ordered rules.
+
+    The config round-trips as JSON through env and the bank-config API, so this
+    is defensive: malformed entries are skipped rather than raising, and list
+    order is preserved (first match wins in :func:`_effective_scope_limit`).
+    """
+    if not isinstance(raw, list):
+        return []
+    rules: list[_ScopeLimitRule] = []
+    for entry in raw:
+        if not isinstance(entry, dict):
+            continue
+        scope = entry.get("scope")
+        limit = entry.get("limit")
+        if not isinstance(scope, list) or not scope:
+            continue
+        if not all(isinstance(g, str) and g for g in scope):
+            continue
+        # bool is an int subclass — reject True/False masquerading as a limit.
+        if not isinstance(limit, int) or isinstance(limit, bool):
+            continue
+        rules.append(_ScopeLimitRule(globs=tuple(scope), limit=limit))
+    return rules
+
+
+def _scope_matches_globs(globs: tuple[str, ...], tags: list[str]) -> bool:
+    """Exact-cover match between a scope pattern and a concrete tag set.
+
+    True iff every tag is covered by at least one glob AND every glob covers at
+    least one tag (no uncovered tags, no vacuous globs). Untagged scopes never
+    match, so a scope limit never applies to untagged observations (consistent
+    with the ``and fact_tags`` guard at the call site). Matching is
+    case-sensitive (``fnmatchcase``) for deterministic cross-platform behaviour.
+    """
+    tagset = set(tags)
+    if not tagset:
+        return False
+    if not all(any(fnmatchcase(t, g) for g in globs) for t in tagset):
+        return False
+    if not all(any(fnmatchcase(t, g) for t in tagset) for g in globs):
+        return False
+    return True
+
+
+def _effective_scope_limit(config: Any, fact_tags: list[str]) -> int:
+    """Resolve the observation cap for one concrete consolidation scope.
+
+    The first rule in ``observation_scope_limits`` whose pattern exact-covers
+    ``fact_tags`` wins; otherwise falls back to the bank-wide
+    ``max_observations_per_scope``. Wildcards live only here, matched against the
+    already-resolved concrete tags — the SQL count stays exact and indexed.
+    """
+    if config is None:
+        return -1
+    for rule in _parse_scope_limit_rules(getattr(config, "observation_scope_limits", None)):
+        if _scope_matches_globs(rule.globs, fact_tags):
+            return rule.limit
+    return config.max_observations_per_scope
+
+
 def _build_response_model(max_creates: int | None = None) -> type[_ConsolidationBatchResponse]:
     """Build a response model, optionally constraining max creates via JSON schema."""
     if max_creates is None or max_creates < 0:
@@ -1403,11 +1497,15 @@ async def _process_memory_batch(
         # All memories in the batch share the same tag set (enforced by batching)
         fact_tags = memories[0].get("tags") or [] if memories else []
 
-    # 2b. Compute remaining observation slots for this scope (if limit configured)
-    max_obs = config.max_observations_per_scope if config is not None else -1
+    # 2b. Compute remaining observation slots for this scope (if limit configured).
+    # The cap is resolved per-scope: an observation_scope_limits rule may override
+    # the bank-wide max_observations_per_scope for scopes matching its tag pattern.
+    max_obs = _effective_scope_limit(config, fact_tags)
     remaining_observation_slots: int | None = None
-    if max_obs > 0 and fact_tags:
-        current_count = await _count_observations_for_scope(conn, bank_id, fact_tags)
+    if max_obs >= 0 and fact_tags:
+        # max_obs == 0 means "no new observations": there are no slots regardless
+        # of the current count, so skip the count query for that case.
+        current_count = await _count_observations_for_scope(conn, bank_id, fact_tags) if max_obs > 0 else 0
         remaining_observation_slots = max(max_obs - current_count, 0)
         if remaining_observation_slots == 0:
             logger.info(
@@ -2045,7 +2143,7 @@ async def _consolidate_batch_with_llm(
 
     # Build capacity note for the prompt when observation limit is configured
     observation_capacity_note: str | None = None
-    if remaining_observation_slots is not None and max_observations_per_scope > 0:
+    if remaining_observation_slots is not None and max_observations_per_scope >= 0:
         if remaining_observation_slots == 0:
             observation_capacity_note = (
                 f"OBSERVATION LIMIT REACHED ({max_observations_per_scope}/{max_observations_per_scope}). "

hindsight-api-slim/hindsight_api/engine/cross_encoder.py

@@ -27,34 +27,21 @@ from ..config import (
     DEFAULT_RERANKER_LITELLM_MODEL,
     DEFAULT_RERANKER_LITELLM_SDK_MODEL,
     DEFAULT_RERANKER_LOCAL_BATCH_SIZE,
-    DEFAULT_RERANKER_LOCAL_FORCE_CPU,
-    DEFAULT_RERANKER_LOCAL_MAX_CONCURRENT,
     DEFAULT_RERANKER_LOCAL_MODEL,
-    DEFAULT_RERANKER_LOCAL_TRUST_REMOTE_CODE,
-    DEFAULT_RERANKER_PROVIDER,
     DEFAULT_RERANKER_SILICONFLOW_BASE_URL,
     DEFAULT_RERANKER_SILICONFLOW_MODEL,
     DEFAULT_RERANKER_TEI_BATCH_SIZE,
-    DEFAULT_RERANKER_TEI_HTTP_TIMEOUT,
     DEFAULT_RERANKER_TEI_MAX_CONCURRENT,
     DEFAULT_RERANKER_ZEROENTROPY_MODEL,
     DEFAULT_ZEROENTROPY_BASE_URL,
     ENV_RERANKER_ALIBABA_API_KEY,
     ENV_RERANKER_COHERE_API_KEY,
-    ENV_RERANKER_COHERE_MODEL,
     ENV_RERANKER_FLASHRANK_CACHE_DIR,
     ENV_RERANKER_FLASHRANK_CPU_MEM_ARENA,
     ENV_RERANKER_FLASHRANK_MODEL,
     ENV_RERANKER_GOOGLE_PROJECT_ID,
-    ENV_RERANKER_LOCAL_FORCE_CPU,
-    ENV_RERANKER_LOCAL_MAX_CONCURRENT,
-    ENV_RERANKER_LOCAL_MODEL,
-    ENV_RERANKER_LOCAL_TRUST_REMOTE_CODE,
     ENV_RERANKER_PROVIDER,
     ENV_RERANKER_SILICONFLOW_API_KEY,
-    ENV_RERANKER_TEI_BATCH_SIZE,
-    ENV_RERANKER_TEI_HTTP_TIMEOUT,
-    ENV_RERANKER_TEI_MAX_CONCURRENT,
     ENV_RERANKER_TEI_URL,
     ENV_RERANKER_ZEROENTROPY_API_KEY,
 )
@@ -303,7 +290,6 @@ class LocalSTCrossEncoder(CrossEncoderModel):
         - bucket_batching: sort pairs by token length to reduce padding waste (36-54% speedup)
         - batch_size: explicit batch size for predict() calls (MPS optimal: 32)
         """
-        import numpy as np
 
         try:
             if self.bucket_batching and len(pairs) > 1:
@@ -1679,7 +1665,7 @@ def create_cross_encoder_from_env() -> CrossEncoderModel:
         return CohereCrossEncoder(
             api_key=api_key,
             model=config.reranker_openrouter_model,
-            base_url="https://openrouter.ai/api/v1/rerank",
+            base_url=config.reranker_openrouter_base_url,
             timeout=config.reranker_openrouter_timeout,
         )
     elif provider == "flashrank":

hindsight-api-slim/hindsight_api/engine/db/ops.py

@@ -19,7 +19,6 @@ and mirrors Django's ``DatabaseOperations`` architecture.
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 from typing import Any
-from uuid import UUID
 
 from .base import DatabaseConnection
 from .result import ResultRow

hindsight-api-slim/hindsight_api/engine/db/ops_oracle.py

@@ -8,8 +8,6 @@ columns can't appear in GROUP BY).
 import json
 import uuid as uuid_mod
 from datetime import UTC, datetime
-from typing import Any
-from uuid import UUID
 
 from .base import DatabaseConnection
 from .ops import DataAccessOps, TagListingParts

hindsight-api-slim/hindsight_api/engine/db/ops_postgresql.py

@@ -4,11 +4,6 @@ Uses unnest(), LATERAL, DISTINCT ON, and native array operations for
 efficient batch operations.
 """
 
-import json
-from datetime import UTC, datetime
-from typing import Any
-from uuid import UUID
-
 from .base import DatabaseConnection
 from .ops import DataAccessOps, TagListingParts
 from .result import ResultRow
@@ -620,7 +615,6 @@ class PostgreSQLOps(DataAccessOps):
         per_entity_limit: int,
     ) -> tuple[list[ResultRow], list[ResultRow], list[ResultRow]]:
         # v0.5.6 array ops: unnest, &&, COUNT(DISTINCT) on source_memory_ids.
-        from ..schema import fq_table
 
         entity_rows = await conn.fetch(
             f"""

hindsight-api-slim/hindsight_api/engine/db/optional_routines.py

@@ -106,6 +106,15 @@ SCHEMAS_WITH_PENDING_WORK = OptionalRoutine(
         deployment.
       * Should be cheap and idempotent — called every poll cycle (~30s).
 
+    The poller trusts the result wholesale: any schema the routine does
+    not return is treated as having no work this cycle. It does NOT
+    second-guess omissions with a per-schema scan — that would re-run the
+    exact queries this routine exists to avoid. Consequently the routine
+    is *only* appropriate for multi-tenant deployments. Single-schema
+    (default/public only) installs should NOT create it: the per-schema
+    fallback below is a single cheap EXISTS check that covers ``public``
+    correctly and cannot starve.
+
     Fallback when the routine is absent: per-schema ``EXISTS`` queries
     from Python (~4ms per schema). The server-side path is a single-
     round-trip optimisation worth ~200ms in deployments with thousands

hindsight-api-slim/hindsight_api/engine/embeddings.py

@@ -26,11 +26,8 @@ from ..config import (
     DEFAULT_EMBEDDINGS_GEMINI_MODEL,
     DEFAULT_EMBEDDINGS_LITELLM_MODEL,
     DEFAULT_EMBEDDINGS_LITELLM_SDK_MODEL,
-    DEFAULT_EMBEDDINGS_LOCAL_FORCE_CPU,
     DEFAULT_EMBEDDINGS_LOCAL_MODEL,
-    DEFAULT_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE,
     DEFAULT_EMBEDDINGS_OPENAI_MODEL,
-    DEFAULT_EMBEDDINGS_PROVIDER,
     DEFAULT_EMBEDDINGS_ZEROENTROPY_BATCH_SIZE,
     DEFAULT_EMBEDDINGS_ZEROENTROPY_DIMENSIONS,
     DEFAULT_EMBEDDINGS_ZEROENTROPY_ENCODING_FORMAT,
@@ -40,13 +37,6 @@ from ..config import (
     DEFAULT_ZEROENTROPY_BASE_URL,
     ENV_EMBEDDINGS_COHERE_API_KEY,
     ENV_EMBEDDINGS_GEMINI_API_KEY,
-    ENV_EMBEDDINGS_LOCAL_FORCE_CPU,
-    ENV_EMBEDDINGS_LOCAL_MODEL,
-    ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE,
-    ENV_EMBEDDINGS_ONNX_DIMENSIONS,
-    ENV_EMBEDDINGS_ONNX_MODEL_ID,
-    ENV_EMBEDDINGS_ONNX_MODEL_PATH,
-    ENV_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH,
     ENV_EMBEDDINGS_OPENAI_API_KEY,
     ENV_EMBEDDINGS_OPENAI_BASE_URL,
     ENV_EMBEDDINGS_OPENAI_MODEL,
@@ -57,6 +47,7 @@ from ..config import (
     ENV_EMBEDDINGS_ZEROENTROPY_ENCODING_FORMAT,
     ENV_LLM_API_KEY,
 )
+from .bank_attribution import apply_bank_attribution
 
 logger = logging.getLogger(__name__)
 
@@ -705,6 +696,7 @@ class OpenAIEmbeddings(Embeddings):
             }
             if self.dimensions is not None:
                 request["dimensions"] = self.dimensions
+            apply_bank_attribution(request)
 
             response = self._client.embeddings.create(**request)
 
@@ -1347,6 +1339,21 @@ class LiteLLMSDKEmbeddings(Embeddings):
         return all_embeddings
 
 
+# Gemini Embedding 2+ multimodal models return a SINGLE aggregated embedding
+# for a multi-input request instead of one vector per input (see
+# https://ai.google.dev/gemini-api/docs/embeddings#embedding-aggregation). For
+# these models we must embed one input per call to preserve the 1:1 input→vector
+# alignment the rest of the pipeline relies on. The marker matches preview and GA
+# names (e.g. "gemini-embedding-2-preview", "gemini-embedding-2"), with or
+# without a "google/" or "models/" prefix.
+_GEMINI_AGGREGATING_MODEL_MARKER = "gemini-embedding-2"
+
+
+def _gemini_model_aggregates_inputs(model: str) -> bool:
+    """Whether the model aggregates a multi-input request into one embedding."""
+    return _GEMINI_AGGREGATING_MODEL_MARKER in model.lower()
+
+
 class GeminiEmbeddings(Embeddings):
     """
     Google embeddings via the google.genai SDK.
@@ -1356,6 +1363,10 @@ class GeminiEmbeddings(Embeddings):
     2. Vertex AI with service account or Application Default Credentials (ADC)
 
     Uses the embed_content API: client.models.embed_content(model, contents)
+
+    Gemini Embedding 2+ multimodal models aggregate a multi-input request into a
+    single embedding, so for those the batch size is forced to 1 (one input per
+    call) to keep one vector per input.
     """
 
     def __init__(
@@ -1510,9 +1521,13 @@ class GeminiEmbeddings(Embeddings):
 
         all_embeddings = []
 
+        # Gemini Embedding 2+ multimodal models return one aggregated vector for a
+        # multi-input request, so embed one input per call to keep 1:1 alignment.
+        batch_size = 1 if _gemini_model_aggregates_inputs(self.model) else self.batch_size
+
         # Process in batches
-        for i in range(0, len(texts), self.batch_size):
-            batch = texts[i : i + self.batch_size]
+        for i in range(0, len(texts), batch_size):
+            batch = texts[i : i + batch_size]
 
             embed_kwargs = {"model": self.model, "contents": batch}
             if self._embed_config is not None:
@@ -1520,7 +1535,13 @@ class GeminiEmbeddings(Embeddings):
 
             result = self._client.models.embed_content(**embed_kwargs)
 
-            all_embeddings.extend([emb.values for emb in result.embeddings])
+            embeddings = result.embeddings or []
+            if len(embeddings) != len(batch):
+                raise RuntimeError(
+                    f"Gemini embeddings backend returned {len(embeddings)} vectors for "
+                    f"{len(batch)} input texts (model {self.model}); expected exact 1:1 alignment"
+                )
+            all_embeddings.extend([emb.values for emb in embeddings])
 
         # L2-normalize when output_dimensionality is set — Gemini only returns
         # normalized vectors at full 3072 dims; truncated dims need re-normalization

hindsight-api-slim/hindsight_api/engine/entity_resolver.py

@@ -834,14 +834,12 @@ class EntityResolver:
 
             best_candidate = None
             best_score = 0.0
-            best_name_similarity = 0.0
 
             nearby_entity_set = {e["text"].lower() for e in nearby_entities if e["text"] != entity_text}
 
             for row in candidates:
                 candidate_id = row["id"]
                 canonical_name = row["canonical_name"]
-                metadata = row["metadata"]
                 last_seen = row["last_seen"]
                 score = 0.0
 
@@ -888,7 +886,6 @@ class EntityResolver:
                 if score > best_score:
                     best_score = score
                     best_candidate = candidate_id
-                    best_name_similarity = name_similarity
 
             # Threshold for considering it the same entity
             threshold = 0.6

hindsight-api-slim/hindsight_api/engine/interface.py

@@ -10,7 +10,7 @@ from datetime import datetime
 from typing import TYPE_CHECKING, Any
 
 if TYPE_CHECKING:
-    from hindsight_api.engine.memory_engine import Budget
+    from hindsight_api.engine.memory_engine import BankLlmHealthInfo, Budget
     from hindsight_api.engine.response_models import RecallResult, ReflectResult
     from hindsight_api.engine.search.tags import TagsMatch
     from hindsight_api.models import RequestContext
@@ -483,6 +483,20 @@ class MemoryEngineInterface(ABC):
         """
         ...
 
+    @abstractmethod
+    async def check_bank_llm(
+        self,
+        bank_id: str,
+        *,
+        request_context: "RequestContext",
+    ) -> "BankLlmHealthInfo":
+        """
+        Probe the LLM consolidation would use for this bank. Deliberate connectivity
+        test (one real minimal call); never returns the API key. See
+        MemoryEngine.check_bank_llm.
+        """
+        ...
+
     @abstractmethod
     async def get_entity(
         self,

hindsight-api-slim/hindsight_api/engine/llm_interface.py

@@ -8,7 +8,7 @@ enabling support for multiple LLM backends (OpenAI, Anthropic, Gemini, Codex, et
 from abc import ABC, abstractmethod
 from typing import Any
 
-from .response_models import LLMToolCallResult, TokenUsage
+from .response_models import LLMToolCallResult
 
 
 class LLMInterface(ABC):

hindsight-api-slim/hindsight_api/engine/llm_wrapper.py

@@ -11,14 +11,10 @@ import time
 import uuid
 from contextlib import AsyncExitStack
 from pathlib import Path
-from typing import Any
-
-import httpx
-from openai import APIConnectionError, APIStatusError, AsyncOpenAI, LengthFinishReasonError
+from typing import TYPE_CHECKING, Any
 
 # Vertex AI imports (conditional - for LLMProvider to pass credentials to GeminiLLM)
 try:
-    import google.auth
     from google.oauth2 import service_account
 
     VERTEXAI_AVAILABLE = True
@@ -27,16 +23,14 @@ except ImportError:
 
 from ..config import (
     DEFAULT_LLM_MAX_CONCURRENT,
-    DEFAULT_LLM_TIMEOUT,
     ENV_CONSOLIDATION_LLM_MAX_CONCURRENT,
-    ENV_LLM_GROQ_SERVICE_TIER,
     ENV_LLM_MAX_CONCURRENT,
-    ENV_LLM_TIMEOUT,
     ENV_REFLECT_LLM_MAX_CONCURRENT,
     ENV_RETAIN_LLM_MAX_CONCURRENT,
 )
-from ..metrics import get_metrics_collector
-from .response_models import TokenUsage
+
+if TYPE_CHECKING:
+    from .response_models import LLMToolCallResult
 
 # Seed applied to every Groq request for deterministic behavior.
 DEFAULT_LLM_SEED = 4242
@@ -232,6 +226,7 @@ _PROVIDERS_WITHOUT_API_KEY = frozenset(
         "litellm",
         "litellmrouter",
         "bedrock",
+        "nous",
     }
 )
 
@@ -249,6 +244,7 @@ def create_llm_provider(
     reasoning_effort: str,
     groq_service_tier: str | None = None,
     openai_service_tier: str | None = None,
+    bedrock_service_tier: str | None = None,
     extra_body: dict[str, Any] | None = None,
     default_headers: dict[str, str] | None = None,
     vertexai_project_id: str | None = None,
@@ -269,6 +265,7 @@ def create_llm_provider(
         reasoning_effort: Reasoning effort level for supported providers.
         groq_service_tier: Groq service tier (for Groq provider) - "on_demand", "flex", or "auto".
         openai_service_tier: OpenAI service tier (for OpenAI provider) - None (default) or "flex" (50% cheaper).
+        bedrock_service_tier: Bedrock service tier (for Bedrock provider) - None (default), "flex", "priority", or "reserved".
         extra_body: Extra request-body params merged into the provider's native
             call. Threaded into OpenAI-compatible, Fireworks, Anthropic, Gemini/
             VertexAI and LiteLLM providers (each merges them in its own parameter
@@ -284,7 +281,6 @@ def create_llm_provider(
     Returns:
         LLMInterface implementation for the specified provider.
     """
-    from .llm_interface import LLMInterface
     from .providers import (
         AnthropicLLM,
         ClaudeCodeLLM,
@@ -401,6 +397,7 @@ def create_llm_provider(
             model=bedrock_model,
             reasoning_effort=reasoning_effort,
             extra_body=extra_body,
+            bedrock_service_tier=bedrock_service_tier,
         )
 
     elif provider_lower == "llamacpp":
@@ -434,6 +431,21 @@ def create_llm_provider(
             extra_body=extra_body,
         )
 
+    elif provider_lower == "nous":
+        # Nous Portal is OpenAI-compatible on the wire; NousLLM adds rotating
+        # inference:invoke JWT auth read natively from ~/.hermes/auth.json
+        # (no static api_key, no hermes_cli dependency — same shape as Codex).
+        from hindsight_api.engine.providers.nous_llm import NousLLM
+
+        return NousLLM(
+            provider=provider,
+            api_key=api_key,
+            base_url=base_url,
+            model=model,
+            reasoning_effort=reasoning_effort,
+            extra_body=extra_body,
+        )
+
     elif provider_lower in (
         "openai",
         "groq",
@@ -478,6 +490,7 @@ class LLMProvider:
         reasoning_effort: str = "low",
         groq_service_tier: str | None = None,
         openai_service_tier: str | None = None,
+        bedrock_service_tier: str | None = None,
         gemini_safety_settings: list | None = None,
         prompt_cache_enabled: bool = False,
         extra_body: dict[str, Any] | None = None,
@@ -495,6 +508,7 @@ class LLMProvider:
             reasoning_effort: Reasoning effort level for supported providers.
             groq_service_tier: Groq service tier ("on_demand", "flex", "auto") - from config.
             openai_service_tier: OpenAI service tier (None or "flex") - from config.
+            bedrock_service_tier: Bedrock service tier (None, "flex", "priority", "reserved") - from config.
             gemini_safety_settings: Safety settings for Gemini/VertexAI providers.
             extra_body: Extra request-body params merged into the provider's native call
                 (OpenAI-compatible, Fireworks, Anthropic, Gemini/VertexAI, LiteLLM).
@@ -517,6 +531,7 @@ class LLMProvider:
         # Service tiers from hierarchical config (not env vars)
         self.groq_service_tier = groq_service_tier
         self.openai_service_tier = openai_service_tier
+        self.bedrock_service_tier = bedrock_service_tier
         # Gemini safety settings (instance default; can be overridden per-request via context var)
         self.gemini_safety_settings = gemini_safety_settings
         # Gemini prompt caching: when True, retain extraction (and any future
@@ -563,6 +578,7 @@ class LLMProvider:
             "zai",
             "opencode-go",
             "fireworks",
+            "nous",
         ]
         if self.provider not in valid_providers:
             raise ValueError(f"Invalid LLM provider: {self.provider}. Must be one of: {', '.join(valid_providers)}")
@@ -587,6 +603,8 @@ class LLMProvider:
                 self.base_url = "https://api.z.ai/api/coding/paas/v4"
             elif self.provider == "opencode-go":
                 self.base_url = "https://opencode.ai/zen/go/v1"
+            elif self.provider == "nous":
+                self.base_url = "https://inference-api.nousresearch.com/v1"
 
         # Prepare Vertex AI config (if applicable)
         vertexai_project_id = None
@@ -679,6 +697,7 @@ class LLMProvider:
             reasoning_effort=self.reasoning_effort,
             groq_service_tier=self.groq_service_tier,
             openai_service_tier=self.openai_service_tier,
+            bedrock_service_tier=self.bedrock_service_tier,
             extra_body=self.extra_body,
             default_headers=self.default_headers,
             vertexai_project_id=vertexai_project_id,
@@ -1120,6 +1139,7 @@ class LLMProvider:
             DEFAULT_LLM_REASONING_EFFORT,
             ENV_LLM_API_KEY,
             ENV_LLM_BASE_URL,
+            ENV_LLM_BEDROCK_SERVICE_TIER,
             ENV_LLM_DEFAULT_HEADERS,
             ENV_LLM_EXTRA_BODY,
             ENV_LLM_MODEL,
@@ -1151,6 +1171,7 @@ class LLMProvider:
             reasoning_effort=os.getenv(ENV_LLM_REASONING_EFFORT, DEFAULT_LLM_REASONING_EFFORT),
             extra_body=extra_body,
             default_headers=default_headers,
+            bedrock_service_tier=os.getenv(ENV_LLM_BEDROCK_SERVICE_TIER) or None,
         )
 
 

hindsight-api-slim/hindsight_api/engine/providers/anthropic_llm.py

@@ -14,7 +14,7 @@ import logging
 import time
 from typing import Any
 
-from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
+from hindsight_api.engine.llm_interface import LLMInterface
 from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
 from hindsight_api.metrics import get_metrics_collector
 

hindsight-api-slim/hindsight_api/engine/providers/claude_code_llm.py

@@ -15,7 +15,7 @@ from typing import Any
 
 from pydantic import ValidationError
 
-from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
+from hindsight_api.engine.llm_interface import LLMInterface
 from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
 from hindsight_api.metrics import get_metrics_collector
 

hindsight-api-slim/hindsight_api/engine/providers/codex_llm.py

@@ -24,7 +24,7 @@ from typing import Any
 
 import httpx
 
-from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
+from hindsight_api.engine.llm_interface import LLMInterface
 from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
 from hindsight_api.metrics import get_metrics_collector
 
@@ -397,7 +397,6 @@ class CodexLLM(LLMInterface):
         }
 
         url = f"{self.base_url}/codex/responses"
-        last_exception = None
 
         # Manual attempt tracking instead of ``for attempt in range(...)`` so
         # that the reactive-refresh path can retry once without consuming a
@@ -428,7 +427,6 @@ class CodexLLM(LLMInterface):
                         if attempt < max_retries:
                             backoff = min(initial_backoff * (2**attempt), max_backoff)
                             await asyncio.sleep(backoff)
-                            last_exception = e
                             attempt += 1
                             continue
                         raise
@@ -490,7 +488,6 @@ class CodexLLM(LLMInterface):
                 return result
 
             except httpx.HTTPStatusError as e:
-                last_exception = e
                 status_code = e.response.status_code
 
                 # Auth error: try one OAuth refresh + retry before giving up.
@@ -549,7 +546,6 @@ class CodexLLM(LLMInterface):
                     raise
 
             except httpx.RequestError as e:
-                last_exception = e
                 if attempt < max_retries:
                     backoff = min(initial_backoff * (2**attempt), max_backoff)
                     logger.warning(f"Codex connection error (attempt {attempt + 1}/{max_retries + 1}): {e}")
@@ -564,10 +560,6 @@ class CodexLLM(LLMInterface):
                 logger.error(f"Unexpected Codex error: {type(e).__name__}: {e}")
                 raise
 
-        if last_exception:
-            raise last_exception
-        raise RuntimeError("Codex call failed after all retries")
-
     async def _parse_sse_stream(self, response: httpx.Response) -> str:
         """
         Parse Server-Sent Events (SSE) stream from Codex API.

hindsight-api-slim/hindsight_api/engine/providers/gemini_llm.py

@@ -8,9 +8,9 @@ This provider supports both:
 
 import asyncio
 import base64
+import io
 import json
 import logging
-import os
 import time
 from contextvars import ContextVar
 from typing import Any
@@ -19,7 +19,7 @@ from google import genai
 from google.genai import errors as genai_errors
 from google.genai import types as genai_types
 
-from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
+from hindsight_api.engine.llm_interface import LLMInterface
 from hindsight_api.engine.llm_wrapper import parse_llm_json
 from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
 from hindsight_api.metrics import get_metrics_collector
@@ -35,7 +35,6 @@ _safety_settings_ctx: ContextVar[list | None] = ContextVar("gemini_safety_settin
 
 # Vertex AI imports (optional)
 try:
-    import google.auth
     from google.oauth2 import service_account
 
     VERTEXAI_AVAILABLE = True
@@ -43,6 +42,14 @@ except ImportError:
     VERTEXAI_AVAILABLE = False
 
 
+def _to_int(value: Any) -> int:
+    """Coerce Gemini's optional/string completion counts to int, defaulting to 0."""
+    try:
+        return int(value)
+    except (ValueError, TypeError):
+        return 0
+
+
 class GeminiLLM(LLMInterface):
     """
     LLM provider for Google Gemini and Vertex AI.
@@ -826,6 +833,282 @@ class GeminiLLM(LLMInterface):
             tools=tools,
         )
 
+    # ── Batch API (Gemini API only — not Vertex AI) ─────────────────────────
+    #
+    # Google's Gemini Batch API gives a flat 50% discount on input + output
+    # tokens with a 24h completion SLA (https://ai.google.dev/gemini-api/docs/batch-api).
+    # The retain orchestrator and ``fact_extraction`` consumer speak the
+    # OpenAI-batch interface contract, so these overrides translate that shape
+    # to/from Gemini's file-upload → ``batches.create`` → ``batches.get`` →
+    # download flow — nothing downstream changes (same pattern as FireworksLLM).
+    #
+    # Interface contract preserved (see fact_extraction.py result handling)::
+    #     result["response"]["body"]["choices"][0]["message"]["content"]
+
+    async def supports_batch_api(self) -> bool:
+        """True for the Gemini API; False for Vertex AI.
+
+        Only ``provider="gemini"`` is supported: it exposes the file-upload
+        Batch API used below. Vertex AI's batch path is GCS/BigQuery-backed (no
+        file-upload analogue), so it stays unsupported here — the startup
+        validation then surfaces a clear error instead of silently falling back
+        to synchronous, full-price calls.
+        """
+        return self.provider == "gemini"
+
+    async def submit_batch(
+        self,
+        requests: list[dict[str, Any]],
+        endpoint: str = "/v1/chat/completions",
+        completion_window: str = "24h",
+    ) -> dict[str, Any]:
+        """Submit a batch of (OpenAI-shaped) requests to the Gemini Batch API."""
+        if not await self.supports_batch_api():
+            raise NotImplementedError(f"Batch API not supported for provider: {self.provider}")
+
+        # endpoint/completion_window are part of the shared LLMInterface batch
+        # contract (used by the OpenAI path) but have no analogue on Gemini: the
+        # request shape is fixed (generateContent) and the SLA is server-side.
+        # Kept for signature compatibility with the shared retain driver.
+        logger.info(f"Submitting Gemini batch with {len(requests)} requests")
+
+        jsonl = self._translate_requests(requests)
+
+        # Upload the JSONL as a Gemini file (mime_type must be "jsonl"; a
+        # BytesIO has no path for the SDK to infer it from).
+        file_obj = io.BytesIO(jsonl.encode("utf-8"))
+        uploaded = await self._client.aio.files.upload(
+            file=file_obj,
+            config=genai_types.UploadFileConfig(mime_type="jsonl", display_name="hindsight-batch-input"),
+        )
+
+        batch = await self._client.aio.batches.create(
+            model=self.model,
+            src=uploaded.name,
+            config=genai_types.CreateBatchJobConfig(display_name="hindsight-batch"),
+        )
+
+        logger.info(f"Gemini batch submitted: {batch.name}, state={self._state_name(batch.state)}")
+
+        return {
+            "batch_id": batch.name,
+            "status": self._normalize_state(batch.state),
+            "input_file_id": uploaded.name,
+            "request_count": len(requests),
+        }
+
+    async def get_batch_status(self, batch_id: str) -> dict[str, Any]:
+        """Get the status of a Gemini batch job, in the shared status shape."""
+        if not await self.supports_batch_api():
+            raise NotImplementedError(f"Batch API not supported for provider: {self.provider}")
+
+        batch = await self._client.aio.batches.get(name=batch_id)
+
+        stats = batch.completion_stats
+        successful = _to_int(getattr(stats, "successful_count", None)) if stats else 0
+        failed = _to_int(getattr(stats, "failed_count", None)) if stats else 0
+        incomplete = _to_int(getattr(stats, "incomplete_count", None)) if stats else 0
+
+        result: dict[str, Any] = {
+            "batch_id": batch.name,
+            "status": self._normalize_state(batch.state),
+            "request_counts": {
+                "total": successful + failed + incomplete,
+                "completed": successful,
+                "failed": failed,
+            },
+        }
+
+        if batch.dest and getattr(batch.dest, "file_name", None):
+            result["output_file_id"] = batch.dest.file_name
+        if batch.error:
+            result["errors"] = self._error_to_dict(batch.error)
+
+        return result
+
+    async def retrieve_batch_results(self, batch_id: str) -> list[dict[str, Any]]:
+        """Download and normalize completed Gemini batch results."""
+        if not await self.supports_batch_api():
+            raise NotImplementedError(f"Batch API not supported for provider: {self.provider}")
+
+        batch = await self._client.aio.batches.get(name=batch_id)
+
+        status = self._normalize_state(batch.state)
+        if status != "completed":
+            raise ValueError(f"Gemini batch {batch_id} is not completed yet (state: {self._state_name(batch.state)})")
+
+        dest = batch.dest
+        if not dest or not getattr(dest, "file_name", None):
+            raise ValueError(
+                f"Gemini batch {batch_id} completed but reported no output file "
+                f"(submit_batch always uses file mode, so this is unexpected)"
+            )
+
+        content = await self._client.aio.files.download(file=dest.file_name)
+        text = content.decode("utf-8") if isinstance(content, (bytes, bytearray)) else str(content)
+
+        # The output is a JSONL error file plus results merged into one stream;
+        # error lines carry an `error` so partial failures surface per key
+        # instead of vanishing (JOB_STATE_PARTIALLY_SUCCEEDED maps to completed).
+        results: list[dict[str, Any]] = []
+        for line in text.strip().split("\n"):
+            if line.strip():
+                results.append(self._normalize_output_line(json.loads(line)))
+
+        logger.info(f"Retrieved {len(results)} results for Gemini batch {batch_id}")
+        return results
+
+    # ----- pure translation/normalization helpers (unit-tested) ----------
+
+    @staticmethod
+    def _translate_requests(requests: list[dict[str, Any]]) -> str:
+        """OpenAI batch requests -> Gemini batch input JSONL.
+
+        Each output line is ``{"key": <custom_id>, "request": <GenerateContentRequest>}``;
+        the model is supplied to ``batches.create`` so it is omitted per-line.
+        """
+        lines = []
+        for req in requests:
+            gemini_request = GeminiLLM._openai_body_to_gemini_request(req.get("body") or {})
+            lines.append(json.dumps({"key": req.get("custom_id"), "request": gemini_request}, ensure_ascii=False))
+        return "\n".join(lines)
+
+    @staticmethod
+    def _openai_body_to_gemini_request(body: dict[str, Any]) -> dict[str, Any]:
+        """OpenAI chat-completions body -> Gemini ``GenerateContentRequest`` JSON.
+
+        Mirrors the synchronous ``call`` path: system messages become
+        ``systemInstruction``; a ``response_format`` json_schema forces JSON
+        output (``responseMimeType``), appends the schema as a textual hint, and
+        grammar-enforces via ``responseJsonSchema`` when ``strict`` is set.
+        """
+        system_texts: list[str] = []
+        contents: list[dict[str, Any]] = []
+        for msg in body.get("messages") or []:
+            role = msg.get("role", "user")
+            text = msg.get("content", "") or ""
+            if role == "system":
+                system_texts.append(text)
+            elif role == "assistant":
+                contents.append({"role": "model", "parts": [{"text": text}]})
+            else:
+                contents.append({"role": "user", "parts": [{"text": text}]})
+
+        generation_config: dict[str, Any] = {}
+        if body.get("temperature") is not None:
+            generation_config["temperature"] = body["temperature"]
+        if body.get("max_completion_tokens") is not None:
+            generation_config["maxOutputTokens"] = body["max_completion_tokens"]
+
+        response_format = body.get("response_format")
+        if isinstance(response_format, dict) and response_format.get("type") == "json_schema":
+            json_schema = response_format.get("json_schema") or {}
+            schema = json_schema.get("schema")
+            generation_config["responseMimeType"] = "application/json"
+            if schema:
+                system_texts.append(
+                    "You must respond with valid JSON matching this schema:\n" + json.dumps(schema, ensure_ascii=False)
+                )
+                if json_schema.get("strict"):
+                    generation_config["responseJsonSchema"] = schema
+
+        request: dict[str, Any] = {"contents": contents}
+        if system_texts:
+            request["systemInstruction"] = {"parts": [{"text": "\n\n".join(system_texts)}]}
+        if generation_config:
+            request["generationConfig"] = generation_config
+        return request
+
+    @staticmethod
+    def _normalize_output_line(line: dict[str, Any]) -> dict[str, Any]:
+        """Gemini batch output line -> OpenAI-batch-output shape.
+
+        Target: ``{"custom_id", "response": {"body": {"choices": [...], "usage": {...}}}, "error"}``
+        so the consumer's ``result["response"]["body"]["choices"][0]...`` works and
+        it can read ``body["usage"]`` for token accounting (the consumer reports
+        zero usage otherwise).
+        """
+        custom_id = line.get("key") if line.get("key") is not None else line.get("custom_id")
+        error = line.get("error")
+        if error:
+            return {"custom_id": custom_id, "response": None, "error": error}
+
+        response = line.get("response") or {}
+        body: dict[str, Any] = {"choices": [{"message": {"content": GeminiLLM._extract_text_from_response(response)}}]}
+        usage = GeminiLLM._usage_from_response(response)
+        if usage is not None:
+            body["usage"] = usage
+        return {"custom_id": custom_id, "response": {"body": body}, "error": None}
+
+    @staticmethod
+    def _extract_text_from_response(response: dict[str, Any]) -> str:
+        """Concatenate the text parts of a (JSON) GenerateContentResponse."""
+        candidates = response.get("candidates") or []
+        if not candidates:
+            return ""
+        content = candidates[0].get("content") or {}
+        parts = content.get("parts") or []
+        return "".join(p.get("text", "") for p in parts if isinstance(p, dict) and p.get("text"))
+
+    @staticmethod
+    def _usage_from_response(response: dict[str, Any]) -> dict[str, Any] | None:
+        """Gemini ``usageMetadata`` -> OpenAI-shaped ``usage`` block, or None.
+
+        The batch consumer accumulates token usage from ``body["usage"]`` using
+        OpenAI key names, so translate here to keep the output contract uniform
+        across providers. Handles both the REST camelCase (downloaded JSONL) and
+        snake_case spellings defensively.
+        """
+        meta = response.get("usageMetadata") or response.get("usage_metadata")
+        if not isinstance(meta, dict):
+            return None
+        prompt = meta.get("promptTokenCount") or meta.get("prompt_token_count") or 0
+        completion = meta.get("candidatesTokenCount") or meta.get("candidates_token_count") or 0
+        total = meta.get("totalTokenCount") or meta.get("total_token_count") or 0
+        return {"prompt_tokens": prompt, "completion_tokens": completion, "total_tokens": total}
+
+    @staticmethod
+    def _normalize_state(state: Any) -> str:
+        """Gemini ``JobState`` -> the retain driver's status strings.
+
+        Unknown / in-flight states map to ``in_progress`` so the driver keeps
+        polling; ``PARTIALLY_SUCCEEDED`` maps to ``completed`` (per-line errors
+        surface the partial failures during retrieval).
+        """
+        name = GeminiLLM._state_name(state).upper()
+        if name in ("JOB_STATE_SUCCEEDED", "JOB_STATE_PARTIALLY_SUCCEEDED"):
+            return "completed"
+        if name == "JOB_STATE_FAILED":
+            return "failed"
+        if name in ("JOB_STATE_CANCELLED", "JOB_STATE_CANCELLING"):
+            return "cancelled"
+        if name == "JOB_STATE_EXPIRED":
+            return "expired"
+        return "in_progress"
+
+    @staticmethod
+    def _state_name(state: Any) -> str:
+        """Extract the bare ``JOB_STATE_*`` name from a JobState enum or string."""
+        if state is None:
+            return ""
+        name = getattr(state, "name", None)
+        if name:
+            return str(name)
+        text = str(state)
+        if "." in text:
+            text = text.rsplit(".", 1)[-1]
+        return text
+
+    @staticmethod
+    def _error_to_dict(error: Any) -> dict[str, Any]:
+        """Coerce a Gemini JobError into a JSON-serializable dict for logging."""
+        if hasattr(error, "model_dump"):
+            try:
+                return error.model_dump(exclude_none=True)
+            except Exception:
+                pass
+        return {"message": str(error)}
+
     async def cleanup(self) -> None:
         """Clean up resources (close connections, etc.)."""
         # Gemini client doesn't require explicit cleanup

hindsight-api-slim/hindsight_api/engine/providers/litellm_llm.py

@@ -49,6 +49,7 @@ class LiteLLMLLM(LLMInterface):
         reasoning_effort: str = "low",
         timeout: float = 300.0,
         extra_body: dict[str, Any] | None = None,
+        bedrock_service_tier: str | None = None,
         **kwargs: Any,
     ):
         super().__init__(provider, api_key, base_url, model, reasoning_effort, **kwargs)
@@ -60,6 +61,7 @@ class LiteLLMLLM(LLMInterface):
         # drops any the target model rejects (litellm.drop_params=True below).
         # Sourced from llm_extra_body (env: HINDSIGHT_API_LLM_EXTRA_BODY).
         self._extra_body: dict[str, Any] = extra_body or {}
+        self.bedrock_service_tier = bedrock_service_tier
 
         try:
             import litellm
@@ -119,6 +121,10 @@ class LiteLLMLLM(LLMInterface):
         for key, value in self._extra_body.items():
             kwargs.setdefault(key, value)
 
+        # Bedrock service tier: flex (50% cheaper), priority, or reserved
+        if self.model.startswith("bedrock/") and self.bedrock_service_tier is not None:
+            kwargs["service_tier"] = self.bedrock_service_tier
+
         return kwargs
 
     # ── per-model output-tokens cap (shared with Router subclass) ────────────

hindsight-api-slim/hindsight_api/engine/providers/nous_auth.py

@@ -0,0 +1,463 @@
+"""
+Native Nous Portal OAuth authentication manager.
+
+The Nous Portal inference endpoint (https://inference-api.nousresearch.com/v1)
+speaks the OpenAI-compatible wire format but authenticates with a short-lived,
+inference-scoped JWT rather than a static API key. Hermes obtains that JWT once
+via an interactive browser login (``hermes portal``) and persists the resulting
+OAuth state — ``access_token`` + ``refresh_token`` — under ``providers.nous`` in
+``~/.hermes/auth.json``.
+
+This manager reads that file *directly* and refreshes the access token itself,
+exactly mirroring ``codex_auth.py`` (read ``~/.codex/auth.json`` + native
+refresh). It deliberately does **not** import the Hermes ``hermes_cli`` package:
+that package is the interactive CLI, not a library Hindsight can depend on. The
+refresh request shape is mirrored from Hermes' own resolver
+(``POST {portal}/api/oauth/token`` with an ``x-nous-refresh-token`` header and a
+``grant_type=refresh_token`` form body), so server-side changes affect both
+clients identically. The inference bearer is the access token itself — in
+Hermes' state the ``agent_key`` field is literally ``= access_token``.
+
+Single-use refresh tokens
+-------------------------
+Nous refresh tokens are single-use with server-side reuse-detection: if two
+processes refresh with the same ``refresh_token``, or a rotated token is not
+persisted back, the Portal revokes the whole session as a theft signal. Because
+Hindsight shares ``~/.hermes/auth.json`` with a possibly-running Hermes agent,
+every refresh here is performed while holding the **same cross-process advisory
+lock Hermes uses** (``~/.hermes/auth.lock`` via ``fcntl.flock``) and re-reads the
+latest ``refresh_token`` from disk under that lock before exchanging it. That is
+the protocol Hermes follows too, so the two coordinate safely through the file.
+
+Usage
+-----
+    mgr = NousAuthManager.from_file()
+    token = mgr.ensure_fresh_token()        # proactive; refreshes if near expiry
+    ...                                       # use token as Bearer
+    mgr.refresh_tokens(force=True)           # reactive, on a 401
+"""
+
+from __future__ import annotations
+
+import base64
+import binascii
+import contextlib
+import json
+import logging
+import os
+import tempfile
+import threading
+import time
+from collections.abc import Iterator
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+try:
+    import fcntl
+except ImportError:  # pragma: no cover - Windows
+    fcntl = None  # type: ignore[assignment]
+
+logger = logging.getLogger(__name__)
+
+
+# ---------------------------------------------------------------------------
+# Constants — mirrored from Hermes' canonical Nous resolver
+# (hermes_cli/auth.py: DEFAULT_NOUS_* and _refresh_access_token). Endpoints and
+# client id are overridable via the same env vars Hermes honours, so a staging
+# Portal or a future change can be pointed at without a code change.
+# ---------------------------------------------------------------------------
+
+_NOUS_PORTAL_BASE_URL = (
+    os.environ.get("HERMES_PORTAL_BASE_URL")
+    or os.environ.get("NOUS_PORTAL_BASE_URL")
+    or "https://portal.nousresearch.com"
+)
+_NOUS_INFERENCE_BASE_URL = os.environ.get("NOUS_INFERENCE_BASE_URL") or "https://inference-api.nousresearch.com/v1"
+_NOUS_CLIENT_ID = "hermes-cli"
+
+# Proactively refresh this many seconds before the JWT ``exp`` claim — matches
+# the 120s skew Hermes' own runtime resolver uses for Nous.
+_NOUS_TOKEN_REFRESH_SKEW_SECONDS = 120
+
+# OAuth error codes the Portal returns when the refresh_token itself is no
+# longer usable. These are terminal — retrying will not succeed; the user must
+# re-run ``hermes portal``.
+_NOUS_TERMINAL_REFRESH_ERROR_CODES = frozenset(
+    {"invalid_grant", "invalid_token", "refresh_token_reused", "refresh_token_expired"}
+)
+
+_AUTH_LOCK_TIMEOUT_SECONDS = 20.0
+
+
+def _default_auth_file() -> Path:
+    return Path.home() / ".hermes" / "auth.json"
+
+
+class NousNotLoggedInError(RuntimeError):
+    """Raised when ``~/.hermes/auth.json`` has no usable Nous OAuth state.
+
+    Remediation: run ``hermes portal`` to log in to Nous Portal.
+    """
+
+
+class NousRefreshExpiredError(RuntimeError):
+    """Raised when the Nous refresh_token itself is permanently invalid.
+
+    The user must re-run ``hermes portal`` to obtain new credentials. Callers
+    should surface a clear remediation message and stop retrying.
+    """
+
+
+@contextlib.contextmanager
+def _hermes_auth_lock(auth_file: Path, timeout_seconds: float = _AUTH_LOCK_TIMEOUT_SECONDS) -> Iterator[None]:
+    """Cross-process advisory lock on the Hermes auth store.
+
+    Uses ``<auth_file>.lock`` (i.e. ``~/.hermes/auth.lock``) with
+    ``fcntl.flock(LOCK_EX)`` — the exact same lock file and primitive Hermes'
+    ``_auth_store_lock`` takes — so a refresh here is mutually exclusive with a
+    concurrently-running Hermes agent. Degrades to a no-op (with a debug log)
+    where ``fcntl`` is unavailable (Windows); the single-process in-memory lock
+    still serialises this process's own refreshes.
+    """
+    if fcntl is None:  # pragma: no cover - Windows
+        logger.debug("fcntl unavailable; Nous refresh proceeds without a cross-process lock.")
+        yield
+        return
+
+    lock_path = auth_file.with_suffix(".lock")
+    lock_path.parent.mkdir(parents=True, exist_ok=True)
+    with open(lock_path, "a+") as lock_file:
+        deadline = time.monotonic() + max(1.0, timeout_seconds)
+        while True:
+            try:
+                fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
+                break
+            except (BlockingIOError, OSError):
+                if time.monotonic() >= deadline:
+                    raise TimeoutError("Timed out waiting for the Hermes auth store lock") from None
+                time.sleep(0.05)
+        try:
+            yield
+        finally:
+            with contextlib.suppress(OSError):
+                fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN)
+
+
+class NousAuthManager:
+    """Sync Nous Portal OAuth credential manager.
+
+    Holds the access_token + refresh_token in memory and handles
+    proactive/reactive refresh. A ``threading.Lock`` gives single-flight
+    semantics within the process; the cross-process ``fcntl`` lock guards
+    against a concurrent Hermes agent (see module docstring).
+    """
+
+    def __init__(
+        self,
+        access_token: str,
+        refresh_token: str | None,
+        auth_file: Path,
+        *,
+        portal_base_url: str = _NOUS_PORTAL_BASE_URL,
+        inference_base_url: str = _NOUS_INFERENCE_BASE_URL,
+        client_id: str = _NOUS_CLIENT_ID,
+    ) -> None:
+        self.access_token = access_token
+        self.refresh_token = refresh_token
+        self._auth_file = auth_file
+        self._portal_base_url = portal_base_url.rstrip("/")
+        self._inference_base_url = inference_base_url.rstrip("/")
+        self._client_id = client_id
+        self._lock = threading.Lock()
+        self._http_client = httpx.Client(timeout=30.0)
+
+    # ------------------------------------------------------------------
+    # Construction
+    # ------------------------------------------------------------------
+
+    @classmethod
+    def from_file(cls, auth_file: Path | None = None) -> "NousAuthManager":
+        """Build a manager from ``providers.nous`` in the Hermes auth store.
+
+        Raises
+        ------
+        NousNotLoggedInError:
+            If the file is missing, unreadable, or has no Nous OAuth state with
+            an ``access_token``.
+        """
+        if auth_file is None:
+            auth_file = _default_auth_file()
+
+        if not auth_file.exists():
+            raise NousNotLoggedInError(
+                f"Hermes auth file not found: {auth_file}. Run 'hermes portal' to log in to Nous Portal."
+            )
+
+        try:
+            with open(auth_file) as f:
+                data = json.load(f)
+        except (OSError, json.JSONDecodeError) as e:
+            raise NousNotLoggedInError(f"Could not read Hermes auth file {auth_file}: {type(e).__name__}") from e
+
+        state = cls._nous_state(data)
+        if not state:
+            raise NousNotLoggedInError(
+                "Hermes is not logged into Nous Portal (no providers.nous OAuth state). Run 'hermes portal'."
+            )
+
+        access_token = state.get("access_token")
+        if not isinstance(access_token, str) or not access_token:
+            raise NousNotLoggedInError("Nous OAuth state has no access_token. Re-authenticate with 'hermes portal'.")
+
+        return cls(
+            access_token=access_token,
+            refresh_token=state.get("refresh_token"),
+            auth_file=auth_file,
+            portal_base_url=cls._optional_url(state.get("portal_base_url")) or _NOUS_PORTAL_BASE_URL,
+            inference_base_url=cls._optional_url(state.get("inference_base_url")) or _NOUS_INFERENCE_BASE_URL,
+            client_id=str(state.get("client_id") or _NOUS_CLIENT_ID),
+        )
+
+    @staticmethod
+    def _nous_state(data: dict[str, Any]) -> dict[str, Any]:
+        """Pull the ``providers.nous`` state dict out of a loaded auth store."""
+        providers = data.get("providers")
+        if not isinstance(providers, dict):
+            return {}
+        state = providers.get("nous")
+        return state if isinstance(state, dict) else {}
+
+    @staticmethod
+    def _optional_url(value: Any) -> str | None:
+        return value.rstrip("/") if isinstance(value, str) and value.strip() else None
+
+    @property
+    def base_url(self) -> str:
+        return self._inference_base_url
+
+    # ------------------------------------------------------------------
+    # Token state
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def load_refresh_token_from_file(auth_file: Path) -> str | None:
+        """Read ``providers.nous.refresh_token`` from ``auth_file``.
+
+        Returns ``None`` when the file is unreadable or omits the field. Does
+        not raise — the caller degrades to using the in-memory token.
+        """
+        try:
+            with open(auth_file) as f:
+                data = json.load(f)
+        except (OSError, json.JSONDecodeError):
+            return None
+        return NousAuthManager._nous_state(data).get("refresh_token")
+
+    @staticmethod
+    def _decode_jwt_exp_unixtime(token: str) -> int | None:
+        """Return the JWT ``exp`` claim as a unix timestamp, or None on failure.
+
+        The signature is not verified — the server is the source of truth on
+        acceptance. This only schedules proactive refresh.
+        """
+        try:
+            parts = token.split(".")
+            if len(parts) < 2:
+                return None
+            payload_b64 = parts[1]
+            padding = "=" * (-len(payload_b64) % 4)
+            payload = json.loads(base64.urlsafe_b64decode(payload_b64 + padding).decode("utf-8"))
+            exp = payload.get("exp")
+            return int(exp) if exp is not None else None
+        except (ValueError, TypeError, json.JSONDecodeError, binascii.Error):
+            return None
+
+    def _token_is_stale(self, skew_seconds: int = _NOUS_TOKEN_REFRESH_SKEW_SECONDS) -> bool:
+        """True when the cached access_token is past expiry (with skew).
+
+        Returns False when expiry cannot be determined — we'd rather use a
+        possibly-expired token and recover via the reactive 401 path than
+        refresh aggressively on every request when ``exp`` is unparseable.
+        """
+        exp = self._decode_jwt_exp_unixtime(self.access_token)
+        if exp is None:
+            return False
+        return exp <= int(time.time()) + skew_seconds
+
+    # ------------------------------------------------------------------
+    # Persistence
+    # ------------------------------------------------------------------
+
+    def _persist_state_atomic(self, updated: dict[str, Any]) -> None:
+        """Patch ``providers.nous`` in ``_auth_file`` and write atomically.
+
+        Re-reads the on-disk store first so fields written by Hermes (other
+        providers, the credential pool, rotated tokens) are never clobbered,
+        then patches only the Nous OAuth fields and ``os.replace``s into place
+        (atomic on POSIX within the same filesystem). Must be called while
+        holding :func:`_hermes_auth_lock`.
+        """
+        try:
+            with open(self._auth_file) as f:
+                loaded = json.load(f)
+            current: dict[str, Any] = loaded if isinstance(loaded, dict) else {}
+        except (OSError, json.JSONDecodeError):
+            current = {}
+
+        providers = current.get("providers")
+        if not isinstance(providers, dict):
+            providers = {}
+            current["providers"] = providers
+        state = providers.get("nous")
+        if not isinstance(state, dict):
+            state = {}
+            providers["nous"] = state
+
+        state.update(updated)
+        # The inference bearer is the access token itself; keep agent_key in
+        # sync so Hermes' own resolver/status sees the rotation too.
+        state["agent_key"] = updated.get("access_token", state.get("access_token"))
+        current["updated_at"] = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+
+        parent = self._auth_file.parent
+        parent.mkdir(parents=True, exist_ok=True)
+        fd, tmp_path = tempfile.mkstemp(prefix=".auth.", suffix=".json.tmp", dir=str(parent))
+        try:
+            with os.fdopen(fd, "w") as f:
+                json.dump(current, f, indent=2)
+                f.flush()
+                os.fsync(f.fileno())
+            with contextlib.suppress(OSError):
+                os.chmod(tmp_path, 0o600)
+            os.replace(tmp_path, self._auth_file)
+        except Exception:
+            with contextlib.suppress(OSError):
+                os.unlink(tmp_path)
+            raise
+
+    # ------------------------------------------------------------------
+    # Refresh
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def _extract_oauth_error_code(response: httpx.Response) -> str | None:
+        """Pull the OAuth error code out of a 4xx refresh response, if present."""
+        try:
+            body = response.json()
+        except (json.JSONDecodeError, ValueError):
+            return None
+        if not isinstance(body, dict):
+            return None
+        err = body.get("error")
+        if isinstance(err, str):
+            return err
+        if isinstance(err, dict) and isinstance(err.get("code"), str):
+            return err["code"]
+        code = body.get("error_code")
+        return code if isinstance(code, str) else None
+
+    def refresh_tokens(self, reason: str = "", *, force: bool = False) -> None:
+        """Single-flight Nous OAuth token refresh.
+
+        Serialised through ``self._lock`` (in-process single-flight) and
+        :func:`_hermes_auth_lock` (cross-process, vs a running Hermes agent).
+        The latest ``refresh_token`` is re-read from disk under the lock before
+        the exchange — single-use tokens make using a stale in-memory RT a
+        session-revoking mistake.
+
+        Raises
+        ------
+        NousRefreshExpiredError:
+            On a terminal refresh error (expired/reused/invalid grant).
+        RuntimeError:
+            For other refresh failures (network, 5xx, missing refresh_token).
+        """
+        token_before_lock = self.access_token
+        with self._lock:
+            if force:
+                if self.access_token != token_before_lock:
+                    return  # another caller already refreshed while we waited
+            elif not self._token_is_stale():
+                return
+
+            with _hermes_auth_lock(self._auth_file):
+                # Re-read the freshest refresh_token persisted by whoever rotated
+                # last (this process or Hermes). Using a stale RT is exactly what
+                # trips the Portal's single-use reuse-detection.
+                disk_rt = self.load_refresh_token_from_file(self._auth_file)
+                if disk_rt:
+                    self.refresh_token = disk_rt
+
+                if not self.refresh_token:
+                    raise RuntimeError(
+                        "Nous access_token is expired but no refresh_token is available. "
+                        "Run 'hermes portal' to re-authenticate."
+                    )
+
+                log_reason = f" ({reason})" if reason else ""
+                logger.info(f"Refreshing Nous Portal access_token{log_reason}")
+
+                try:
+                    response = self._http_client.post(
+                        f"{self._portal_base_url}/api/oauth/token",
+                        headers={"x-nous-refresh-token": self.refresh_token},
+                        data={"grant_type": "refresh_token", "client_id": self._client_id},
+                        timeout=30.0,
+                    )
+                except httpx.RequestError as e:
+                    raise RuntimeError(f"Nous OAuth refresh network error: {type(e).__name__}") from e
+
+                if response.status_code != 200:
+                    code = self._extract_oauth_error_code(response)
+                    if code in _NOUS_TERMINAL_REFRESH_ERROR_CODES or response.status_code in (400, 401):
+                        raise NousRefreshExpiredError(
+                            f"Nous refresh_token is no longer valid (status={response.status_code}, "
+                            f"error={code or 'none'}). Run 'hermes portal' to re-authenticate."
+                        )
+                    raise RuntimeError(f"Nous OAuth refresh failed with HTTP {response.status_code}")
+
+                try:
+                    body = response.json()
+                except (json.JSONDecodeError, ValueError) as e:
+                    raise RuntimeError(f"Nous OAuth refresh returned non-JSON body: {e}") from e
+
+                new_access = body.get("access_token")
+                if not new_access:
+                    raise RuntimeError("Nous OAuth refresh returned no access_token")
+                new_refresh = body.get("refresh_token") or self.refresh_token
+
+                # Update in-memory state first so waiters see fresh credentials
+                # even if the disk write fails.
+                self.access_token = new_access
+                self.refresh_token = new_refresh
+
+                persisted: dict[str, Any] = {"access_token": new_access, "refresh_token": new_refresh}
+                expires_in = body.get("expires_in")
+                if isinstance(expires_in, (int, float)):
+                    persisted["expires_at"] = datetime.fromtimestamp(
+                        time.time() + float(expires_in), tz=timezone.utc
+                    ).isoformat()
+                try:
+                    self._persist_state_atomic(persisted)
+                except OSError as e:
+                    logger.warning(
+                        f"Nous refresh succeeded but persisting auth.json failed: {type(e).__name__}. "
+                        "In-memory credentials are current; the on-disk rotated token was not saved."
+                    )
+                logger.info("Nous Portal access_token refreshed successfully")
+
+    def ensure_fresh_token(self) -> str:
+        """Refresh proactively if near/at expiry, then return the bearer token.
+
+        Cheap when fresh (a JWT exp decode + comparison).
+        """
+        if self._token_is_stale():
+            self.refresh_tokens(reason="proactive (token near expiry)")
+        return self.access_token
+
+    def close(self) -> None:
+        """Close the underlying HTTP client."""
+        self._http_client.close()

hindsight-api-slim/hindsight_api/engine/providers/nous_llm.py

@@ -0,0 +1,167 @@
+"""
+Nous Portal LLM provider for Hindsight.
+
+Thin wrapper over :class:`OpenAICompatibleLLM`. The Nous Portal speaks the
+OpenAI chat-completions wire format, so all request/response handling is
+inherited unchanged. The only thing Nous needs on top is a rotating,
+inference-scoped JWT (there is no static API key in the Hermes login flow),
+which :class:`NousAuthManager` reads from ``~/.hermes/auth.json`` and refreshes
+natively — the same pattern as the Codex provider, with no dependency on the
+``hermes_cli`` package. See ``nous_auth.py`` for the auth mechanics.
+
+Configure with::
+
+    llm_provider = "nous"
+    llm_base_url = "https://inference-api.nousresearch.com/v1"   # or omit
+    llm_model    = "deepseek/deepseek-v4-flash"                  # any Nous slug
+
+No API key is set in config; the token comes from the shared Hermes auth store
+after a one-time ``hermes portal`` login.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any
+
+from openai import APIStatusError, AsyncOpenAI
+
+from hindsight_api.engine.providers.nous_auth import (
+    NousAuthManager,
+    NousNotLoggedInError,
+    NousRefreshExpiredError,
+)
+from hindsight_api.engine.providers.openai_compatible_llm import OpenAICompatibleLLM
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["NousLLM", "NousAuthManager", "NousNotLoggedInError", "NousRefreshExpiredError"]
+
+
+class NousLLM(OpenAICompatibleLLM):
+    """OpenAI-compatible provider for the Nous Portal with rotating-JWT auth."""
+
+    def __init__(
+        self,
+        provider: str,
+        api_key: str,  # Ignored — the token is read from ~/.hermes/auth.json
+        base_url: str,
+        model: str,
+        reasoning_effort: str = "low",
+        **kwargs: Any,
+    ):
+        try:
+            self._auth = NousAuthManager.from_file()
+        except NousNotLoggedInError as e:
+            raise RuntimeError(
+                f"Failed to load Nous Portal credentials: {e}\n\n"
+                "To set up Nous authentication:\n"
+                "1. Install Hermes: https://hermes-agent.nousresearch.com\n"
+                "2. Log in to Nous Portal: hermes portal\n"
+                "3. Verify: hermes portal status\n\n"
+                "Or use a different provider (openai, anthropic, gemini) with an API key."
+            ) from e
+
+        # Single-flight async refresh lock — concurrent coroutines racing toward
+        # an expired token produce one network refresh.
+        self._auth_lock = asyncio.Lock()
+
+        token = self._auth.access_token
+        resolved_base = base_url or self._auth.base_url
+        # Parent validates provider against a fixed list; present as "openai"
+        # (identical wire format) while retaining the true identity for logs.
+        super().__init__(
+            provider="openai",
+            api_key=token,
+            base_url=resolved_base,
+            model=model,
+            reasoning_effort=reasoning_effort,
+            **kwargs,
+        )
+        self._nous_provider_name = provider
+        logger.info(
+            "Nous LLM initialized: model=%s base_url=%s (rotating inference:invoke JWT)",
+            self.model,
+            self.base_url,
+        )
+
+    # ------------------------------------------------------------------
+    # Token lifecycle
+    # ------------------------------------------------------------------
+
+    def _rebuild_client(self) -> None:
+        """Rebuild the OpenAI SDK client against the current token."""
+        self.api_key = self._auth.access_token
+        self._client = AsyncOpenAI(
+            api_key=self.api_key,
+            base_url=self.base_url,
+            max_retries=0,
+            timeout=self.timeout,
+        )
+
+    async def _ensure_fresh_token(self) -> None:
+        """Proactively refresh if the JWT is near expiry; rebuild on change.
+
+        Cheap when fresh (a JWT exp decode). The blocking refresh (network +
+        cross-process file lock) is offloaded to a thread so the event loop is
+        never stalled.
+        """
+        if not self._auth._token_is_stale():
+            return
+        await self._refresh(reason="proactive (token near expiry)", force=False)
+
+    async def _refresh(self, *, reason: str, force: bool) -> None:
+        token_before = self.api_key
+        async with self._auth_lock:
+            if force:
+                if self.api_key != token_before:
+                    return  # another coroutine already refreshed
+            elif not self._auth._token_is_stale():
+                return
+            await asyncio.to_thread(lambda: self._auth.refresh_tokens(reason, force=force))
+            if self._auth.access_token != self.api_key:
+                self._rebuild_client()
+
+    async def _with_auth_retry(self, fn: Any, label: str, *args: Any, **kwargs: Any) -> Any:
+        """Run an OpenAI-compatible call, refreshing once on a 401.
+
+        The proactive refresh covers most expiries; a token can still be
+        rejected mid-flight if Hermes rotated it out from under us or the exp
+        claim was unparseable. One reactive refresh + retry is the safety net.
+        """
+        await self._ensure_fresh_token()
+        try:
+            return await fn(*args, **kwargs)
+        except APIStatusError as e:
+            if getattr(e, "status_code", None) != 401:
+                raise
+            logger.warning("Nous 401 (%s) — forcing token refresh and retrying once.", label)
+            try:
+                await self._refresh(reason=f"reactive (HTTP 401 on {label})", force=True)
+            except NousRefreshExpiredError as refresh_err:
+                raise RuntimeError(
+                    "Nous authentication failed and the refresh_token is no longer valid.\n"
+                    "Run 'hermes portal' to re-authenticate."
+                ) from refresh_err
+            return await fn(*args, **kwargs)
+
+    # ------------------------------------------------------------------
+    # Overrides
+    # ------------------------------------------------------------------
+
+    async def verify_connection(self) -> None:
+        await self._ensure_fresh_token()
+        return await super().verify_connection()
+
+    async def call(self, *args: Any, **kwargs: Any) -> Any:
+        return await self._with_auth_retry(super().call, "call", *args, **kwargs)
+
+    async def call_with_tools(self, *args: Any, **kwargs: Any) -> Any:
+        return await self._with_auth_retry(super().call_with_tools, "call_with_tools", *args, **kwargs)
+
+    async def cleanup(self) -> None:
+        self._auth.close()
+        parent_cleanup = getattr(super(), "cleanup", None)
+        if parent_cleanup is not None:
+            await parent_cleanup()

hindsight-api-slim/hindsight_api/engine/providers/openai_compatible_llm.py

@@ -33,6 +33,7 @@ import httpx
 from openai import APIConnectionError, APIStatusError, AsyncOpenAI, LengthFinishReasonError
 
 from hindsight_api.config import DEFAULT_LLM_TIMEOUT, ENV_LLM_TIMEOUT
+from hindsight_api.engine.bank_attribution import apply_bank_attribution
 from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
 from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
 from hindsight_api.metrics import get_metrics_collector
@@ -595,6 +596,8 @@ class OpenAICompatibleLLM(LLMInterface):
                     call_params["messages"] = _ensure_json_word_in_user_message(call_params["messages"])
                     call_params["response_format"] = {"type": "json_object"}
 
+        apply_bank_attribution(call_params)
+
         last_exception = None
 
         for attempt in range(max_retries + 1):
@@ -945,6 +948,8 @@ class OpenAICompatibleLLM(LLMInterface):
         if extra_body:
             call_params["extra_body"] = extra_body
 
+        apply_bank_attribution(call_params)
+
         last_exception = None
 
         for attempt in range(max_retries + 1):

hindsight-api-slim/hindsight_api/engine/query_analyzer.py

@@ -6,12 +6,17 @@ structured information like temporal constraints.
 """
 
 import logging
-import re
 from abc import ABC, abstractmethod
 from datetime import datetime, timedelta
 
 from pydantic import BaseModel, Field
 
+from hindsight_api.engine.temporal_periods import (
+    NO_TEMPORAL_CONSTRAINT,
+    extract_period,
+    is_embedded_cjk_dateparser_match,
+)
+
 logger = logging.getLogger(__name__)
 
 
@@ -123,9 +128,12 @@ class DateparserQueryAnalyzer(QueryAnalyzer):
 
         # Check for period expressions first (these need special handling)
         query_lower = query.lower()
-        period_result = self._extract_period(query_lower, reference_date)
-        if period_result is not None:
-            return QueryAnalysis(temporal_constraint=period_result)
+        period_result = extract_period(query_lower, reference_date)
+        if period_result is NO_TEMPORAL_CONSTRAINT:
+            return QueryAnalysis(temporal_constraint=None)
+        if isinstance(period_result, tuple):
+            start_date, end_date = period_result
+            return QueryAnalysis(temporal_constraint=TemporalConstraint(start_date=start_date, end_date=end_date))
 
         # Lazy load dateparser (only imports on first call, then cached)
         self.load()
@@ -158,7 +166,12 @@ class DateparserQueryAnalyzer(QueryAnalyzer):
 
         # Filter out false positives (common words parsed as dates)
         false_positives = {"do", "may", "march", "will", "can", "sat", "sun", "mon", "tue", "wed", "thu", "fri"}
-        valid_results = [(text, date) for text, date in results if text.lower() not in false_positives or len(text) > 3]
+        valid_results = [
+            (text, date)
+            for text, date in results
+            if (text.lower() not in false_positives or len(text) > 3)
+            and not is_embedded_cjk_dateparser_match(query, text)
+        ]
 
         if not valid_results:
             return QueryAnalysis(temporal_constraint=None)
@@ -172,127 +185,6 @@ class DateparserQueryAnalyzer(QueryAnalyzer):
 
         return QueryAnalysis(temporal_constraint=TemporalConstraint(start_date=start_date, end_date=end_date))
 
-    def _extract_period(self, query: str, reference_date: datetime) -> TemporalConstraint | None:
-        """
-        Extract period-based temporal expressions (week, month, year, weekend).
-
-        These need special handling as they represent date ranges, not single dates.
-        Supports multiple languages.
-        """
-
-        def constraint(start: datetime, end: datetime) -> TemporalConstraint:
-            return TemporalConstraint(
-                start_date=start.replace(hour=0, minute=0, second=0, microsecond=0),
-                end_date=end.replace(hour=23, minute=59, second=59, microsecond=999999),
-            )
-
-        # Yesterday patterns (English, Spanish, Italian, French, German)
-        if re.search(r"\b(yesterday|ayer|ieri|hier|gestern)\b", query, re.IGNORECASE):
-            d = reference_date - timedelta(days=1)
-            return constraint(d, d)
-
-        # Today patterns
-        if re.search(r"\b(today|hoy|oggi|aujourd\'?hui|heute)\b", query, re.IGNORECASE):
-            return constraint(reference_date, reference_date)
-
-        # "a couple of days ago" / "a few days ago" patterns
-        # These are imprecise so we create a range
-        if re.search(r"\b(a\s+)?couple\s+(of\s+)?days?\s+ago\b", query, re.IGNORECASE):
-            # "a couple of days" = approximately 2 days, give range of 1-3 days
-            return constraint(reference_date - timedelta(days=3), reference_date - timedelta(days=1))
-
-        if re.search(r"\b(a\s+)?few\s+days?\s+ago\b", query, re.IGNORECASE):
-            # "a few days" = approximately 3-4 days, give range of 2-5 days
-            return constraint(reference_date - timedelta(days=5), reference_date - timedelta(days=2))
-
-        # "a couple of weeks ago" / "a few weeks ago" patterns
-        if re.search(r"\b(a\s+)?couple\s+(of\s+)?weeks?\s+ago\b", query, re.IGNORECASE):
-            # "a couple of weeks" = approximately 2 weeks, give range of 1-3 weeks
-            return constraint(reference_date - timedelta(weeks=3), reference_date - timedelta(weeks=1))
-
-        if re.search(r"\b(a\s+)?few\s+weeks?\s+ago\b", query, re.IGNORECASE):
-            # "a few weeks" = approximately 3-4 weeks, give range of 2-5 weeks
-            return constraint(reference_date - timedelta(weeks=5), reference_date - timedelta(weeks=2))
-
-        # "a couple of months ago" / "a few months ago" patterns
-        if re.search(r"\b(a\s+)?couple\s+(of\s+)?months?\s+ago\b", query, re.IGNORECASE):
-            # "a couple of months" = approximately 2 months, give range of 1-3 months
-            return constraint(reference_date - timedelta(days=90), reference_date - timedelta(days=30))
-
-        if re.search(r"\b(a\s+)?few\s+months?\s+ago\b", query, re.IGNORECASE):
-            # "a few months" = approximately 3-4 months, give range of 2-5 months
-            return constraint(reference_date - timedelta(days=150), reference_date - timedelta(days=60))
-
-        # Last week patterns (English, Spanish, Italian, French, German)
-        if re.search(
-            r"\b(last\s+week|la\s+semana\s+pasada|la\s+settimana\s+scorsa|la\s+semaine\s+derni[eè]re|letzte\s+woche)\b",
-            query,
-            re.IGNORECASE,
-        ):
-            start = reference_date - timedelta(days=reference_date.weekday() + 7)
-            return constraint(start, start + timedelta(days=6))
-
-        # Last month patterns
-        if re.search(
-            r"\b(last\s+month|el\s+mes\s+pasado|il\s+mese\s+scorso|le\s+mois\s+dernier|letzten?\s+monat)\b",
-            query,
-            re.IGNORECASE,
-        ):
-            first = reference_date.replace(day=1)
-            end = first - timedelta(days=1)
-            start = end.replace(day=1)
-            return constraint(start, end)
-
-        # Last year patterns
-        if re.search(
-            r"\b(last\s+year|el\s+a[ñn]o\s+pasado|l\'anno\s+scorso|l\'ann[ée]e\s+derni[eè]re|letztes?\s+jahr)\b",
-            query,
-            re.IGNORECASE,
-        ):
-            year = reference_date.year - 1
-            return constraint(datetime(year, 1, 1), datetime(year, 12, 31))
-
-        # Last weekend patterns
-        if re.search(
-            r"\b(last\s+weekend|el\s+fin\s+de\s+semana\s+pasado|lo\s+scorso\s+fine\s+settimana|le\s+week-?end\s+dernier|letztes?\s+wochenende)\b",
-            query,
-            re.IGNORECASE,
-        ):
-            days_since_sat = (reference_date.weekday() + 2) % 7
-            if days_since_sat == 0:
-                days_since_sat = 7
-            sat = reference_date - timedelta(days=days_since_sat)
-            return constraint(sat, sat + timedelta(days=1))
-
-        # Month + Year patterns (e.g., "June 2024", "junio 2024", "giugno 2024")
-        month_patterns = {
-            "january|enero|gennaio|janvier|januar": 1,
-            "february|febrero|febbraio|f[ée]vrier|februar": 2,
-            "march|marzo|mars|m[äa]rz": 3,
-            "april|abril|aprile|avril": 4,
-            "may|mayo|maggio|mai": 5,
-            "june|junio|giugno|juin|juni": 6,
-            "july|julio|luglio|juillet|juli": 7,
-            "august|agosto|ao[uû]t": 8,
-            "september|septiembre|settembre|septembre": 9,
-            "october|octubre|ottobre|octobre|oktober": 10,
-            "november|noviembre|novembre": 11,
-            "december|diciembre|dicembre|d[ée]cembre|dezember": 12,
-        }
-
-        for pattern, month_num in month_patterns.items():
-            match = re.search(rf"\b({pattern})\s+(\d{{4}})\b", query, re.IGNORECASE)
-            if match:
-                year = int(match.group(2))
-                start = datetime(year, month_num, 1)
-                if month_num == 12:
-                    end = datetime(year, 12, 31)
-                else:
-                    end = datetime(year, month_num + 1, 1) - timedelta(days=1)
-                return constraint(start, end)
-
-        return None
-
 
 class TransformerQueryAnalyzer(QueryAnalyzer):
     """

hindsight-api-slim/hindsight_api/engine/reflect/agent.py

@@ -14,6 +14,7 @@ import re
 import time
 from typing import TYPE_CHECKING, Any, Awaitable, Callable
 
+from ...config import get_config
 from .models import DirectiveInfo, LLMCall, ReflectAgentResult, TokenUsageSummary, ToolCall
 from .prompts import (
     _extract_directive_rules,
@@ -340,6 +341,7 @@ async def run_reflect_agent(
     budget: str | None = None,
     max_context_tokens: int = 100_000,
     llm_output_language: str | None = None,
+    cancel_check: Callable[[], None] | None = None,
 ) -> ReflectAgentResult:
     """
     Execute the reflect agent loop using native tool calling.
@@ -376,12 +378,16 @@ async def run_reflect_agent(
     # Extract directive rules for tool schema (if any)
     directive_rules = _extract_directive_rules(directives) if directives else None
 
-    # Get tools for this agent (with directive compliance field if directives exist)
+    # Get tools for this agent (with directive compliance field if directives exist).
+    # The expand tool only reads back raw source text (chunks/documents), so it is
+    # useless and excluded when document text storage is disabled.
+    include_expand = get_config().store_document_text
     tools = get_reflect_tools(
         directive_rules=directive_rules,
         include_mental_models=has_mental_models,
         include_observations=include_observations,
         include_recall=include_recall,
+        include_expand=include_expand,
     )
     # Build set of enabled tool names to guard against LLM hallucinating disabled tool calls
     enabled_tools: frozenset[str] = frozenset(t["function"]["name"] for t in tools if t.get("type") == "function")
@@ -488,6 +494,13 @@ async def run_reflect_agent(
     # under ``auto`` tool choice. None means the full forced path still applies.
     stop_forcing_from_iteration: int | None = None
     for iteration in range(max_iterations):
+        # Cooperative cancellation checkpoint: abort the agent loop between
+        # iterations if the caller (e.g. an HTTP client) has gone away, rather
+        # than spending another LLM round-trip on a result nobody will read
+        # (issue #2122). Raises OperationCancelledError when fired.
+        if cancel_check is not None:
+            cancel_check()
+
         is_last = iteration == max_iterations - 1
 
         if is_last:
@@ -500,7 +513,9 @@ async def run_reflect_agent(
                 messages=[
                     {
                         "role": "system",
-                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                        "content": build_final_system_prompt(
+                            bank_profile.get("mission"), llm_output_language, directives
+                        ),
                     },
                     {"role": "user", "content": prompt},
                 ],
@@ -560,7 +575,9 @@ async def run_reflect_agent(
                 messages=[
                     {
                         "role": "system",
-                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                        "content": build_final_system_prompt(
+                            bank_profile.get("mission"), llm_output_language, directives
+                        ),
                     },
                     {"role": "user", "content": prompt},
                 ],
@@ -679,7 +696,9 @@ async def run_reflect_agent(
                 messages=[
                     {
                         "role": "system",
-                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                        "content": build_final_system_prompt(
+                            bank_profile.get("mission"), llm_output_language, directives
+                        ),
                     },
                     {"role": "user", "content": prompt},
                 ],
@@ -803,7 +822,9 @@ async def run_reflect_agent(
                 messages=[
                     {
                         "role": "system",
-                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                        "content": build_final_system_prompt(
+                            bank_profile.get("mission"), llm_output_language, directives
+                        ),
                     },
                     {"role": "user", "content": prompt},
                 ],
@@ -908,7 +929,9 @@ async def run_reflect_agent(
             hallucinated_tools = []
             for tc in other_tools:
                 norm = _normalize_tool_name(tc.name)
-                if enabled_tools is not None and norm not in enabled_tools and norm not in ("done", "expand"):
+                # "done" is always available. "expand" is governed by enabled_tools
+                # (it is excluded when text storage is disabled), so it is not hardcoded here.
+                if enabled_tools is not None and norm not in enabled_tools and norm != "done":
                     hallucinated_tools.append(tc)
                 else:
                     allowed_tools.append(tc)
@@ -1236,8 +1259,10 @@ async def _execute_tool(
     # Normalize tool name for various LLM output formats
     tool_name = _normalize_tool_name(tool_name)
 
-    # Guard against LLMs hallucinating calls to tools that were not provided
-    if enabled_tools is not None and tool_name not in enabled_tools and tool_name not in ("done", "expand"):
+    # Guard against LLMs hallucinating calls to tools that were not provided.
+    # "done" is always available; "expand" is governed by enabled_tools (excluded
+    # when text storage is disabled), so it is not hardcoded as always-allowed here.
+    if enabled_tools is not None and tool_name not in enabled_tools and tool_name != "done":
         return {"error": f"Tool '{tool_name}' is not available. Use only the tools provided to you."}
 
     if tool_name == "search_mental_models":

hindsight-api-slim/hindsight_api/engine/reflect/delta_ops.py

@@ -26,10 +26,13 @@ or stay the same per refresh, never get worse.
 
 from __future__ import annotations
 
+import json
 import logging
 from typing import Annotated, Any, Literal, Union
 
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, ValidationError
+
+from hindsight_api.engine.llm_wrapper import parse_llm_json
 
 from .structured_doc import (
     Block,
@@ -144,6 +147,27 @@ Operation = Annotated[
     Field(discriminator="op"),
 ]
 
+_OPERATION_ADAPTER: TypeAdapter[Operation] = TypeAdapter(Operation)
+
+
+def _validate_operations_list(raw_ops: Any) -> tuple[list[Operation], list[dict[str, Any]]]:
+    """Validate each operation independently; drop invalid ops instead of failing the batch."""
+    if not isinstance(raw_ops, list):
+        raise TypeError(f"operations must be a list, got {type(raw_ops)!r}")
+    valid: list[Operation] = []
+    skipped: list[dict[str, Any]] = []
+    for i, item in enumerate(raw_ops):
+        try:
+            valid.append(_OPERATION_ADAPTER.validate_python(item))
+        except ValidationError as exc:
+            skipped.append({"index": i, "op": item, "error": exc.errors(include_url=False)})
+            logger.warning(
+                "[STRUCTURED_DELTA] skipping invalid operation at index %s: %s",
+                i,
+                exc.errors(include_url=False),
+            )
+    return valid, skipped
+
 
 class DeltaOperationList(BaseModel):
     """Container for the operations produced by an LLM delta call."""
@@ -152,6 +176,104 @@ class DeltaOperationList(BaseModel):
     operations: list[Operation] = Field(default_factory=list)
 
 
+class DeltaAllOpsInvalidError(ValueError):
+    """Raised when the model emitted operations but none survived validation.
+
+    Distinct from an empty ``operations`` array (a legitimate no-op): here every
+    op was malformed, so returning zero valid ops would make the caller apply
+    nothing and silently drop this refresh's new facts. Raising instead lets the
+    caller fall back to a full rewrite, which still integrates the new facts.
+    """
+
+
+def _finalize_operations(valid: list[Operation], skipped: list[dict[str, Any]]) -> DeltaOperationList:
+    """Build the result, but refuse a wholesale validation failure as a silent no-op."""
+    if skipped and not valid:
+        raise DeltaAllOpsInvalidError(f"all {len(skipped)} delta operation(s) failed validation")
+    return DeltaOperationList(operations=valid)
+
+
+def _extract_balanced_json_object(text: str) -> str | None:
+    """Return the first top-level ``{...}`` slice, ignoring trailing junk."""
+    start = text.find("{")
+    if start < 0:
+        return None
+    depth = 0
+    in_string = False
+    escape = False
+    for i in range(start, len(text)):
+        ch = text[i]
+        if in_string:
+            if escape:
+                escape = False
+            elif ch == "\\":
+                escape = True
+            elif ch == '"':
+                in_string = False
+            continue
+        if ch == '"':
+            in_string = True
+        elif ch == "{":
+            depth += 1
+        elif ch == "}":
+            depth -= 1
+            if depth == 0:
+                return text[start : i + 1]
+    return None
+
+
+def parse_delta_operation_list(raw: Any) -> DeltaOperationList:
+    """Parse structured-delta LLM output into a validated operation list."""
+    if isinstance(raw, DeltaOperationList):
+        return raw
+    if isinstance(raw, dict):
+        ops_raw = raw.get("operations", [])
+        valid, skipped = _validate_operations_list(ops_raw)
+        if skipped:
+            logger.info(
+                "[STRUCTURED_DELTA] parsed %s op(s), skipped %s invalid op(s) from dict payload",
+                len(valid),
+                len(skipped),
+            )
+        return _finalize_operations(valid, skipped)
+
+    text = (raw or "").strip()
+    if not text:
+        return DeltaOperationList()
+
+    candidates: list[str] = [text]
+    extracted = _extract_balanced_json_object(text)
+    if extracted and extracted != text:
+        candidates.append(extracted)
+
+    last_error: Exception | None = None
+    for candidate in candidates:
+        try:
+            payload = parse_llm_json(candidate)
+        except json.JSONDecodeError as exc:
+            last_error = exc
+            continue
+        if not isinstance(payload, dict) or "operations" not in payload:
+            last_error = ValueError("delta payload must be an object with an operations array")
+            continue
+        try:
+            valid, skipped = _validate_operations_list(payload["operations"])
+        except TypeError as exc:
+            last_error = exc
+            continue
+        if skipped:
+            logger.info(
+                "[STRUCTURED_DELTA] parsed %s op(s), skipped %s invalid op(s)",
+                len(valid),
+                len(skipped),
+            )
+        return _finalize_operations(valid, skipped)
+
+    if last_error is not None:
+        raise last_error
+    return DeltaOperationList()
+
+
 # Application ---------------------------------------------------------------
 
 

hindsight-api-slim/hindsight_api/engine/reflect/prompts.py

@@ -604,16 +604,44 @@ Just provide the direct answer with proper markdown formatting.
 CRITICAL: This is a NON-CONVERSATIONAL system. NEVER ask follow-up questions, offer to search again, suggest alternatives, or end with anything like "Would you like me to..." or "Let me know if...". The user cannot reply. Your answer must be complete and self-contained."""
 
 
-def build_final_system_prompt(mission: str | None = None, llm_output_language: str | None = None) -> str:
+# The final synthesis is a SEPARATE LLM call with its own system prompt — the
+# agent/reasoning system prompt (which carries directives and the language rule)
+# is NOT in scope here. So this default language rule, and the directives, must
+# be repeated for the answer-writing model. Without it, weaker models drift to
+# English even when the question/facts are in another language or a directive
+# demands a specific one (the cause of flaky multilingual reflect tests).
+_FINAL_LANGUAGE_RULE = (
+    "## LANGUAGE\n"
+    "- Respond in the SAME language as the user's question "
+    "(e.g. a question in Chinese gets a Chinese answer; Japanese → Japanese).\n"
+    "- If a directive above specifies a response language, follow the directive — "
+    "it takes precedence over this default."
+)
+
+
+def build_final_system_prompt(
+    mission: str | None = None,
+    llm_output_language: str | None = None,
+    directives: list[dict[str, Any]] | None = None,
+) -> str:
     """Build the final synthesis system prompt, using mission as role when set.
 
-    When ``llm_output_language`` is set, the response is forced into that
-    language regardless of the query/source language.
+    ``directives`` are re-injected here (they live in the agent/reasoning prompt,
+    but the final answer is a separate call) so output-constraining rules — most
+    visibly response language — are honoured by the model that actually writes
+    the answer. When ``llm_output_language`` is set it forces that language
+    regardless of the query/source/directive language (config override wins).
     """
     from hindsight_api.engine.prompt_utils import escape_for_prompt, output_language_directive
 
     role_section = escape_for_prompt(mission.strip()) if mission else _DEFAULT_FINAL_ROLE
-    return _FINAL_SYSTEM_PROMPT_BASE.format(role_section=role_section) + output_language_directive(llm_output_language)
+
+    parts = [build_directives_section(directives) if directives else ""]
+    parts.append(_FINAL_SYSTEM_PROMPT_BASE.format(role_section=role_section))
+    parts.append(_FINAL_LANGUAGE_RULE)
+    parts.append(build_directives_reminder(directives) if directives else "")
+
+    return "\n\n".join(p.strip() for p in parts if p.strip()) + output_language_directive(llm_output_language)
 
 
 # Backward-compatible constant for non-identity missions
@@ -706,7 +734,65 @@ Examples
   ``{"operations": [{"op": "replace_block", "section_id": "overview",
   "index": 0, "block": {"type": "paragraph", "text": "Updated summary."}}]}``
 - Remove an obsolete block →
-  ``{"operations": [{"op": "remove_block", "section_id": "status", "index": 2}]}``"""
+  ``{"operations": [{"op": "remove_block", "section_id": "status", "index": 2}]}``
+
+JSON STRING RULES (critical)
+- Every ``text`` and ``items`` string must be valid JSON: escape ``"`` as ``\\"``,
+  backslashes as ``\\\\``, and newlines as ``\\n``. Do not use raw backticks inside
+  strings unless needed; prefer plain quotes for file paths.
+- ``replace_block``, ``insert_block``, and ``remove_block`` MUST include ``index`` (0-based block position in that section). Use ``replace_section_blocks`` only when replacing every block in a section.
+
+- Do not append extra ``]`` or ``}`` after the closing ``}`` of the root object."""
+
+_STRUCTURED_DELTA_DEFAULT_MAX_INPUT_TOKENS = 24_000
+
+
+def _truncate_cl100k(text: str, max_tokens: int) -> str:
+    """Truncate text to at most max_tokens using cl100k_base."""
+    if max_tokens <= 0:
+        return ""
+    from .tokenization import count_cl100k_tokens
+
+    if count_cl100k_tokens(text) <= max_tokens:
+        return text
+    enc = __import__("tiktoken").get_encoding("cl100k_base")
+    return enc.decode(enc.encode(text)[:max_tokens])
+
+
+def _fit_structured_delta_prompt_parts(
+    *,
+    source_query: str,
+    current_document_json: str,
+    candidate_markdown: str,
+    facts_block: str,
+    budget_hint: str,
+    task_footer: str,
+    max_input_tokens: int,
+) -> tuple[str, str, str, bool]:
+    """Shrink large prompt sections to fit within max_input_tokens (cl100k estimate)."""
+    from .tokenization import count_cl100k_tokens
+
+    fixed = (
+        f"## Topic\n{source_query}\n\n"
+        f"## CURRENT DOCUMENT (apply ops to this; reference section ids as listed)\n"
+        f"```json\n\n```\n\n"
+        f"## NEW INFORMATION SYNTHESIS (context for how new facts relate to the topic)\n"
+        f"```markdown\n\n```\n\n"
+        f"## SUPPORTING FACTS (new since last refresh — integrate these)\n"
+        f"{budget_hint}\n\n"
+        f"{task_footer}"
+    )
+    facts_header = "## SUPPORTING FACTS (new since last refresh — integrate these)\n"
+    facts_prefix_tokens = count_cl100k_tokens(facts_header)
+    reserved_facts = min(4096, max(512, max_input_tokens // 8))
+    doc_budget = max(1024, (max_input_tokens - count_cl100k_tokens(fixed) - reserved_facts) * 55 // 100)
+    cand_budget = max(512, (max_input_tokens - count_cl100k_tokens(fixed) - reserved_facts) * 30 // 100)
+    facts_budget = max(256, reserved_facts - facts_prefix_tokens)
+    doc_json = _truncate_cl100k(current_document_json, doc_budget)
+    candidate = _truncate_cl100k(candidate_markdown, cand_budget)
+    facts_body = _truncate_cl100k(facts_block, facts_budget)
+    truncated = doc_json != current_document_json or candidate != candidate_markdown or facts_body != facts_block
+    return doc_json, candidate, facts_body, truncated
 
 
 def build_structured_delta_prompt(
@@ -716,6 +802,7 @@ def build_structured_delta_prompt(
     supporting_facts: list[dict[str, Any]],
     source_query: str,
     max_output_tokens: int | None = None,
+    max_input_tokens: int | None = None,
 ) -> str:
     """Build the user prompt for a structured-delta mental model refresh.
 
@@ -746,19 +833,39 @@ def build_structured_delta_prompt(
             "block-level ops) so the response always parses as valid JSON."
         )
 
-    return (
-        f"## Topic\n{source_query}\n\n"
-        f"## CURRENT DOCUMENT (apply ops to this; reference section ids as listed)\n"
-        f"```json\n{current_document_json}\n```\n\n"
-        f"## NEW INFORMATION SYNTHESIS (context for how new facts relate to the topic)\n"
-        f"```markdown\n{candidate_markdown}\n```\n\n"
-        f"## SUPPORTING FACTS (new since last refresh — integrate these)\n{facts_block}"
-        f"{budget_hint}\n\n"
+    task_footer = (
         "## Task\n"
         "Output a JSON object matching the operations schema. Integrate the new "
         "supporting facts into CURRENT DOCUMENT. Add, update, or remove content "
         "as needed. Preserve unchanged sections and blocks by not mentioning them."
     )
+    input_cap = max_input_tokens if max_input_tokens is not None else _STRUCTURED_DELTA_DEFAULT_MAX_INPUT_TOKENS
+    doc_json, candidate, facts_body, input_truncated = _fit_structured_delta_prompt_parts(
+        source_query=source_query,
+        current_document_json=current_document_json,
+        candidate_markdown=candidate_markdown,
+        facts_block=facts_block,
+        budget_hint=budget_hint,
+        task_footer=task_footer,
+        max_input_tokens=input_cap,
+    )
+    truncation_note = ""
+    if input_truncated:
+        truncation_note = (
+            "\n\n*Note: Document, synthesis, or facts were truncated to fit the model "
+            "context window. Prefer minimal, high-leverage operations.*"
+        )
+
+    return (
+        f"## Topic\n{source_query}\n\n"
+        f"## CURRENT DOCUMENT (apply ops to this; reference section ids as listed)\n"
+        f"```json\n{doc_json}\n```\n\n"
+        f"## NEW INFORMATION SYNTHESIS (context for how new facts relate to the topic)\n"
+        f"```markdown\n{candidate}\n```\n\n"
+        f"## SUPPORTING FACTS (new since last refresh — integrate these)\n{facts_body}"
+        f"{budget_hint}{truncation_note}\n\n"
+        f"{task_footer}"
+    )
 
 
 DELTA_SYSTEM_PROMPT = """You are performing a surgical delta update to an existing mental model document.

hindsight-api-slim/hindsight_api/engine/reflect/tools_schema.py

@@ -232,6 +232,7 @@ def get_reflect_tools(
     include_mental_models: bool = True,
     include_observations: bool = True,
     include_recall: bool = True,
+    include_expand: bool = True,
 ) -> list[dict]:
     """
     Get the list of tools for the reflect agent.
@@ -247,6 +248,9 @@ def get_reflect_tools(
         include_mental_models: Whether to include the search_mental_models tool.
         include_observations: Whether to include the search_observations tool.
         include_recall: Whether to include the recall tool.
+        include_expand: Whether to include the expand tool. Disabled when raw
+            document/chunk text is not stored, since expand only reads back
+            source text and would return empty results.
 
     Returns:
         List of tool definitions in OpenAI format
@@ -260,7 +264,8 @@ def get_reflect_tools(
     if include_recall:
         tools.append(TOOL_RECALL)
 
-    tools.append(TOOL_EXPAND)
+    if include_expand:
+        tools.append(TOOL_EXPAND)
 
     # Use directive-aware done tool if directives are present
     if directive_rules:

hindsight-api-slim/hindsight_api/engine/response_models.py

@@ -105,6 +105,34 @@ class TokenUsage(BaseModel):
         )
 
 
+class ExtractedFact(BaseModel):
+    """A single candidate fact produced by dry-run extraction (no resolution/links/persistence).
+
+    A deliberate subset of the persisted memory-unit shape — only the fields a fresh extraction
+    yields. Storage/consolidation/curation fields (id, document_id, chunk_id, proof_count, state, …)
+    are omitted because nothing is stored. Entities are raw, unresolved names.
+    """
+
+    text: str = Field(description="The extracted fact text.")
+    fact_type: str = Field(description="Perspective classification: 'world' or 'experience'.")
+    occurred_start: str | None = Field(default=None, description="ISO timestamp the fact's event started, if dated.")
+    occurred_end: str | None = Field(default=None, description="ISO timestamp the fact's event ended, if dated.")
+    entities: list[str] = Field(
+        default_factory=list, description="Raw (unresolved) entity names mentioned in the fact."
+    )
+
+
+class DryRunExtractionResult(BaseModel):
+    """Result of dry-run fact extraction: candidate facts plus aggregated LLM token usage."""
+
+    facts: list[ExtractedFact] = Field(
+        default_factory=list, description="Candidate facts the retain step would extract."
+    )
+    usage: TokenUsage = Field(
+        default_factory=TokenUsage, description="Aggregated token usage across the extraction LLM calls."
+    )
+
+
 class DispositionTraits(BaseModel):
     """
     Disposition traits for a memory bank.

hindsight-api-slim/hindsight_api/engine/retain/bank_utils.py

@@ -4,7 +4,6 @@ bank profile utilities for disposition and mission management.
 
 import json
 import logging
-import re
 import uuid
 from dataclasses import dataclass
 from typing import TypedDict

hindsight-api-slim/hindsight_api/engine/retain/chunk_storage.py

@@ -8,6 +8,7 @@ import hashlib
 import logging
 from dataclasses import dataclass
 
+from ...config import get_config
 from ..memory_engine import fq_table
 from .types import ChunkMetadata
 
@@ -88,6 +89,11 @@ async def store_chunks_batch(
     if not chunks:
         return {}
 
+    # When document text storage is disabled, persist empty chunk_text (the
+    # column is NOT NULL) while still computing content_hash from the real text
+    # so delta-retain dedup is unaffected.
+    store_text = get_config().store_document_text
+
     # Prepare chunk data for batch insert
     chunk_ids = []
     chunk_texts = []
@@ -98,7 +104,7 @@ async def store_chunks_batch(
     for chunk in chunks:
         chunk_id = f"{bank_id}_{document_id}_{chunk.chunk_index}"
         chunk_ids.append(chunk_id)
-        chunk_texts.append(chunk.chunk_text)
+        chunk_texts.append(chunk.chunk_text if store_text else "")
         chunk_indices.append(chunk.chunk_index)
         content_hashes.append(compute_chunk_hash(chunk.chunk_text))
         chunk_id_map[chunk.chunk_index] = chunk_id

hindsight-api-slim/hindsight_api/engine/retain/embedding_utils.py

@@ -3,6 +3,7 @@ Embedding generation utilities for memory units.
 """
 
 import asyncio
+import contextvars
 import logging
 from typing import Literal, Protocol
 
@@ -89,7 +90,14 @@ async def generate_embeddings_batch(
     """
     try:
         loop = asyncio.get_event_loop()
-        embeddings = await loop.run_in_executor(None, _encode_with_input_type, embeddings_backend, texts, input_type)
+        # run_in_executor runs the encode in a worker thread, which does NOT inherit
+        # the caller's contextvars. Capture the current context and run the encode
+        # inside it so context-dependent behavior (e.g. per-bank `user` attribution
+        # read via get_current_bank_id()) survives the thread hop.
+        ctx = contextvars.copy_context()
+        embeddings = await loop.run_in_executor(
+            None, lambda: ctx.run(_encode_with_input_type, embeddings_backend, texts, input_type)
+        )
     except Exception as e:
         raise Exception(f"Failed to generate batch embeddings: {str(e)}")
 

hindsight-api-slim/hindsight_api/engine/retain/fact_extraction.py

@@ -14,7 +14,6 @@ from typing import Any, Literal, cast
 
 from pydantic import BaseModel, ConfigDict, Field, create_model, field_validator
 
-from ...config import get_config
 from ..llm_wrapper import LLMConfig, OutputTooLongError, sanitize_llm_output
 from ..operation_metadata import RetainExtractionErrors
 from ..response_models import TokenUsage
@@ -406,64 +405,93 @@ class VerbatimFactExtractionResponse(BaseModel):
     facts: list[VerbatimExtractedFact] = Field(description="List of metadata entries (one per chunk)")
 
 
-def chunk_text(text: str, max_chars: int) -> list[str]:
+# Separators for sentence-aware recursive text splitting, ordered most- to
+# least-preferred. The final "" lets the splitter break mid-word as a last
+# resort so a chunk can never exceed the size budget.
+_RECURSIVE_TEXT_SEPARATORS = [
+    "\n\n",  # Paragraph breaks
+    "\n",  # Line breaks
+    ". ",  # Sentence endings
+    "! ",  # Exclamations
+    "? ",  # Questions
+    "; ",  # Semicolons
+    ", ",  # Commas
+    " ",  # Words
+    "",  # Characters (last resort)
+]
+
+
+def _split_oversized_unit(text: str, max_chars: int) -> list[str]:
+    """Sentence-aware split of a single unit that overflowed the budget.
+
+    Used when one JSONL line / conversation turn is so large it can't be kept
+    whole within the configured structured-chunk limit. The resulting fragments
+    are no longer valid JSON, but the fact extractor treats every chunk as plain
+    text.
+    """
+    from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+    splitter = RecursiveCharacterTextSplitter(
+        chunk_size=max_chars,
+        chunk_overlap=0,
+        length_function=len,
+        is_separator_regex=False,
+        separators=_RECURSIVE_TEXT_SEPARATORS,
+    )
+    return splitter.split_text(text)
+
+
+def chunk_text(text: str, max_chars: int, structured_chunk_size: int | None = None) -> list[str]:
     """
     Split text into chunks, preserving conversation structure when possible.
 
-    For JSON conversation arrays (user/assistant turns), splits at turn boundaries
-    while preserving speaker context. For plain text, uses sentence-aware splitting.
+    For JSON conversation arrays (user/assistant turns) and JSONL (newline-delimited
+    JSON objects), splits at turn/line boundaries so no object is split across chunks.
+    A single turn/line that overflows ``max_chars`` is kept whole only up to
+    ``structured_chunk_size``. When unset, that limit defaults to ``max_chars``.
+    For plain text, uses sentence-aware splitting.
 
     Args:
-        text: Input text to chunk (plain text or JSON conversation)
-        max_chars: Maximum characters per chunk (default 120k ≈ 30k tokens)
+        text: Input text to chunk (plain text, JSON conversation, or JSONL)
+        max_chars: Target maximum characters per chunk
+        structured_chunk_size: Maximum characters for a single JSONL line or
+            conversation turn to keep whole. Defaults to ``max_chars``.
 
     Returns:
         List of text chunks, roughly under max_chars
     """
-    from langchain_text_splitters import RecursiveCharacterTextSplitter
-
     # If text is small enough, return as-is
     if len(text) <= max_chars:
         return [text]
 
+    structured_limit = structured_chunk_size if structured_chunk_size is not None else max_chars
+
     # Try to parse as JSON conversation array
     try:
         parsed = json.loads(text)
         if isinstance(parsed, list) and all(isinstance(turn, dict) for turn in parsed):
             # This looks like a conversation - chunk at turn boundaries
-            return _chunk_conversation(parsed, max_chars)
+            return _chunk_conversation(parsed, max_chars, structured_limit)
     except (json.JSONDecodeError, ValueError):
         pass
 
+    # Try to parse as JSONL (newline-delimited JSON objects, e.g. session logs)
+    jsonl_chunks = _chunk_jsonl(text, max_chars, structured_limit)
+    if jsonl_chunks is not None:
+        return jsonl_chunks
+
     # Fall back to sentence-aware text splitting
-    splitter = RecursiveCharacterTextSplitter(
-        chunk_size=max_chars,
-        chunk_overlap=0,
-        length_function=len,
-        is_separator_regex=False,
-        separators=[
-            "\n\n",  # Paragraph breaks
-            "\n",  # Line breaks
-            ". ",  # Sentence endings
-            "! ",  # Exclamations
-            "? ",  # Questions
-            "; ",  # Semicolons
-            ", ",  # Commas
-            " ",  # Words
-            "",  # Characters (last resort)
-        ],
-    )
-
-    return splitter.split_text(text)
+    return _split_oversized_unit(text, max_chars)
 
 
-def _chunk_conversation(turns: list[dict], max_chars: int) -> list[str]:
+def _chunk_conversation(turns: list[dict], max_chars: int, structured_limit: int) -> list[str]:
     """
     Chunk a conversation array at turn boundaries, preserving complete turns.
 
     Args:
         turns: List of conversation turn dicts (with 'role' and 'content' keys)
         max_chars: Maximum characters per chunk
+        structured_limit: Maximum characters for a single turn to keep whole
 
     Returns:
         List of JSON-serialized chunks, each containing complete turns
@@ -473,28 +501,105 @@ def _chunk_conversation(turns: list[dict], max_chars: int) -> list[str]:
     current_chunk = []
     current_size = 2  # Account for "[]"
 
-    for turn in turns:
-        # Estimate size of this turn when serialized (with comma separator)
-        turn_json = json.dumps(turn, ensure_ascii=False)
-        turn_size = len(turn_json) + 1  # +1 for comma
-
-        # If adding this turn would exceed limit and we have turns, save current chunk
-        if current_size + turn_size > max_chars and current_chunk:
+    def _flush() -> None:
+        nonlocal current_chunk, current_size
+        if current_chunk:
             chunks.append(json.dumps(current_chunk, ensure_ascii=False))
             current_chunk = []
             current_size = 2  # Reset to "[]"
 
+    for turn in turns:
+        # Estimate size of this turn when serialized (with comma separator)
+        turn_json = json.dumps(turn, ensure_ascii=False)
+        turn_unit_size = len(turn_json)
+        turn_size = turn_unit_size + 1  # +1 for comma
+
+        # A turn too large to keep whole even alone: flush, then split it as
+        # text so no chunk runs far over budget (the extractor won't re-chunk).
+        if turn_unit_size > structured_limit:
+            _flush()
+            chunks.extend(_split_oversized_unit(turn_json, structured_limit))
+            continue
+
+        # If adding this turn would exceed limit and we have turns, save current chunk
+        if current_size + turn_size > max_chars and current_chunk:
+            _flush()
+
         # Add turn to current chunk
         current_chunk.append(turn)
         current_size += turn_size
 
     # Add final chunk if non-empty
-    if current_chunk:
-        chunks.append(json.dumps(current_chunk, ensure_ascii=False))
+    _flush()
 
     return chunks if chunks else [json.dumps(turns, ensure_ascii=False)]
 
 
+def _chunk_jsonl(text: str, max_chars: int, structured_limit: int) -> list[str] | None:
+    """Chunk newline-delimited JSON (JSONL) at line boundaries.
+
+    Detects JSONL — two or more non-empty lines, each a complete JSON object —
+    and packs whole lines into chunks so no line is split across chunks (multiple
+    short lines may share a chunk). A line that overflows ``max_chars`` is kept
+    whole only up to ``structured_limit``. Returns ``None`` if the input is not
+    JSONL, so the caller falls back to plain-text splitting.
+
+    Args:
+        text: Input text to inspect/chunk.
+        max_chars: Maximum characters per chunk.
+        structured_limit: Maximum characters for a single JSONL line to
+            keep whole.
+
+    Returns:
+        List of JSONL chunks (lines joined by newline), or ``None`` if not JSONL.
+    """
+    lines = [line for line in text.splitlines() if line.strip()]
+    if len(lines) < 2:
+        return None
+
+    for line in lines:
+        try:
+            obj = json.loads(line)
+        except (json.JSONDecodeError, ValueError):
+            return None
+        if not isinstance(obj, dict):
+            return None
+
+    chunks: list[str] = []
+    current_chunk: list[str] = []
+    current_size = 0
+
+    def _flush() -> None:
+        nonlocal current_chunk, current_size
+        if current_chunk:
+            chunks.append("\n".join(current_chunk))
+            current_chunk = []
+            current_size = 0
+
+    for line in lines:
+        line_unit_size = len(line)
+        line_size = len(line) + 1  # +1 for the joining newline
+
+        # A line too large to keep whole even alone: flush, then split it as
+        # text so no chunk runs far over budget (the extractor won't re-chunk).
+        if line_unit_size > structured_limit:
+            _flush()
+            chunks.extend(_split_oversized_unit(line, structured_limit))
+            continue
+
+        # If adding this line would exceed the limit and we have lines, flush.
+        # A line up to structured_limit is kept whole (a bounded overflow).
+        if current_size + line_size > max_chars and current_chunk:
+            _flush()
+
+        current_chunk.append(line)
+        current_size += line_size
+
+    _flush()
+
+    return chunks
+
+
 # =============================================================================
 # FACT EXTRACTION PROMPTS
 # =============================================================================
@@ -1634,7 +1739,11 @@ async def extract_facts_from_text(
         - chunks: List of tuples (chunk_text, fact_count) for each chunk
         - usage: Aggregated token usage across all LLM calls
     """
-    chunks = chunk_text(text, max_chars=config.retain_chunk_size)
+    chunks = chunk_text(
+        text,
+        max_chars=config.retain_chunk_size,
+        structured_chunk_size=config.retain_structured_chunk_size,
+    )
 
     # Log chunk count before starting LLM requests
     total_chars = sum(len(c) for c in chunks)
@@ -1776,8 +1885,7 @@ async def extract_facts_from_contents_batch_api(
 
     logger.info(f"Using Batch API for fact extraction ({len(contents)} contents)")
 
-    # Check config for extraction mode and causal link extraction (used throughout)
-    extraction_mode = config.retain_extraction_mode
+    # Check config for causal link extraction (used throughout)
     extract_causal_links = config.retain_extract_causal_links
 
     # Check if provider supports batch API
@@ -1818,7 +1926,11 @@ async def extract_facts_from_contents_batch_api(
     prompt, response_schema = _build_extraction_prompt_and_schema(config)
 
     for content_index, item in enumerate(contents):
-        chunks = chunk_text(item.content, max_chars=config.retain_chunk_size)
+        chunks = chunk_text(
+            item.content,
+            max_chars=config.retain_chunk_size,
+            structured_chunk_size=config.retain_structured_chunk_size,
+        )
 
         for chunk_index_in_content, chunk in enumerate(chunks):
             all_chunks_info.append((chunk, content_index, chunk_index_in_content, item.event_date, item.context))
@@ -2239,7 +2351,11 @@ def _extract_facts_chunks(
     global_chunk_idx = 0
 
     for content_index, content in enumerate(contents):
-        chunks = chunk_text(content.content, config.retain_chunk_size)
+        chunks = chunk_text(
+            content.content,
+            config.retain_chunk_size,
+            structured_chunk_size=config.retain_structured_chunk_size,
+        )
         for chunk in chunks:
             chunks_metadata.append(
                 ChunkMetadata(

hindsight-api-slim/hindsight_api/engine/retain/fact_storage.py

@@ -399,7 +399,12 @@ async def _upsert_document_row(
     INSERT so that re-ingesting a document (which deletes + inserts the row)
     keeps the original creation timestamp. ``updated_at`` is always set to
     ``NOW()`` on both INSERT and the ON CONFLICT UPDATE branch.
+
+    When ``store_document_text`` is disabled, the raw source text
+    is dropped and ``original_text`` is stored as NULL. The ``content_hash`` is
+    still computed from the real content so delta-retain dedup is unaffected.
     """
+    original_text = combined_content if get_config().store_document_text else None
     await conn.execute(
         f"""
         INSERT INTO {fq_table("documents")} (id, bank_id, original_text, content_hash, retain_params, tags, created_at, updated_at)
@@ -413,7 +418,7 @@ async def _upsert_document_row(
         """,
         document_id,
         bank_id,
-        combined_content,
+        original_text,
         content_hash,
         json.dumps(retain_params) if retain_params else None,
         document_tags or [],

hindsight-api-slim/hindsight_api/engine/retain/link_utils.py

@@ -574,12 +574,10 @@ async def compute_semantic_links_ann(
     # the transaction end handles both.
     rows: list = []
     async with conn.transaction():
-        # Transaction-local ANN tuning. Each supported backend exposes its own
-        # GUC (hnsw.ef_search on pgvector, vchordrq.probes on vchord); the
-        # dispatcher returns the right knob for the configured backend with a
-        # value tuned for top-50 semantic link creation (lower recall but much
-        # lower latency than the recall-side default). SET LOCAL auto-reverts
-        # at commit, so we don't pollute the pool for subsequent queries.
+        # Transaction-local ANN tuning. The dispatcher only returns GUCs that
+        # are safe to apply at session/transaction scope for the configured
+        # backend. VectorChord probe values are index-shaped, so vchordrq uses
+        # index storage fallback parameters instead of a blanket SET LOCAL.
         for guc, value in ann_search_tuning_settings(configured_vector_extension(), kind="low_latency"):
             await conn.execute(f"SET LOCAL {guc} = {value}")
 
@@ -636,7 +634,7 @@ async def compute_semantic_links_ann(
             logger.debug(f"[ANN] fact_type={fact_type}: {len(ft_rows)} rows in {time_mod.time() - t_query:.3f}s")
             rows.extend(ft_rows)
     # Transaction commits here. _ann_seeds is dropped (ON COMMIT DROP).
-    # hnsw.ef_search reverts (SET LOCAL).
+    # Transaction-local ANN tuning reverts (SET LOCAL).
 
     for row in rows:
         sim = float(min(1.0, max(0.0, row["similarity"])))
@@ -802,8 +800,6 @@ async def create_causal_links_batch(
     try:
         import time as time_mod
 
-        create_start = time_mod.time()
-
         # Build links list
         links = []
         for fact_idx, causal_relations in enumerate(causal_relations_per_fact):

hindsight-api-slim/hindsight_api/engine/retain/orchestrator.py

@@ -15,18 +15,166 @@ from dataclasses import dataclass
 from datetime import UTC, datetime
 from typing import Any
 
+from ...extensions.memory_defense import (
+    DefenseAction,
+    DefenseDecision,
+    MemoryDefenseExtension,
+    apply_redaction,
+    parse_policy,
+)
 from ...worker.stage import set_stage
-from ..db.base import DatabaseBackend
 from ..db_utils import acquire_with_retry
 from ..memory_engine import count_tokens, fq_table
 from . import bank_utils
 
 
+@dataclass
+class BlockedViolation:
+    """One item blocked by the Memory Defense policy (surfaced in the 422 body)."""
+
+    index: int
+    detector: str | None
+    message: str
+
+
+class MemoryDefenseAllBlockedError(Exception):
+    """Raised when every item in a retain batch is blocked by the Memory Defense policy."""
+
+    def __init__(self, violations: list[BlockedViolation]) -> None:
+        self.violations = violations
+        super().__init__(f"all {len(violations)} items blocked by Memory Defense policy")
+
+
 def utcnow():
     """Get current UTC time."""
     return datetime.now(UTC)
 
 
+def _redact_document_body(body: str, config: Any) -> str:
+    """Apply Memory Defense redaction to a document body.
+
+    Per-item screening only scrubs the chunked content that goes through
+    `screen()`. When a sub-batch carries `document_body_override` (the full
+    original text of an oversized item — see `_split_contents_into_sub_batches`),
+    that override bypasses screening and would persist verbatim into
+    `documents.original_text`. Apply the same redactor here so the document
+    body is scrubbed regardless of which path produced it.
+    """
+    try:
+        policy = parse_policy(getattr(config, "memory_defense", None))
+    except Exception:
+        return body
+    if not policy.enabled:
+        return body
+    if not any(r.on == "sensitive_data" for r in policy.rules):
+        return body
+    return apply_redaction(body).content
+
+
+async def _fire_memory_defense_webhook(
+    webhook_manager: Any,
+    *,
+    conn: Any,
+    schema: str | None,
+    bank_id: str,
+    operation_id: str | None,
+    document_id: str | None,
+    decision: DefenseDecision,
+) -> None:
+    """Fire a memory_defense.triggered webhook for a non-allow decision.
+
+    No-op when no webhook manager is wired or none is subscribed. Delivery
+    failures are swallowed so screening never blocks a retain.
+    """
+    if webhook_manager is None:
+        return
+    try:
+        from ...webhooks import (
+            MemoryDefenseEventData,
+            MemoryDefenseHit,
+            WebhookEvent,
+            WebhookEventType,
+        )
+
+        # Translate per-match raw dicts on the decision into MemoryDefenseHit
+        # entries on the wire. The decision's hits list is already fingerprinted
+        # by apply_redaction (the raw value never lands in hits, by contract),
+        # so this is purely a shape conversion. None when no per-hit data is
+        # available so receivers can distinguish "no preview info" from
+        # "scanned, nothing matched" (the latter wouldn't be a webhook delivery
+        # in the first place).
+        decision_hits = getattr(decision, "hits", None) or []
+        hits: list[MemoryDefenseHit] | None = [
+            MemoryDefenseHit(
+                detector=str(h.get("detector") or ""),
+                preview=str(h.get("preview") or ""),
+            )
+            for h in decision_hits
+            if h.get("detector") and h.get("preview")
+        ] or None
+
+        event = WebhookEvent(
+            event=WebhookEventType.MEMORY_DEFENSE_TRIGGERED,
+            bank_id=bank_id,
+            operation_id=operation_id or "",
+            status=decision.action.value,
+            timestamp=utcnow(),
+            data=MemoryDefenseEventData(
+                action=decision.action.value,
+                detector=decision.detector,
+                document_id=document_id,
+                matched_types=decision.matched_types or None,
+                message=decision.message or None,
+                hits=hits,
+                # Optional SIEM-enrichment fields populated by downstream
+                # extensions (e.g. hindsight-cloud's _CloudDefenseDecision
+                # subclass). Read via getattr so OSS doesn't need to know
+                # about extension subclasses. Combined with the manager's
+                # exclude_none serialization, missing values stay absent
+                # from the wire entirely rather than appearing as null.
+                severity=getattr(decision, "severity", None),
+                api_key_name=getattr(decision, "api_key_name", None),
+                memory_unit_id=getattr(decision, "memory_unit_id", None),
+                receipt_uri=getattr(decision, "receipt_uri", None),
+            ),
+        )
+        await webhook_manager.fire_event_with_conn(event, conn, schema=schema)
+    except Exception:
+        logger.warning("memory_defense webhook delivery failed", exc_info=True)
+
+
+def _audit_memory_defense(
+    audit_logger: Any,
+    *,
+    bank_id: str,
+    document_id: str | None,
+    decision: DefenseDecision,
+) -> None:
+    """Write a fire-and-forget ``memory_defense`` audit entry for a non-allow decision.
+
+    No-op when audit logging is disabled (the logger gates on its own config).
+    The action taken (redact/block) and what matched live in the entry metadata.
+    """
+    if audit_logger is None:
+        return
+    from ..audit import AuditEntry
+
+    entry = AuditEntry(
+        action="memory_defense",
+        transport="system",
+        bank_id=bank_id,
+        metadata={
+            "action": decision.action.value,
+            "detector": decision.detector,
+            "document_id": document_id,
+            "matched_types": decision.matched_types,
+            "message": decision.message,
+        },
+    )
+    entry.ended_at = entry.started_at  # point-in-time policy decision (duration 0)
+    audit_logger.log_fire_and_forget(entry)
+
+
 def _merge_processed_content_tokens(a: int | None, b: int | None) -> int | None:
     """Combine the processed-content-tokens signal across sub-results.
 
@@ -158,6 +306,8 @@ def _build_retain_params(contents_dicts, document_tags=None, doc_contents=None):
             )
         if first_item.get("metadata"):
             retain_params["metadata"] = first_item["metadata"]
+        if first_item.get("observation_scopes") is not None:
+            retain_params["observation_scopes"] = first_item["observation_scopes"]
 
     return retain_params, merged_tags
 
@@ -425,6 +575,9 @@ async def retain_batch(
     document_body_override: str | None = None,
     chunk_index_offset: int = 0,
     progress_callback: "Callable[..., Awaitable[None]] | None" = None,
+    webhook_manager: Any = None,
+    memory_defense_extension: "MemoryDefenseExtension | None" = None,
+    audit_logger: Any = None,
 ) -> tuple[list[list[str]], TokenUsage, int | None]:
     """
     Process a batch of content through the retain pipeline.
@@ -515,6 +668,10 @@ async def retain_batch(
                     db_semaphore=db_semaphore,
                     document_body_override=document_body_override,
                     chunk_index_offset=chunk_index_offset,
+                    progress_callback=progress_callback,
+                    webhook_manager=webhook_manager,
+                    memory_defense_extension=memory_defense_extension,
+                    audit_logger=audit_logger,
                 )
                 for group_idx, orig_idx in enumerate(original_indices[doc_key]):
                     if group_idx < len(group_ids):
@@ -523,6 +680,80 @@ async def retain_batch(
                 total_processed_tokens = _merge_processed_content_tokens(total_processed_tokens, group_processed)
             return result_unit_ids, total_usage, total_processed_tokens
 
+    # --- Memory Defense pre-extraction screening ---
+    # Delegate to the loaded extension. `config` is a resolved HindsightConfig
+    # object at this point (see _retain_batch_async_internal). On a non-allow
+    # decision we redact in place or drop the item, and fire a
+    # memory_defense.triggered webhook when one is configured.
+    _policy = parse_policy(getattr(config, "memory_defense", None))
+    _blocked_violations: list[BlockedViolation] = []
+
+    if memory_defense_extension is not None and _policy.enabled:
+        async with acquire_with_retry(pool) as _defense_conn:
+            for _idx, _content in enumerate(contents):
+                # Prefer the per-item document_id over the batch-level value so
+                # the decision and webhook carry the document the caller
+                # submitted, not whichever doc_id the batch happens to share.
+                _item_doc_id = contents_dicts[_idx].get("document_id") or document_id
+
+                _decision = await memory_defense_extension.screen(
+                    policy=_policy,
+                    bank_id=bank_id,
+                    document_id=_item_doc_id,
+                    content=_content.content,
+                    tags=_content.tags,
+                )
+
+                if _decision.action is DefenseAction.ALLOW:
+                    continue
+
+                if _decision.action is DefenseAction.REDACT:
+                    _redacted = _decision.redacted_content or _content.content
+                    _content.content = _redacted
+                    # Mirror the redaction into the raw dict so the document
+                    # body persisted further down the pipeline also stores the
+                    # redacted text, not the verbatim secret.
+                    contents_dicts[_idx]["content"] = _redacted
+                elif _decision.action is DefenseAction.BLOCK:
+                    _blocked_violations.append(
+                        BlockedViolation(
+                            index=_idx,
+                            detector=_decision.detector,
+                            message=_decision.message,
+                        )
+                    )
+
+                await _fire_memory_defense_webhook(
+                    webhook_manager,
+                    conn=_defense_conn,
+                    schema=schema,
+                    bank_id=bank_id,
+                    operation_id=operation_id,
+                    document_id=_item_doc_id,
+                    decision=_decision,
+                )
+                _audit_memory_defense(
+                    audit_logger,
+                    bank_id=bank_id,
+                    document_id=_item_doc_id,
+                    decision=_decision,
+                )
+
+    if _blocked_violations:
+        # All items blocked → raise so the HTTP layer can return 422.
+        if len(_blocked_violations) == len(contents):
+            raise MemoryDefenseAllBlockedError(_blocked_violations)
+
+        # Remove blocked items from the pipeline.
+        _skip_indices = {v.index for v in _blocked_violations}
+        if _skip_indices:
+            _surviving = [i for i in range(len(contents)) if i not in _skip_indices]
+            contents = [contents[i] for i in _surviving]
+            contents_dicts = [contents_dicts[i] for i in _surviving]
+            # If nothing survives, return empty results immediately.
+            if not contents:
+                return [[] for _ in contents_dicts], TokenUsage(), 0
+
     # Resolve effective document_id early so both delta and streaming paths
     # can find existing chunks from a prior attempt. On retry, a generated
     # document_id is recovered from operation result_metadata.document_ids[0].
@@ -667,10 +898,15 @@ async def retain_batch(
     # retain code paths.
     chunk_batch_size = getattr(config, "retain_chunk_batch_size", 100)
     chunk_size = getattr(config, "retain_chunk_size", 3000)
+    structured_chunk_size = getattr(config, "retain_structured_chunk_size", None)
     all_pre_chunks: list[str] = []
     chunk_to_content: list[int] = []  # maps chunk index -> index into contents
     for content_idx, content in enumerate(contents):
-        content_chunks = fact_extraction.chunk_text(content.content, chunk_size)
+        content_chunks = fact_extraction.chunk_text(
+            content.content,
+            chunk_size,
+            structured_chunk_size=structured_chunk_size,
+        )
         all_pre_chunks.extend(content_chunks)
         chunk_to_content.extend([content_idx] * len(content_chunks))
 
@@ -895,7 +1131,9 @@ async def _streaming_retain_batch(
     # so documents.original_text stores the complete payload, not just this
     # slice (issue #1838).
     if document_body_override is not None:
-        combined_content = document_body_override
+        # The override is the unmodified original body — apply redaction so
+        # secrets in oversized inputs don't bypass screening.
+        combined_content = _redact_document_body(document_body_override, config)
     else:
         combined_content = "\n".join([c.get("content", "") for c in contents_dicts])
     # Memory: contents_dicts content strings are now captured in combined_content.
@@ -1686,6 +1924,7 @@ async def _try_delta_retain(
             start_time,
             outbox_callback,
             document_body_override=document_body_override,
+            config=config,
         )
 
     # Build content items for only the changed/new chunks
@@ -1703,6 +1942,7 @@ async def _try_delta_retain(
             start_time,
             outbox_callback,
             document_body_override=document_body_override,
+            config=config,
         )
 
     # Freshness recheck BEFORE the (expensive) LLM extraction.
@@ -1754,6 +1994,7 @@ async def _try_delta_retain(
                 start_time,
                 outbox_callback,
                 document_body_override=document_body_override,
+                config=config,
             )
         log_buffer.append(
             f"[delta] Recheck: {len(recheck.changed) + len(recheck.new) + len(recheck.removed)} chunks still differ — "
@@ -1829,9 +2070,10 @@ async def _try_delta_retain(
                 step_start = time.time()
                 # When this sub-batch is one slice of an oversized item
                 # split across multiple sub-batches, store the full body
-                # (issue #1838) instead of just the slice.
+                # (issue #1838) instead of just the slice. Redact the
+                # override since it bypassed per-chunk screening.
                 if document_body_override is not None:
-                    combined_content = document_body_override
+                    combined_content = _redact_document_body(document_body_override, config)
                 else:
                     combined_content = "\n".join([c.get("content", "") for c in contents_dicts])
                 retain_params, merged_tags = _build_retain_params(contents_dicts, document_tags)
@@ -1960,6 +2202,7 @@ async def _delta_metadata_only(
     outbox_callback,
     *,
     document_body_override: str | None = None,
+    config: Any = None,
 ):
     """Handle the case where no chunks changed — just update document metadata and tags."""
     async with acquire_with_retry(pool) as conn:
@@ -1972,8 +2215,9 @@ async def _delta_metadata_only(
             )
             # When this sub-batch is a slice of an oversized item, write the
             # full original body (issue #1838) instead of just the slice.
+            # Redact the override since it bypassed per-chunk screening.
             if document_body_override is not None:
-                combined_content = document_body_override
+                combined_content = _redact_document_body(document_body_override, config)
             else:
                 combined_content = "\n".join([c.get("content", "") for c in contents_dicts])
             retain_params, merged_tags = _build_retain_params(contents_dicts, document_tags)
@@ -2042,9 +2286,14 @@ def _chunk_contents_for_delta(contents: list[RetainContent], config) -> dict[int
     """
     result = {}
     global_chunk_idx = 0
+    chunk_size = getattr(config, "retain_chunk_size", 3000)
+    structured_chunk_size = getattr(config, "retain_structured_chunk_size", None)
     for content in contents:
-        chunk_size = getattr(config, "retain_chunk_size", 3000)
-        chunks = fact_extraction.chunk_text(content.content, chunk_size)
+        chunks = fact_extraction.chunk_text(
+            content.content,
+            chunk_size,
+            structured_chunk_size=structured_chunk_size,
+        )
         for chunk_text in chunks:
             result[global_chunk_idx] = chunk_text
             global_chunk_idx += 1

hindsight-api-slim/hindsight_api/engine/retain/types.py

@@ -24,7 +24,9 @@ class RetainContentDict(TypedDict, total=False):
         tags: Visibility scope tags for this content item (optional)
         observation_scopes: How to scope observations for consolidation (optional).
             "per_tag" runs one pass per individual tag; "combined" (default) runs a
-            single pass with all tags; a list[list[str]] specifies exact passes.
+            single pass with all tags; "shared" runs a single pass over one global,
+            untagged scope so memories consolidate together regardless of tags;
+            a list[list[str]] specifies exact passes.
         update_mode: How to handle existing documents with the same document_id (optional).
             "replace" (default) deletes old data and reprocesses. "append" concatenates
             new content to the existing document and reprocesses.
@@ -38,7 +40,7 @@ class RetainContentDict(TypedDict, total=False):
     entities: list[dict[str, str]]  # [{"text": "...", "type": "..."}]
     tags: list[str]  # Visibility scope tags
     observation_scopes: (
-        Literal["per_tag", "combined", "all_combinations"] | list[list[str]]
+        Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]]
     )  # Observation scopes for consolidation
     update_mode: Literal["replace", "append"]
 
@@ -57,7 +59,7 @@ class RetainContent:
     metadata: dict[str, str] = field(default_factory=dict)
     entities: list[dict[str, str]] = field(default_factory=list)  # User-provided entities
     tags: list[str] = field(default_factory=list)  # Visibility scope tags
-    observation_scopes: Literal["per_tag", "combined", "all_combinations"] | list[list[str]] | None = (
+    observation_scopes: Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]] | None = (
         None  # Observation scopes
     )
 
@@ -124,7 +126,7 @@ class ExtractedFact:
     mentioned_at: datetime | None = None
     metadata: dict[str, str] = field(default_factory=dict)
     tags: list[str] = field(default_factory=list)  # Visibility scope tags
-    observation_scopes: Literal["per_tag", "combined", "all_combinations"] | list[list[str]] | None = (
+    observation_scopes: Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]] | None = (
         None  # Observation scopes
     )
 
@@ -176,7 +178,7 @@ class ProcessedFact:
     tags: list[str] = field(default_factory=list)
 
     # Observation scopes for consolidation
-    observation_scopes: Literal["per_tag", "combined", "all_combinations"] | list[list[str]] | None = None
+    observation_scopes: Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]] | None = None
 
     @property
     def is_duplicate(self) -> bool:

hindsight-api-slim/hindsight_api/engine/search/fusion.py

@@ -2,8 +2,6 @@
 Helper functions for hybrid search (semantic + BM25 + graph).
 """
 
-from typing import Any
-
 from .types import MergedCandidate, RetrievalResult
 
 
@@ -156,39 +154,3 @@ def interleave_fusion(result_lists: list[list[RetrievalResult]]) -> list[MergedC
         )
         for pos, doc_id in enumerate(ordered_ids)
     ]
-
-
-def normalize_scores_on_deltas(results: list[dict[str, Any]], score_keys: list[str]) -> list[dict[str, Any]]:
-    """
-    Normalize scores based on deltas (min-max normalization within result set).
-
-    This ensures all scores are in [0, 1] range based on the spread in THIS result set.
-
-    Args:
-        results: List of result dicts
-        score_keys: Keys to normalize (e.g., ["recency", "frequency"])
-
-    Returns:
-        Results with normalized scores added as "{key}_normalized"
-    """
-    for key in score_keys:
-        values = [r.get(key, 0.0) for r in results if key in r]
-
-        if not values:
-            continue
-
-        min_val = min(values)
-        max_val = max(values)
-        delta = max_val - min_val
-
-        if delta > 0:
-            for r in results:
-                if key in r:
-                    r[f"{key}_normalized"] = (r[key] - min_val) / delta
-        else:
-            # All values are the same, set to 0.5
-            for r in results:
-                if key in r:
-                    r[f"{key}_normalized"] = 0.5
-
-    return results

hindsight-api-slim/hindsight_api/engine/search/reranking.py

@@ -99,9 +99,15 @@ def apply_combined_scoring(
 
     for sr in scored_results:
         # Recency: linear decay over 365 days → [0.1, 1.0]; neutral 0.5 if no date.
+        # Use the unit's effective time (occurred_start, then mentioned_at, then
+        # occurred_end) — the same COALESCE order as retrieval._coalesce_date — so a
+        # memory that carries only a mentioned_at / occurred_end (e.g. conversation
+        # facts or ongoing states that intentionally lack occurred_start) still gets
+        # correct recency ordering instead of a flat neutral 0.5.
         sr.recency = 0.5
-        if sr.retrieval.occurred_start:
-            occurred = sr.retrieval.occurred_start
+        effective = sr.retrieval.occurred_start or sr.retrieval.mentioned_at or sr.retrieval.occurred_end
+        if effective:
+            occurred = effective
             if occurred.tzinfo is None:
                 occurred = occurred.replace(tzinfo=UTC)
             days_ago = (now - occurred).total_seconds() / 86400

hindsight-api-slim/hindsight_api/engine/search/retrieval.py

@@ -13,7 +13,7 @@ import logging
 import re
 from dataclasses import dataclass, field
 from datetime import UTC, datetime
-from typing import Any, Optional
+from typing import TYPE_CHECKING, Any, Optional
 
 from ...config import get_config
 from ..db_utils import acquire_with_retry
@@ -24,6 +24,9 @@ from .link_expansion_retrieval import LinkExpansionRetriever
 from .tags import TagGroup, TagsMatch, build_tag_groups_where_clause, build_tags_where_clause_simple
 from .types import GraphRetrievalTimings, RetrievalResult
 
+if TYPE_CHECKING:
+    from ..query_analyzer import QueryAnalyzer
+
 logger = logging.getLogger(__name__)
 
 

hindsight-api-slim/hindsight_api/engine/search/tags.py

@@ -2,14 +2,18 @@
 Tags filtering utilities for retrieval.
 
 Provides SQL building functions for filtering memories by tags.
-Supports four matching modes via TagsMatch enum:
+Supports five matching modes via TagsMatch enum:
 - "any": OR matching, includes untagged memories (default, backward compatible)
 - "all": AND matching, includes untagged memories
 - "any_strict": OR matching, excludes untagged memories
 - "all_strict": AND matching, excludes untagged memories
+- "exact": set-equality matching, excludes untagged memories
 
 OR matching (any/any_strict): Memory matches if ANY of its tags overlap with request tags
 AND matching (all/all_strict): Memory matches if ALL request tags are present in its tags
+EXACT matching: Memory matches only if its tag set EQUALS the request tag set (order-
+    independent). Used for observation "scope" filtering, where each observation lives
+    under exactly one scope (its full tag set) and "scope [a]" must not match "[a, b]".
 """
 
 from __future__ import annotations
@@ -18,7 +22,7 @@ from typing import Annotated, Literal
 
 from pydantic import BaseModel, ConfigDict, Field
 
-TagsMatch = Literal["any", "all", "any_strict", "all_strict"]
+TagsMatch = Literal["any", "all", "any_strict", "all_strict", "exact"]
 
 
 def _parse_tags_match(match: TagsMatch) -> tuple[str, bool]:
@@ -38,6 +42,10 @@ def _parse_tags_match(match: TagsMatch) -> tuple[str, bool]:
         return "&&", False
     elif match == "all_strict":
         return "@>", False
+    elif match == "exact":
+        # Set equality is handled by the callers via `@> AND <@`; the operator
+        # here is unused. Untagged rows never equal a non-empty scope.
+        return "@>", False
     else:
         # Default to "any" behavior
         return "&&", True
@@ -78,6 +86,13 @@ def build_tags_where_clause(
         return "", [], param_offset
 
     column = f"{table_alias}tags" if table_alias else "tags"
+
+    if match == "exact":
+        # Set equality (order-independent): superset AND subset. Untagged rows
+        # (empty array) never satisfy `@>` of a non-empty scope, so they're excluded.
+        clause = f"AND ({column} @> ${param_offset} AND {column} <@ ${param_offset})"
+        return clause, [tags], param_offset + 1
+
     operator, include_untagged = _parse_tags_match(match)
 
     if include_untagged:
@@ -115,6 +130,12 @@ def build_tags_where_clause_simple(
         return ""
 
     column = f"{table_alias}tags" if table_alias else "tags"
+
+    if match == "exact":
+        # Set equality (order-independent): superset AND subset. Untagged rows
+        # (empty array) never satisfy `@>` of a non-empty scope, so they're excluded.
+        return f"AND ({column} @> ${param_num} AND {column} <@ ${param_num})"
+
     operator, include_untagged = _parse_tags_match(match)
 
     if include_untagged:
@@ -164,7 +185,11 @@ def filter_results_by_tags(
             # else: skip untagged
         else:
             result_tags_set = set(result_tags)
-            if is_any_match:
+            if match == "exact":
+                # Set equality: tag set must match the scope exactly
+                if result_tags_set == tags_set:
+                    filtered.append(result)
+            elif is_any_match:
                 # Any overlap
                 if result_tags_set & tags_set:
                     filtered.append(result)
@@ -241,6 +266,9 @@ def _build_group_clause(
     """
     if isinstance(group, TagGroupLeaf):
         column = f"{table_alias}tags" if table_alias else "tags"
+        if group.match == "exact":
+            clause = f"({column} @> ${param_offset} AND {column} <@ ${param_offset})"
+            return clause, [group.tags], param_offset + 1
         operator, include_untagged = _parse_tags_match(group.match)
         if include_untagged:
             clause = f"({column} IS NULL OR {column} = '{{}}' OR {column} {operator} ${param_offset})"
@@ -349,6 +377,8 @@ def _match_group(result: object, group: TagGroup) -> bool:
             return include_untagged
         else:
             result_tags_set = set(result_tags)
+            if group.match == "exact":
+                return result_tags_set == tags_set
             if is_any_match:
                 return bool(result_tags_set & tags_set)
             else:

hindsight-api-slim/hindsight_api/engine/storage/gcs.py

@@ -2,7 +2,7 @@
 
 import logging
 import os
-from datetime import datetime, timedelta, timezone
+from datetime import timedelta, timezone
 
 import obstore as obs
 from obstore.store import GCSStore

hindsight-api-slim/hindsight_api/engine/temporal_periods.py

@@ -0,0 +1,155 @@
+"""Explicit period extraction helpers for DateparserQueryAnalyzer.
+
+This module keeps the public period-extraction API and the non-Chinese period
+rules. Chinese rules live in chinese_temporal_periods.py because that rule set is
+substantially larger and has different boundary behavior from whitespace-based
+languages.
+"""
+
+import calendar
+import re
+import unicodedata
+from datetime import datetime, timedelta
+
+DateRange = tuple[datetime, datetime]
+
+
+class NoTemporalConstraintSentinel:
+    pass
+
+
+NO_TEMPORAL_CONSTRAINT = NoTemporalConstraintSentinel()
+
+__all__ = [
+    "NO_TEMPORAL_CONSTRAINT",
+    "extract_period",
+    "is_embedded_cjk_dateparser_match",
+]
+
+
+def _is_cjk_character(char: str) -> bool:
+    return "\u4e00" <= char <= "\u9fff"
+
+
+def is_embedded_cjk_dateparser_match(query: str, matched_text: str) -> bool:
+    from hindsight_api.engine.chinese_temporal_periods import (
+        is_embedded_cjk_dateparser_match as chinese_is_embedded_cjk_dateparser_match,
+    )
+
+    return chinese_is_embedded_cjk_dateparser_match(query, matched_text)
+
+
+def _constraint(start: datetime, end: datetime) -> DateRange:
+    return (
+        start.replace(hour=0, minute=0, second=0, microsecond=0),
+        end.replace(hour=23, minute=59, second=59, microsecond=999999),
+    )
+
+
+def _month_end(year: int, month: int) -> datetime:
+    return datetime(year, month, calendar.monthrange(year, month)[1])
+
+
+def _extract_non_chinese_period(query: str, reference_date: datetime) -> DateRange | None:
+    if re.search(r"\b(yesterday|ayer|ieri|hier|gestern)\b", query, re.IGNORECASE):
+        d = reference_date - timedelta(days=1)
+        return _constraint(d, d)
+
+    if re.search(r"\b(today|hoy|oggi|aujourd\'?hui|heute)\b", query, re.IGNORECASE):
+        return _constraint(reference_date, reference_date)
+
+    if re.search(r"\b(a\s+)?couple\s+(of\s+)?days?\s+ago\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=3), reference_date - timedelta(days=1))
+
+    if re.search(r"\b(a\s+)?few\s+days?\s+ago\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=5), reference_date - timedelta(days=2))
+
+    if re.search(r"\b(a\s+)?couple\s+(of\s+)?weeks?\s+ago\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(weeks=3), reference_date - timedelta(weeks=1))
+
+    if re.search(r"\b(a\s+)?few\s+weeks?\s+ago\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(weeks=5), reference_date - timedelta(weeks=2))
+
+    if re.search(r"\b(a\s+)?couple\s+(of\s+)?months?\s+ago\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=90), reference_date - timedelta(days=30))
+
+    if re.search(r"\b(a\s+)?few\s+months?\s+ago\b", query, re.IGNORECASE):
+        return _constraint(reference_date - timedelta(days=150), reference_date - timedelta(days=60))
+
+    if re.search(
+        r"\b(last\s+week|la\s+semana\s+pasada|la\s+settimana\s+scorsa|la\s+semaine\s+derni[eè]re|letzte\s+woche)\b",
+        query,
+        re.IGNORECASE,
+    ):
+        start = reference_date - timedelta(days=reference_date.weekday() + 7)
+        return _constraint(start, start + timedelta(days=6))
+
+    if re.search(
+        r"\b(last\s+month|el\s+mes\s+pasado|il\s+mese\s+scorso|le\s+mois\s+dernier|letzten?\s+monat)\b",
+        query,
+        re.IGNORECASE,
+    ):
+        first = reference_date.replace(day=1)
+        end = first - timedelta(days=1)
+        start = end.replace(day=1)
+        return _constraint(start, end)
+
+    if re.search(
+        r"\b(last\s+year|el\s+a[ñn]o\s+pasado|l\'anno\s+scorso|l\'ann[ée]e\s+derni[eè]re|letztes?\s+jahr)\b",
+        query,
+        re.IGNORECASE,
+    ):
+        year = reference_date.year - 1
+        return _constraint(datetime(year, 1, 1), datetime(year, 12, 31))
+
+    if re.search(
+        r"\b(last\s+weekend|el\s+fin\s+de\s+semana\s+pasado|lo\s+scorso\s+fine\s+settimana|le\s+week-?end\s+dernier|letztes?\s+wochenende)\b",
+        query,
+        re.IGNORECASE,
+    ):
+        days_since_sat = (reference_date.weekday() + 2) % 7
+        if days_since_sat == 0:
+            days_since_sat = 7
+        sat = reference_date - timedelta(days=days_since_sat)
+        return _constraint(sat, sat + timedelta(days=1))
+
+    month_patterns = {
+        "january|enero|gennaio|janvier|januar": 1,
+        "february|febrero|febbraio|f[ée]vrier|februar": 2,
+        "march|marzo|mars|m[äa]rz": 3,
+        "april|abril|aprile|avril": 4,
+        "may|mayo|maggio|mai": 5,
+        "june|junio|giugno|juin|juni": 6,
+        "july|julio|luglio|juillet|juli": 7,
+        "august|agosto|ao[uû]t": 8,
+        "september|septiembre|settembre|septembre": 9,
+        "october|octubre|ottobre|octobre|oktober": 10,
+        "november|noviembre|novembre": 11,
+        "december|diciembre|dicembre|d[ée]cembre|dezember": 12,
+    }
+    for pattern, month_num in month_patterns.items():
+        match = re.search(rf"\b({pattern})\s+(\d{{4}})\b", query, re.IGNORECASE)
+        if match:
+            year = int(match.group(2))
+            start = datetime(year, month_num, 1)
+            return _constraint(start, _month_end(year, month_num))
+
+    return None
+
+
+def extract_period(query: str, reference_date: datetime) -> DateRange | NoTemporalConstraintSentinel | None:
+    """Extract explicit period-based temporal expressions.
+
+    Non-Chinese rules are kept here. Chinese rules are delegated to
+    chinese_temporal_periods.py and are skipped entirely for non-CJK queries.
+    """
+    query = unicodedata.normalize("NFKC", query)
+
+    if any(_is_cjk_character(char) for char in query):
+        from hindsight_api.engine.chinese_temporal_periods import extract_chinese_period
+
+        chinese_result = extract_chinese_period(query, reference_date)
+        if chinese_result is not None:
+            return chinese_result
+
+    return _extract_non_chinese_period(query, reference_date)

hindsight-api-slim/hindsight_api/engine/transfer/export.py

@@ -80,6 +80,12 @@ _SKIP_TABLES = frozenset(
         "async_operations",  # in-flight ops; drain on the source before migrating
         "graph_maintenance_queue",  # transient work queue; regenerated on import
         "file_storage",  # raw uploads; documents.original_text is already carried
+        # Curation archive of retired facts — local operational state, not part of
+        # the live knowledge the export replays. Its rows mirror memory_units (stale
+        # embedding) and snapshot source-bank entity ids that the import re-resolves
+        # to fresh ids, so carrying them would only produce dangling associations.
+        # Revert anything worth keeping on the source before migrating.
+        "invalidated_memory_units",
     }
 )
 # Derived columns dropped from carried rows so the target regenerates them with

hindsight-api-slim/hindsight_api/engine/transfer/schema.py

@@ -22,7 +22,7 @@ from pydantic import BaseModel, Field
 # Bump when the archive layout changes in a backward-incompatible way.
 SCHEMA_VERSION = 1
 
-ObservationScopes = Literal["per_tag", "combined", "all_combinations"] | list[list[str]]
+ObservationScopes = Literal["per_tag", "combined", "all_combinations", "shared"] | list[list[str]]
 
 
 class TransferCausalRelation(BaseModel):

hindsight-api-slim/hindsight_api/extensions/__init__.py

@@ -16,11 +16,24 @@ with the system (e.g., running migrations for tenant schemas).
 """
 
 from hindsight_api.extensions.base import Extension
-from hindsight_api.extensions.builtin import ApiKeyTenantExtension, SupabaseTenantExtension
+from hindsight_api.extensions.builtin import (
+    ApiKeyTenantExtension,
+    MemoryDefenseRegexExtension,
+    SupabaseTenantExtension,
+)
 from hindsight_api.extensions.context import DefaultExtensionContext, ExtensionContext
 from hindsight_api.extensions.http import HttpExtension
 from hindsight_api.extensions.loader import load_extension
 from hindsight_api.extensions.mcp import MCPExtension
+from hindsight_api.extensions.memory_defense import (
+    DefenseAction,
+    DefenseDecision,
+    DefensePolicy,
+    MemoryDefenseExtension,
+    PolicyRule,
+    apply_redaction,
+    parse_policy,
+)
 from hindsight_api.extensions.operation_validator import (
     # Bank Management operations
     BankListContext,
@@ -104,4 +117,13 @@ __all__ = [
     "Tenant",
     "TenantContext",
     "TenantExtension",
+    # Memory Defense
+    "DefenseAction",
+    "DefenseDecision",
+    "DefensePolicy",
+    "MemoryDefenseExtension",
+    "MemoryDefenseRegexExtension",
+    "PolicyRule",
+    "apply_redaction",
+    "parse_policy",
 ]

hindsight-api-slim/hindsight_api/extensions/builtin/__init__.py

@@ -13,10 +13,12 @@ Example usage:
     HINDSIGHT_API_TENANT_EXTENSION=hindsight_api.extensions.builtin.supabase_tenant:SupabaseTenantExtension
 """
 
+from hindsight_api.extensions.builtin.memory_defense_regex import MemoryDefenseRegexExtension
 from hindsight_api.extensions.builtin.supabase_tenant import SupabaseTenantExtension
 from hindsight_api.extensions.builtin.tenant import ApiKeyTenantExtension
 
 __all__ = [
     "ApiKeyTenantExtension",
+    "MemoryDefenseRegexExtension",
     "SupabaseTenantExtension",
 ]

hindsight-api-slim/hindsight_api/extensions/builtin/memory_defense_regex.py

@@ -0,0 +1,56 @@
+"""Memory Defense (regex) — the default extension shipping with hindsight-api-slim.
+
+Scrubs known secret/PII patterns from retained content via the
+``sensitive_data`` detector. Matching is pure regex (see ``apply_redaction``):
+no LLM call, no external dependency. A ``sensitive_data`` rule may either
+``redact`` matches in place or ``block`` the item entirely.
+"""
+
+from __future__ import annotations
+
+import logging
+
+from hindsight_api.extensions.memory_defense import (
+    DefenseAction,
+    DefenseDecision,
+    DefensePolicy,
+    MemoryDefenseExtension,
+    apply_redaction,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class MemoryDefenseRegexExtension(MemoryDefenseExtension):
+    """Default Memory Defense — regex-based secret/PII redaction."""
+
+    async def screen(
+        self,
+        *,
+        policy: DefensePolicy,
+        bank_id: str,
+        document_id: str | None,
+        content: str,
+        tags: list[str],
+    ) -> DefenseDecision:
+        if not policy.enabled:
+            return DefenseDecision(action=DefenseAction.ALLOW)
+
+        # The regex extension only runs the sensitive_data detector. If the
+        # policy doesn't include a rule for it, there's nothing to do.
+        rule = next((r for r in policy.rules if r.on == "sensitive_data"), None)
+        if rule is None or rule.action is DefenseAction.ALLOW:
+            return DefenseDecision(action=DefenseAction.ALLOW)
+
+        result = apply_redaction(content)
+        if not result.matched_types:
+            return DefenseDecision(action=DefenseAction.ALLOW)
+
+        return DefenseDecision(
+            action=rule.action,
+            detector="sensitive_data",
+            message=f"Sensitive data pattern matched: {', '.join(result.matched_types)}",
+            redacted_content=result.content if rule.action is DefenseAction.REDACT else None,
+            matched_types=result.matched_types,
+            hits=result.hits,
+        )

hindsight-api-slim/hindsight_api/extensions/context.py

@@ -5,6 +5,7 @@ from typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
     from hindsight_api.engine.interface import MemoryEngineInterface
+    from hindsight_api.webhooks.manager import WebhookManager
 
 
 class ExtensionContext(ABC):
@@ -83,6 +84,8 @@ class DefaultExtensionContext(ExtensionContext):
         self,
         database_url: str,
         memory_engine: "MemoryEngineInterface | None" = None,
+        webhook_manager: "WebhookManager | None" = None,
+        current_schema: str | None = None,
     ):
         """
         Initialize the context.
@@ -90,9 +93,13 @@ class DefaultExtensionContext(ExtensionContext):
         Args:
             database_url: SQLAlchemy database URL for migrations.
             memory_engine: Optional MemoryEngine instance for memory operations.
+            webhook_manager: Optional WebhookManager for firing webhooks.
+            current_schema: Optional current schema name for tenant context.
         """
         self._database_url = database_url
         self._memory_engine = memory_engine
+        self.webhook_manager = webhook_manager
+        self.current_schema = current_schema
 
     async def run_migration(self, schema: str) -> None:
         """Run migrations for a specific schema."""

hindsight-api-slim/hindsight_api/extensions/memory_defense.py

@@ -0,0 +1,271 @@
+"""Memory Defense extension contract and shared policy types.
+
+Lives in extensions/ (not engine/) because it defines the public contract
+between the retain orchestrator and any installed Memory Defense extension —
+the same shape as TenantExtension and OperationValidatorExtension.
+
+api-slim ships the :class:`MemoryDefenseExtension` protocol and a regex default
+that scrubs known secret/PII patterns from retained content.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from enum import Enum
+
+from hindsight_api.extensions.base import Extension
+
+logger = logging.getLogger(__name__)
+
+
+class DefenseAction(str, Enum):
+    ALLOW = "allow"
+    REDACT = "redact"
+    BLOCK = "block"
+
+
+_VALID_ACTIONS = {a.value for a in DefenseAction}
+
+# ``policy.rules[*].on`` names a detector. The OSS extension only screens for
+# ``sensitive_data``; any other name is a silent no-op here and is dispatched
+# by whichever extension is loaded (e.g. hindsight-cloud screens cloud-only
+# detectors). The parser therefore does NOT validate ``on`` against a fixed
+# list — pinning the OSS roster to cloud's would force an OSS bump for every
+# new cloud detector just to avoid 422-ing a write it never interprets. We
+# only require ``on`` to be a non-empty string; entitlement and dispatch are
+# the loaded extension's ``screen()`` job.
+
+
+@dataclass(frozen=True)
+class PolicyRule:
+    on: str
+    action: DefenseAction
+
+
+@dataclass(frozen=True)
+class DefensePolicy:
+    enabled: bool = False
+    rules: tuple[PolicyRule, ...] = ()
+
+
+@dataclass
+class DefenseDecision:
+    action: DefenseAction
+    detector: str | None = None
+    message: str = ""
+    redacted_content: str | None = None
+    matched_types: list[str] = field(default_factory=list)
+    # Per-match fingerprinted previews. Each entry is
+    # ``{"detector": <pattern label>, "preview": <fingerprinted value>}``.
+    # The preview is *never* the raw value — see :func:`_fingerprint_value`.
+    # OSS populates this from ``apply_redaction``; downstream extensions
+    # populate it from their own detectors. Optional: empty when the
+    # match path didn't capture per-hit values.
+    hits: list[dict] = field(default_factory=list)
+
+
+@dataclass
+class RedactionResult:
+    content: str
+    matched_types: list[str]
+    # Same shape as ``DefenseDecision.hits`` — one entry per matched value
+    # (so a single content with two GitHub tokens produces two entries).
+    hits: list[dict] = field(default_factory=list)
+
+
+def _fingerprint_value(value: str) -> str:
+    """Return a redaction-identifiable preview of a matched value.
+
+    The preview keeps the prefix and a short suffix so a SIEM operator can
+    correlate against their credential inventory (the prefix names the
+    provider; the suffix disambiguates specific instances) without the raw
+    secret crossing the wire. Length-aware so short values don't accidentally
+    leak material:
+
+    - Length < 6:  redact entirely (return a fixed-length mask). Catches
+      noise like a single ``-----BEGIN...`` marker line.
+    - Length 6-15: keep the first 2 + last 2 around an ellipsis.
+    - Length > 15: keep the first 4 + last 4 around an ellipsis.
+
+    Examples::
+
+        _fingerprint_value("ghp_AAAA...AAAA" + "A" * 36)  -> "ghp_...AAAA"
+        _fingerprint_value("AKIA" + "B" * 16)              -> "AKIA...BBBB"
+        _fingerprint_value("123-45-6789")                  -> "12...89"
+        _fingerprint_value("abc")                          -> "[redacted]"
+    """
+    n = len(value)
+    if n < 6:
+        return "[redacted]"
+    if n <= 15:
+        return f"{value[:2]}...{value[-2:]}"
+    return f"{value[:4]}...{value[-4:]}"
+
+
+def parse_policy(raw: dict | None) -> DefensePolicy:
+    """Parse a raw bank-config dict into a frozen DefensePolicy.
+
+    Raises ValueError for a missing/empty ``on`` or an unknown action; the
+    HTTP layer converts those into a 422 response.
+    """
+    if raw is None:
+        return DefensePolicy()
+
+    rules: list[PolicyRule] = []
+    for item in raw.get("rules", []) or []:
+        on_raw = item.get("on")
+        if not isinstance(on_raw, str) or not on_raw:
+            raise ValueError(f"invalid on {on_raw!r}; must be a non-empty string")
+        action_raw = item.get("action")
+        if action_raw not in _VALID_ACTIONS:
+            raise ValueError(f"invalid action {action_raw!r}; must be one of {sorted(_VALID_ACTIONS)}")
+        rules.append(PolicyRule(on=on_raw, action=DefenseAction(action_raw)))
+
+    return DefensePolicy(
+        enabled=bool(raw.get("enabled", False)),
+        rules=tuple(rules),
+    )
+
+
+# Secret/PII redaction patterns.
+#
+# Scope: high-confidence patterns with unambiguous prefixes (low false-positive
+# rate). Context-dependent matches (e.g. Cohere/Mistral keys that only stand
+# out near surrounding "cohere"/"mistral" tokens) are NOT covered by pure
+# regex — operators who need that should layer a context-aware secret
+# scanner (detect-secrets, trufflehog) on top.
+#
+# Order matters: more-specific patterns first so broader ones don't consume
+# substrings partially. Example: `sk-ant-...` and `sk-proj-...` must run
+# before the generic `sk-...` pattern.
+_REDACTION_PATTERNS: list[tuple[str, str]] = [
+    # --- AI / LLM providers ---
+    ("anthropic_key", r"\bsk-ant-[A-Za-z0-9_-]{20,}\b"),
+    ("openai_project_key", r"\bsk-proj-[A-Za-z0-9_-]{48,}\b"),
+    ("openai_admin_key", r"\bsk-admin-[A-Za-z0-9_-]{40,}\b"),
+    ("openai_key", r"\bsk-[A-Za-z0-9_-]{20,}\b"),
+    ("google_api_key", r"\bAIza[0-9A-Za-z_-]{35}\b"),
+    ("google_oauth_token", r"\bya29\.[0-9A-Za-z_-]{20,}\b"),
+    ("xai_key", r"\bxai-[A-Za-z0-9]{40,}\b"),
+    ("groq_key", r"\bgsk_[A-Za-z0-9]{20,}\b"),
+    ("huggingface_token", r"\bhf_[A-Za-z0-9]{30,}\b"),
+    ("replicate_token", r"\br8_[A-Za-z0-9]{30,}\b"),
+    ("perplexity_key", r"\bpplx-[A-Za-z0-9]{40,}\b"),
+    ("databricks_token", r"\bdapi[A-Za-z0-9]{32}\b"),
+    # --- Cloud providers ---
+    ("aws_access_key", r"\bAKIA[0-9A-Z]{16}\b"),
+    ("aws_session_token", r"\bASIA[0-9A-Z]{16}\b"),
+    (
+        "aws_secret_key",
+        r"(?i)aws(.{0,20})?(secret|private)?[\s_-]?access[\s_-]?key[\s_-]?[:=][\s\"']*([A-Za-z0-9/+=]{40})",
+    ),
+    ("digitalocean_token", r"\bdop_v1_[a-f0-9]{64}\b"),
+    # --- Source control & CI ---
+    ("github_fg_pat", r"\bgithub_pat_[A-Za-z0-9_]{60,}\b"),
+    ("github_token", r"\bghp_[A-Za-z0-9]{36}\b"),
+    ("github_app_token", r"\bghs_[A-Za-z0-9]{36}\b"),
+    ("github_user_token", r"\bghu_[A-Za-z0-9]{36}\b"),
+    ("github_refresh", r"\bghr_[A-Za-z0-9]{36}\b"),
+    ("github_oauth", r"\bgho_[A-Za-z0-9]{36}\b"),
+    ("gitlab_pat", r"\bglpat-[A-Za-z0-9_-]{20,}\b"),
+    ("npm_token", r"\bnpm_[A-Za-z0-9]{30,}\b"),
+    ("pypi_token", r"\bpypi-AgEIcHlwaS5vcmc[A-Za-z0-9_-]{20,}\b"),
+    # --- Payment processors ---
+    ("stripe_secret", r"\bsk_(?:live|test)_[A-Za-z0-9]{20,}\b"),
+    ("stripe_restricted", r"\brk_(?:live|test)_[A-Za-z0-9]{20,}\b"),
+    ("square_token", r"\bsq0[a-z]{3}-[A-Za-z0-9_-]{22,}\b"),
+    ("braintree_token", r"\baccess_token\$production\$[a-z0-9]{16}\$[a-f0-9]{32}\b"),
+    # --- Communication / email ---
+    ("slack_token", r"\bxox[abpr]-[0-9A-Za-z-]{10,}\b"),
+    ("slack_webhook", r"https://hooks\.slack\.com/services/T[A-Za-z0-9_]{8,}/B[A-Za-z0-9_]{8,}/[A-Za-z0-9_]{20,}"),
+    ("twilio_api_key", r"\bSK[0-9a-fA-F]{32}\b"),
+    ("twilio_account_sid", r"\bAC[0-9a-fA-F]{32}\b"),
+    ("sendgrid_key", r"\bSG\.[A-Za-z0-9_-]{22}\.[A-Za-z0-9_-]{43}\b"),
+    ("mailgun_key", r"\bkey-[A-Za-z0-9]{32}\b"),
+    ("discord_bot", r"\b[MNO][A-Za-z0-9]{23}\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{27}\b"),
+    ("telegram_bot", r"\b[0-9]{8,10}:[A-Za-z0-9_-]{35}\b"),
+    # --- Commerce ---
+    ("shopify_token", r"\bshpat_[a-fA-F0-9]{32}\b"),
+    # --- Database connection strings (creds embedded in URL) ---
+    ("db_url_postgres", r"postgres(?:ql)?://[^\s:/@]+:[^\s/@]+@[^\s]+"),
+    ("db_url_mysql", r"mysql://[^\s:/@]+:[^\s/@]+@[^\s]+"),
+    ("db_url_mongodb", r"mongodb(?:\+srv)?://[^\s:/@]+:[^\s/@]+@[^\s]+"),
+    # --- Private keys & generic credentials ---
+    ("private_key_pem", r"-----BEGIN (?:RSA |EC |DSA |OPENSSH |PGP )?PRIVATE KEY( BLOCK)?-----"),
+    ("jwt", r"\beyJ[A-Za-z0-9_-]{10,}\.eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b"),
+    # --- PII (US-centric defaults; can be tuned per deployment) ---
+    # NOTE: credit_card regex is intentionally narrowed to 13-19 digits with
+    # exact separators to reduce false positives on long product IDs.
+    ("credit_card", r"\b(?:\d{4}[ -]?){3}\d{1,4}\b"),
+    ("ssn_us", r"\b\d{3}-\d{2}-\d{4}\b"),
+]
+_COMPILED_REDACTIONS: list[tuple[str, re.Pattern]] = [
+    (label, re.compile(pattern)) for label, pattern in _REDACTION_PATTERNS
+]
+
+
+def apply_redaction(content: str) -> RedactionResult:
+    """Scrub known secret/PII patterns from content with [REDACTED:type] markers.
+
+    Returns the (possibly unchanged) content alongside:
+      - ``matched_types``: pattern labels that matched (deduplicated, in
+        first-occurrence order). Empty when nothing matched.
+      - ``hits``: per-match fingerprinted previews — one entry per matched
+        substring (so two GitHub tokens in the same content produce two
+        entries). Each entry is ``{"detector": label, "preview": fingerprint}``
+        where ``preview`` is a length-aware redaction of the original value.
+        The raw secret never appears in ``hits``.
+
+    The two-pass shape (find matches first, then substitute) lets us capture
+    raw values for fingerprinting before they're replaced by ``[REDACTED:type]``
+    markers. A single-pass approach would lose the originals.
+    """
+    matched: list[str] = []
+    hits: list[dict] = []
+    for label, pattern in _COMPILED_REDACTIONS:
+        raw_hits = pattern.findall(content)
+        if not raw_hits:
+            continue
+        if label not in matched:
+            matched.append(label)
+        for raw in raw_hits:
+            # findall returns either a string or a tuple of capture groups
+            # depending on the pattern. The redaction-pattern catalog uses a
+            # mix; coerce to the matched substring as best we can.
+            if isinstance(raw, tuple):
+                # Pick the longest non-empty group as the canonical match.
+                non_empty = [g for g in raw if g]
+                raw_str = max(non_empty, key=len) if non_empty else ""
+            else:
+                raw_str = raw
+            if not raw_str:
+                continue
+            hits.append({"detector": label, "preview": _fingerprint_value(raw_str)})
+        content = pattern.sub(f"[REDACTED:{label}]", content)
+    return RedactionResult(content=content, matched_types=matched, hits=hits)
+
+
+class MemoryDefenseExtension(Extension, ABC):
+    """Abstract base for Memory Defense extensions.
+
+    Implementations decide whether to allow, redact, or block a given retain
+    item by inspecting its content against a per-bank policy. The orchestrator
+    applies the returned decision (redacts content / drops blocked items) and
+    fires a webhook for non-allow decisions when one is configured.
+    """
+
+    @abstractmethod
+    async def screen(
+        self,
+        *,
+        policy: DefensePolicy,
+        bank_id: str,
+        document_id: str | None,
+        content: str,
+        tags: list[str],
+    ) -> DefenseDecision:
+        """Inspect content under the given policy and return a decision."""
+        ...

hindsight-api-slim/hindsight_api/extensions/operation_validator.py

@@ -3,7 +3,7 @@
 from abc import ABC, abstractmethod
 from dataclasses import dataclass, field
 from datetime import datetime
-from typing import TYPE_CHECKING, Any
+from typing import TYPE_CHECKING
 
 from hindsight_api.extensions.base import Extension
 

hindsight-api-slim/hindsight_api/mcp_tools.py

@@ -50,6 +50,8 @@ _ALL_TOOLS: frozenset[str] = frozenset(
         "delete_directive",
         "list_memories",
         "get_memory",
+        "update_memory",
+        "invalidate_memory",
         "list_documents",
         "get_document",
         "delete_document",
@@ -228,6 +230,8 @@ def register_mcp_tools(
         "delete_directive",
         "list_memories",
         "get_memory",
+        "update_memory",
+        "invalidate_memory",
         "list_documents",
         "get_document",
         "delete_document",
@@ -299,6 +303,12 @@ def register_mcp_tools(
     if "get_memory" in tools_to_register:
         _register_get_memory(mcp, memory, config)
 
+    if "update_memory" in tools_to_register:
+        _register_update_memory(mcp, memory, config)
+
+    if "invalidate_memory" in tools_to_register:
+        _register_invalidate_memory(mcp, memory, config)
+
     # Document tools
     if "list_documents" in tools_to_register:
         _register_list_documents(mcp, memory, config)
@@ -2293,6 +2303,206 @@ def _register_get_memory(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsCon
                 return {"error": str(e)}
 
 
+def _register_update_memory(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig) -> None:
+    """Register the update_memory (edit) tool."""
+
+    _EDIT_DOC = """
+            Edit a memory unit to correct what was extracted.
+
+            Pass any of text / context / occurred_start / occurred_end / fact_type /
+            entities. For context and the dates, "" clears the field and omitting it
+            leaves it unchanged; entities replaces the fact's entity set ([] detaches
+            all). The memory is re-embedded and its derived observations, links, and
+            graph are recomputed automatically.
+
+            Only raw world/experience facts can be edited; observations are derived.
+            To retire or restore a fact, use invalidate_memory instead.
+    """
+
+    if config.include_bank_id_param:
+
+        @mcp.tool(description=_EDIT_DOC)
+        async def update_memory(
+            memory_id: str,
+            text: str | None = None,
+            context: str | None = None,
+            occurred_start: str | None = None,
+            occurred_end: str | None = None,
+            fact_type: str | None = None,
+            entities: list[str] | None = None,
+            bank_id: str | None = None,
+        ) -> str:
+            """
+            Args:
+                memory_id: The ID of the memory unit to edit.
+                bank_id: Optional bank (defaults to session bank). Use for cross-bank operations.
+            """
+            try:
+                target_bank = bank_id or config.bank_id_resolver()
+                if target_bank is None:
+                    return '{"error": "No bank_id configured"}'
+
+                result = await memory.update_memory_unit(
+                    target_bank,
+                    memory_id,
+                    text=text,
+                    context=context,
+                    occurred_start=occurred_start,
+                    occurred_end=occurred_end,
+                    new_fact_type=fact_type,
+                    entities=entities,
+                    request_context=_get_request_context(config),
+                )
+                if result is None:
+                    return json.dumps({"error": f"Memory '{memory_id}' not found"})
+                return json.dumps(result, indent=2, default=str)
+            except OperationValidationError as e:
+                logger.warning(f"Operation rejected: {e}")
+                return json.dumps({"error": str(e)})
+            except ValueError as e:
+                return json.dumps({"error": str(e)})
+            except Exception as e:
+                logger.error(f"Error updating memory: {e}", exc_info=True)
+                return f'{{"error": "{e}"}}'
+
+    else:
+
+        @mcp.tool(description=_EDIT_DOC)
+        async def update_memory(
+            memory_id: str,
+            text: str | None = None,
+            context: str | None = None,
+            occurred_start: str | None = None,
+            occurred_end: str | None = None,
+            fact_type: str | None = None,
+            entities: list[str] | None = None,
+        ) -> dict:
+            """
+            Args:
+                memory_id: The ID of the memory unit to edit.
+            """
+            try:
+                target_bank = config.bank_id_resolver()
+                if target_bank is None:
+                    return {"error": "No bank_id configured"}
+
+                result = await memory.update_memory_unit(
+                    target_bank,
+                    memory_id,
+                    text=text,
+                    context=context,
+                    occurred_start=occurred_start,
+                    occurred_end=occurred_end,
+                    new_fact_type=fact_type,
+                    entities=entities,
+                    request_context=_get_request_context(config),
+                )
+                if result is None:
+                    return {"error": f"Memory '{memory_id}' not found"}
+                return result
+            except OperationValidationError as e:
+                logger.warning(f"Operation rejected: {e}")
+                return {"error": str(e)}
+            except ValueError as e:
+                return {"error": str(e)}
+            except Exception as e:
+                logger.error(f"Error updating memory: {e}", exc_info=True)
+                return {"error": str(e)}
+
+
+def _register_invalidate_memory(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsConfig) -> None:
+    """Register the invalidate_memory (retire / restore) tool."""
+
+    _INVALIDATE_DOC = """
+            Soft-retire a memory unit (or restore a previously retired one).
+
+            Invalidating moves the fact out of the active set: it's excluded from
+            recall, consolidation, and the knowledge graph, its links are pruned, and
+            its derived observations are recomputed without it — but it's kept for
+            audit and is fully reversible. Pass restore=True to bring it back.
+
+            Only raw world/experience facts can be invalidated; observations are derived.
+    """
+
+    if config.include_bank_id_param:
+
+        @mcp.tool(description=_INVALIDATE_DOC)
+        async def invalidate_memory(
+            memory_id: str,
+            reason: str | None = None,
+            restore: bool = False,
+            bank_id: str | None = None,
+        ) -> str:
+            """
+            Args:
+                memory_id: The ID of the memory unit to retire (or restore).
+                reason: Optional free-text reason recorded when invalidating.
+                restore: Set True to restore a previously invalidated fact.
+                bank_id: Optional bank (defaults to session bank). Use for cross-bank operations.
+            """
+            try:
+                target_bank = bank_id or config.bank_id_resolver()
+                if target_bank is None:
+                    return '{"error": "No bank_id configured"}'
+
+                result = await memory.update_memory_unit(
+                    target_bank,
+                    memory_id,
+                    state="valid" if restore else "invalidated",
+                    reason=reason,
+                    request_context=_get_request_context(config),
+                )
+                if result is None:
+                    return json.dumps({"error": f"Memory '{memory_id}' not found"})
+                return json.dumps(result, indent=2, default=str)
+            except OperationValidationError as e:
+                logger.warning(f"Operation rejected: {e}")
+                return json.dumps({"error": str(e)})
+            except ValueError as e:
+                return json.dumps({"error": str(e)})
+            except Exception as e:
+                logger.error(f"Error invalidating memory: {e}", exc_info=True)
+                return f'{{"error": "{e}"}}'
+
+    else:
+
+        @mcp.tool(description=_INVALIDATE_DOC)
+        async def invalidate_memory(
+            memory_id: str,
+            reason: str | None = None,
+            restore: bool = False,
+        ) -> dict:
+            """
+            Args:
+                memory_id: The ID of the memory unit to retire (or restore).
+                reason: Optional free-text reason recorded when invalidating.
+                restore: Set True to restore a previously invalidated fact.
+            """
+            try:
+                target_bank = config.bank_id_resolver()
+                if target_bank is None:
+                    return {"error": "No bank_id configured"}
+
+                result = await memory.update_memory_unit(
+                    target_bank,
+                    memory_id,
+                    state="valid" if restore else "invalidated",
+                    reason=reason,
+                    request_context=_get_request_context(config),
+                )
+                if result is None:
+                    return {"error": f"Memory '{memory_id}' not found"}
+                return result
+            except OperationValidationError as e:
+                logger.warning(f"Operation rejected: {e}")
+                return {"error": str(e)}
+            except ValueError as e:
+                return {"error": str(e)}
+            except Exception as e:
+                logger.error(f"Error invalidating memory: {e}", exc_info=True)
+                return {"error": str(e)}
+
+
 # =========================================================================
 # DOCUMENT TOOLS
 # =========================================================================
@@ -2981,7 +3191,8 @@ def _register_update_bank(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsCo
                     - retain_mission: Steers what gets extracted during retain().
                     - retain_extraction_mode: 'concise' (default), 'verbose', or 'custom'.
                     - retain_custom_instructions: Custom extraction prompt (active when mode is 'custom').
-                    - retain_chunk_size: Maximum token size for each content chunk.
+                    - retain_chunk_size: Target maximum characters for each content chunk.
+                    - retain_structured_chunk_size: Maximum characters for a single JSONL line or conversation turn to keep whole.
                     - retain_chunk_batch_size: Number of chunks to process in parallel.
                     - enable_observations: Toggle observation consolidation after retain().
                     - observations_mission: Controls observation synthesis rules.
@@ -3040,7 +3251,8 @@ def _register_update_bank(mcp: FastMCP, memory: MemoryEngine, config: MCPToolsCo
                     - retain_mission: Steers what gets extracted during retain().
                     - retain_extraction_mode: 'concise' (default), 'verbose', or 'custom'.
                     - retain_custom_instructions: Custom extraction prompt (active when mode is 'custom').
-                    - retain_chunk_size: Maximum token size for each content chunk.
+                    - retain_chunk_size: Target maximum characters for each content chunk.
+                    - retain_structured_chunk_size: Maximum characters for a single JSONL line or conversation turn to keep whole.
                     - retain_chunk_batch_size: Number of chunks to process in parallel.
                     - enable_observations: Toggle observation consolidation after retain().
                     - observations_mission: Controls observation synthesis rules.

hindsight-api-slim/hindsight_api/metrics.py

@@ -39,6 +39,32 @@ def _get_tenant() -> str:
     return get_current_schema()
 
 
+def _is_client_cancellation(exc: BaseException) -> bool:
+    """Whether *exc* is a client-disconnect cancellation rather than a failure.
+
+    An abandoned recall/reflect raises OperationCancelledError (issue #2122);
+    the HTTP layer re-raises it as ``HTTPException(499) from exc`` (see
+    api/http.py run_cancellable_on_disconnect). The exception itself, or any
+    link in its ``__cause__`` chain, being an OperationCancelledError marks it
+    as a cancellation. Matching on the cause chain rather than a bare status
+    code avoids misclassifying an unrelated 499 as a cancellation. Per the
+    engine contract a cancellation is "not a failure to retry or report"
+    (cancellation.OperationCancelledError), so it must not be counted against
+    ``hindsight.operation.total``.
+    """
+    # Imported lazily to avoid import-time coupling (cf. _get_tenant above).
+    from hindsight_api.cancellation import OperationCancelledError
+
+    cause: BaseException | None = exc
+    seen: set[int] = set()  # guard against a cyclic __cause__ chain
+    while cause is not None and id(cause) not in seen:
+        if isinstance(cause, OperationCancelledError):
+            return True
+        seen.add(id(cause))
+        cause = cause.__cause__
+    return False
+
+
 # Custom bucket boundaries for operation duration (in seconds)
 # Fine granularity in 0-30s range where most operations complete
 DURATION_BUCKETS = (0.1, 0.25, 0.5, 0.75, 1.0, 2.0, 3.0, 5.0, 7.5, 10.0, 15.0, 20.0, 30.0, 60.0, 120.0)
@@ -373,20 +399,31 @@ class MetricsCollector(MetricsCollectorBase):
             attributes["max_tokens"] = str(max_tokens)
 
         success = True
+        cancelled = False
         try:
             yield
-        except Exception:
-            success = False
+        except Exception as exc:
+            # A client disconnect cancels the operation cooperatively (#2122),
+            # raised as OperationCancelledError and re-raised by the HTTP layer
+            # as HTTPException(499) from it. An abandoned request is neither a
+            # success nor a failure, so it is excluded from the metric entirely
+            # rather than inflating either the failure or the success rate on
+            # hindsight.operation.total.
+            if _is_client_cancellation(exc):
+                cancelled = True
+            else:
+                success = False
             raise
         finally:
-            duration = time.time() - start_time
-            attributes["success"] = str(success).lower()
+            if not cancelled:
+                duration = time.time() - start_time
+                attributes["success"] = str(success).lower()
 
-            # Record duration
-            self.operation_duration.record(duration, attributes)
+                # Record duration
+                self.operation_duration.record(duration, attributes)
 
-            # Record operation count
-            self.operation_total.add(1, attributes)
+                # Record operation count
+                self.operation_total.add(1, attributes)
 
     def record_llm_call(
         self,

hindsight-api-slim/hindsight_api/migrations.py

@@ -25,7 +25,9 @@ from pathlib import Path
 from alembic import command
 from alembic.config import Config
 from alembic.script.revision import ResolutionError
+from alembic.util.exc import CommandError
 from sqlalchemy import Connection, create_engine, text
+from sqlalchemy.pool import NullPool
 
 from ._pg_search import normalize_pg_search_tokenizer, pg_search_bm25_columns
 from ._vector_index import (
@@ -131,7 +133,12 @@ def _run_migrations_internal(database_url: str, script_location: str, schema: st
     try:
         with _alembic_lock:
             command.upgrade(alembic_cfg, "heads")
-    except ResolutionError as e:
+    except (ResolutionError, CommandError) as e:
+        # command.upgrade() wraps ResolutionError in CommandError via
+        # ScriptDirectory._catch_revision_errors, so the wrapped form is what
+        # actually reaches us; re-raise CommandErrors with any other cause.
+        if isinstance(e, CommandError) and not isinstance(e.__cause__, ResolutionError):
+            raise
         # This happens during rolling deployments when a newer version of the code
         # has already run migrations, and this older replica doesn't have the new
         # migration files. The database is already at a newer revision than we know.
@@ -241,7 +248,14 @@ def run_migrations(
         #   2. After acquiring the lock, COMMIT the transaction on the advisory-lock
         #      connection itself before running migrations.  pg_advisory_lock is
         #      session-level, so the lock survives the COMMIT.
-        engine = create_engine(migration_url)
+        # NullPool: do not retain the connection in a pool after the migration.
+        # Each schema migration opens a few short-lived engines (here plus the
+        # ensure_* steps); with the default QueuePool those connections linger
+        # until GC, and running many schemas in parallel (migration_concurrency)
+        # multiplies that footprint and exhausts max_connections — observed as
+        # "FATAL: sorry, too many clients already" sweeping 20k schemas at
+        # concurrency 12. NullPool closes the connection on return.
+        engine = create_engine(migration_url, poolclass=NullPool)
         with engine.connect() as conn:
             logger.debug(f"Acquiring migration advisory lock for schema '{schema_name}' (id={lock_id})...")
             while True:
@@ -394,7 +408,7 @@ def check_migration_status(
             return None, None
 
         # Get current revision from database
-        engine = create_engine(to_libpq_url(database_url))
+        engine = create_engine(to_libpq_url(database_url), poolclass=NullPool)
         with engine.connect() as connection:
             context = MigrationContext.configure(connection)
             current_rev = context.get_current_revision()
@@ -567,7 +581,7 @@ def ensure_embedding_dimension(
     """
     schema_name = schema or "public"
 
-    engine = create_engine(to_libpq_url(database_url))
+    engine = create_engine(to_libpq_url(database_url), poolclass=NullPool)
     with engine.connect() as conn:
         # Check if memory_units table exists (proxy for schema being initialized)
         table_exists = conn.execute(
@@ -590,6 +604,10 @@ def ensure_embedding_dimension(
 
         _migrate_table_embedding_dimension(conn, schema_name, "memory_units", required_dimension, vector_ext)
         _migrate_table_embedding_dimension(conn, schema_name, "mental_models", required_dimension, vector_ext)
+        # NOTE: invalidated_memory_units is deliberately omitted. The curation archive has no
+        # embedding column at all (dropped in migration d4f6a8c2e1b3) — invalidate stores no
+        # embedding and revert recomputes one — so there is no archive vector to re-dimension
+        # and a model switch can't trip a dimension mismatch there (#2209).
 
 
 def ensure_vector_extension(
@@ -616,7 +634,7 @@ def ensure_vector_extension(
     """
     schema_name = schema or "public"
 
-    engine = create_engine(to_libpq_url(database_url))
+    engine = create_engine(to_libpq_url(database_url), poolclass=NullPool)
     with engine.connect() as conn:
         # Detect which vector extension should be used
         target_ext = _detect_vector_extension(conn, vector_extension)
@@ -830,7 +848,7 @@ def ensure_text_search_extension(
     schema_name = schema or "public"
     pg_search_tokenizer = normalize_pg_search_tokenizer(pg_search_tokenizer)
 
-    engine = create_engine(to_libpq_url(database_url))
+    engine = create_engine(to_libpq_url(database_url), poolclass=NullPool)
     with engine.connect() as conn:
         # Tables with search_vector columns to check
         tables_to_check = [
@@ -1123,3 +1141,129 @@ def ensure_text_search_extension(
 
         conn.commit()
         logger.info(f"Successfully migrated text search to {text_search_extension}")
+
+
+def _migrate_one_schema_pg(
+    database_url: str,
+    schema: str,
+    *,
+    migration_database_url: str | None,
+    embedding_dimension: int | None,
+    vector_extension: str,
+    text_search_extension: str,
+    pg_search_tokenizer: str | None,
+    ensure_extensions: bool,
+) -> str:
+    """Run migrations + post-migration extension setup for a SINGLE PG schema.
+
+    Module-level (not a closure) so it is picklable and can run inside a
+    ``ProcessPoolExecutor`` worker. The steps run strictly in order — this is
+    the per-tenant sequential unit; parallelism happens only *across* schemas.
+    Returns the schema name on success; raises on the first failing step so the
+    caller can attribute the failure back to this schema.
+    """
+    run_migrations(database_url, schema=schema, migration_database_url=migration_database_url)
+    if embedding_dimension is not None:
+        ensure_embedding_dimension(
+            database_url,
+            embedding_dimension,
+            schema=schema,
+            vector_extension=vector_extension,
+        )
+    if ensure_extensions:
+        ensure_vector_extension(database_url, vector_extension=vector_extension, schema=schema)
+        ensure_text_search_extension(
+            database_url,
+            text_search_extension=text_search_extension,
+            schema=schema,
+            pg_search_tokenizer=pg_search_tokenizer,
+        )
+    return schema
+
+
+def _make_migration_executor(max_workers: int):
+    """Build the executor that runs per-schema migrations in parallel.
+
+    Each schema must run in its OWN process — Alembic's ``command.upgrade()``
+    uses non-thread-safe module globals (serialized in-process by
+    ``_alembic_lock``), so a thread pool would not actually run two upgrades at
+    once. ``spawn`` gives every worker a clean interpreter on all platforms,
+    avoiding the fork-of-a-multithreaded-process deadlock hazard (the API server
+    holds threads/pools when migrations run on startup).
+
+    Factored out so tests can substitute an in-process executor.
+    """
+    import multiprocessing
+    from concurrent.futures import ProcessPoolExecutor
+
+    return ProcessPoolExecutor(max_workers=max_workers, mp_context=multiprocessing.get_context("spawn"))
+
+
+def run_migrations_for_schemas(
+    database_url: str,
+    schemas: list[str],
+    *,
+    concurrency: int = 1,
+    migration_database_url: str | None = None,
+    embedding_dimension: int | None = None,
+    vector_extension: str = "pgvector",
+    text_search_extension: str = "native",
+    pg_search_tokenizer: str | None = None,
+    ensure_extensions: bool = True,
+) -> None:
+    """Run PostgreSQL migrations for many schemas, up to ``concurrency`` at once.
+
+    Within a schema the work is always sequential (migrate → embedding dim →
+    vector ext → text-search ext). Across schemas, when ``concurrency > 1`` each
+    schema is migrated in its OWN process: Alembic's ``command.upgrade()`` relies
+    on non-thread-safe module-level globals (serialized in-process by
+    ``_alembic_lock``), so threads would gain nothing — separate interpreters
+    each get a clean Alembic context. Per-schema advisory locks
+    (``_get_schema_lock_id``) keep concurrent processes from colliding on the
+    same schema across replicas.
+
+    ``database_url`` must already be resolved (e.g. an embedded ``pg0`` instance
+    started in the parent) — workers receive it verbatim and only connect.
+
+    Failures are collected per schema and re-raised together so one bad tenant
+    does not hide the status of the others.
+    """
+    if not schemas:
+        return
+
+    worker_kwargs = dict(
+        migration_database_url=migration_database_url,
+        embedding_dimension=embedding_dimension,
+        vector_extension=vector_extension,
+        text_search_extension=text_search_extension,
+        pg_search_tokenizer=pg_search_tokenizer,
+        ensure_extensions=ensure_extensions,
+    )
+
+    effective = max(1, min(concurrency, len(schemas)))
+    if effective == 1:
+        # Inline, in-process — no subprocess overhead for the common single
+        # tenant / sequential case (and keeps embedded pg0 dev simple).
+        for schema in schemas:
+            _migrate_one_schema_pg(database_url, schema, **worker_kwargs)
+        return
+
+    logger.info("Migrating %d schema(s) with concurrency=%d", len(schemas), effective)
+    errors: dict[str, BaseException] = {}
+    with _make_migration_executor(effective) as executor:
+        futures = {
+            executor.submit(_migrate_one_schema_pg, database_url, schema, **worker_kwargs): schema for schema in schemas
+        }
+        for future in futures:
+            schema = futures[future]
+            try:
+                future.result()
+            except Exception as exc:  # noqa: BLE001 — aggregate per-schema, re-raise below
+                errors[schema] = exc
+                logger.error("Migration failed for schema '%s': %s", schema, exc)
+
+    if errors:
+        failed = ", ".join(sorted(errors))
+        raise RuntimeError(
+            f"Database migrations failed for {len(errors)} of {len(schemas)} schema(s): {failed}"
+        ) from next(iter(errors.values()))

hindsight-api-slim/hindsight_api/models.py

@@ -4,8 +4,12 @@ SQLAlchemy models for the memory system.
 
 from dataclasses import dataclass
 from datetime import datetime
+from typing import TYPE_CHECKING
 from uuid import UUID as PyUUID
 
+if TYPE_CHECKING:
+    from .cancellation import CancellationToken
+
 
 @dataclass
 class RequestContext:
@@ -30,6 +34,21 @@ class RequestContext:
     # validators that want exponential backoff on repeated failures (e.g.
     # "defer for 2^retry_count minutes") without querying the DB themselves.
     retry_count: int = 0
+    # Cooperative cancellation signal for long-running operations. The HTTP
+    # layer sets this to a token that fires when the client disconnects; the
+    # engine checks it at stage boundaries and aborts abandoned work so it stops
+    # consuming CPU/DB resources (issue #2122). None means "never cancelled" —
+    # every checkpoint is a no-op.
+    cancellation: "CancellationToken | None" = None
+
+    def raise_if_cancelled(self) -> None:
+        """Abort the current operation if its cancellation token has fired.
+
+        A no-op when no token is attached, so engine code can call it at every
+        stage boundary without caring whether the caller opted into cancellation.
+        """
+        if self.cancellation is not None:
+            self.cancellation.raise_if_cancelled()
 
 
 from pgvector.sqlalchemy import Vector

hindsight-api-slim/hindsight_api/webhooks/__init__.py

@@ -1,7 +1,15 @@
 """Webhook system for Hindsight API event notifications."""
 
 from .manager import WebhookManager
-from .models import ConsolidationEventData, RetainEventData, WebhookConfig, WebhookEvent, WebhookEventType
+from .models import (
+    ConsolidationEventData,
+    MemoryDefenseEventData,
+    MemoryDefenseHit,
+    RetainEventData,
+    WebhookConfig,
+    WebhookEvent,
+    WebhookEventType,
+)
 
 __all__ = [
     "WebhookManager",
@@ -9,5 +17,7 @@ __all__ = [
     "WebhookEvent",
     "WebhookEventType",
     "ConsolidationEventData",
+    "MemoryDefenseEventData",
+    "MemoryDefenseHit",
     "RetainEventData",
 ]

hindsight-api-slim/hindsight_api/webhooks/manager.py

@@ -70,7 +70,10 @@ class WebhookManager:
         webhook_table = _fq_table("webhooks", schema)
         ops_table = _fq_table("async_operations", schema)
         now = datetime.now(timezone.utc)
-        payload_str = event.model_dump_json()
+        # Drop null fields so receivers don't see promised-but-unfilled keys.
+        # OSS leaves SIEM-enrichment fields (severity, api_key_name, etc.) None
+        # because it doesn't have the data; cloud populates them when it does.
+        payload_str = event.model_dump_json(exclude_none=True)
 
         try:
             async with self._backend.acquire() as conn:
@@ -150,7 +153,10 @@ class WebhookManager:
         webhook_table = _fq_table("webhooks", schema)
         ops_table = _fq_table("async_operations", schema)
         now = datetime.now(timezone.utc)
-        payload_str = event.model_dump_json()
+        # Drop null fields so receivers don't see promised-but-unfilled keys.
+        # OSS leaves SIEM-enrichment fields (severity, api_key_name, etc.) None
+        # because it doesn't have the data; cloud populates them when it does.
+        payload_str = event.model_dump_json(exclude_none=True)
 
         try:
             rows = await self._backend.ops.get_webhooks_for_dispatch(

hindsight-api-slim/hindsight_api/webhooks/models.py

@@ -9,6 +9,7 @@ from pydantic import BaseModel, Field
 class WebhookEventType(StrEnum):
     CONSOLIDATION_COMPLETED = "consolidation.completed"
     RETAIN_COMPLETED = "retain.completed"
+    MEMORY_DEFENSE_TRIGGERED = "memory_defense.triggered"
 
 
 class ConsolidationEventData(BaseModel):
@@ -23,13 +24,52 @@ class RetainEventData(BaseModel):
     tags: list[str] | None = None
 
 
+class MemoryDefenseHit(BaseModel):
+    """A single secret match inside a non-allow decision.
+
+    ``preview`` is a fingerprinted, redaction-identifiable rendering of the
+    matched value (e.g. ``ghp_AAAA...BBBB``) so SIEM operators can correlate
+    against their credential inventory WITHOUT the raw secret crossing the
+    network. Implementations must never put the raw value here.
+    """
+
+    detector: str  # the inner detector that matched (e.g. "GitHub Token")
+    preview: str  # fingerprinted value, never the raw secret
+
+
+class MemoryDefenseEventData(BaseModel):
+    """Payload for a memory_defense.triggered event (one item, one non-allow decision).
+
+    The four base fields (``action``/``detector``/``document_id``/``message``)
+    plus ``matched_types`` are populated by every implementation including OSS's
+    built-in regex defense. The remaining fields are optional SIEM-enrichment
+    surfaces that downstream extensions (e.g. hindsight-cloud) populate when
+    they have richer per-decision context — severity classification, the API
+    key that submitted the retain, fingerprinted hit previews for SIEM
+    correlation, and pointers into the audit trail. OSS leaves them ``None``;
+    receivers should treat absence as "not provided" rather than "no match".
+    """
+
+    action: str  # "redact" or "block"
+    detector: str | None = None  # e.g. "sensitive_data"
+    document_id: str | None = None
+    matched_types: list[str] | None = None  # redaction pattern labels that fired
+    message: str | None = None
+    # --- Optional SIEM enrichment (populated by extensions, not OSS) ---
+    severity: str | None = None  # "low" / "medium" / "high" / "critical"
+    api_key_name: str | None = None  # human-readable name of the submitting API key
+    hits: list[MemoryDefenseHit] | None = None  # per-match fingerprints for correlation
+    memory_unit_id: str | None = None  # drill-down pointer (when the decision was REDACT)
+    receipt_uri: str | None = None  # storage pointer for the audit trail entry
+
+
 class WebhookEvent(BaseModel):
     event: WebhookEventType
     bank_id: str
     operation_id: str
-    status: str  # "completed" or "failed"
+    status: str  # "completed"/"failed" for retain/consolidation; the action ("redact"/"block") for memory_defense
     timestamp: datetime
-    data: ConsolidationEventData | RetainEventData
+    data: ConsolidationEventData | RetainEventData | MemoryDefenseEventData
 
 
 class WebhookHttpConfig(BaseModel):

hindsight-api-slim/hindsight_api/worker/poller.py

@@ -16,7 +16,7 @@ import time
 import traceback
 from collections import Counter
 from collections.abc import Awaitable, Callable, Iterable
-from dataclasses import dataclass, field
+from dataclasses import dataclass
 from typing import TYPE_CHECKING, Any
 
 from ..engine.schema import fq_table_explicit as fq_table
@@ -226,38 +226,23 @@ class WorkerPoller:
         """
         async with self._backend.acquire() as conn:
             if await self._optional_routines.is_installed(conn, "schemas_with_pending_work"):
+                # The routine IS the authority on where work exists: every schema
+                # it returns is claimable, and every schema it does NOT return is
+                # treated as having nothing to do this cycle. That is the entire
+                # point of installing it — one round-trip replaces N per-schema
+                # EXISTS probes. We deliberately do NOT re-verify the omitted
+                # schemas with a per-schema scan: that re-runs the exact queries
+                # the routine exists to avoid, on every idle poll, silently
+                # negating the optimisation.
+                #
+                # Because the result is trusted wholesale, the routine is only
+                # appropriate for multi-tenant deployments. A single-schema
+                # (default/public only) install should NOT create it and instead
+                # falls through to the per-schema path below — a single cheap
+                # EXISTS check that cannot starve. See
+                # ``hindsight_api.engine.db.optional_routines``.
                 rows = await conn.fetch("SELECT * FROM public.schemas_with_pending_work()")
-                routine_active = {self._normalize_poll_schema(r[0]) for r in rows}
-                known_schemas = set(schemas)
-                active = routine_active & known_schemas
-                unknown = routine_active - known_schemas
-                if unknown:
-                    logger.warning(
-                        "Optional PG routine public.schemas_with_pending_work() returned schema(s) "
-                        "not present in tenant discovery: %s",
-                        sorted(str(s) for s in unknown),
-                    )
-
-                # The optional routine returns PostgreSQL schema names, but the poller uses
-                # None for the default schema. Older operator-supplied implementations also
-                # commonly scan tenant_% only; when the default schema is in scope but absent
-                # from the routine result, verify via the fully-correct per-schema fallback so
-                # public single-tenant deployments cannot silently starve.
-                should_verify_with_fallback = (None in known_schemas and None not in active) or (
-                    bool(routine_active) and not active
-                )
-                if not should_verify_with_fallback:
-                    return active
-
-                fallback_active = await self._scan_active_schemas_by_exists(conn, schemas)
-                missed = fallback_active - active
-                if missed:
-                    logger.warning(
-                        "Optional PG routine public.schemas_with_pending_work() missed claimable schema(s) %s; "
-                        "using per-schema fallback for this poll",
-                        sorted(str(s) for s in missed),
-                    )
-                return fallback_active
+                return {self._normalize_poll_schema(r[0]) for r in rows}
 
             return await self._scan_active_schemas_by_exists(conn, schemas)
 
@@ -824,7 +809,6 @@ class WorkerPoller:
             recovered = 0
             for row in rows:
                 operation_id = str(row["operation_id"])
-                task_payload = row["task_payload"]
                 result_metadata = row["result_metadata"]
 
                 # Parse metadata
@@ -838,12 +822,6 @@ class WorkerPoller:
                     f"Recovering batch operation: operation_id={operation_id}, batch_id={batch_id}, provider={batch_provider}"
                 )
 
-                # Parse task_payload
-                if isinstance(task_payload, str):
-                    task_dict = json.loads(task_payload)
-                else:
-                    task_dict = task_payload
-
                 # Mark operation as ready for re-processing
                 # Reset to pending with task_payload intact so worker picks it up again
                 async with self._backend.acquire() as conn:

hindsight-api-slim/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-api-slim"
-version = "0.8.0"
+version = "0.8.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -81,6 +81,10 @@ local-ml = [
     # Local ML models for embeddings/reranking
     "sentence-transformers>=3.3.0",
     "transformers>=4.53.0", # Security fixes for ReDoS vulnerabilities
+    # transformers (incl. latest 5.x) hard-requires tokenizers<=0.23.0 via a
+    # runtime check; without this cap an in-place upgrade can pull tokenizers
+    # 0.23.1 and break local embeddings/reranker startup. See issue #2055.
+    "tokenizers>=0.22.0,<=0.23.0",
     "torch>=2.6.0", # CVE fix for remote code execution
     "einops>=0.8.2",
     "flashrank>=0.2.0",
@@ -100,6 +104,7 @@ local-onnx = [
     # In-process ONNX Runtime embeddings without an Ollama/TEI sidecar
     "onnxruntime>=1.17.0",
     "transformers>=4.53.0",
+    "tokenizers>=0.22.0,<=0.23.0", # See issue #2055 (transformers caps tokenizers<=0.23.0)
     "huggingface-hub>=0.20.0",
     "numpy>=1.26.0",
 ]
@@ -182,23 +187,24 @@ dev = [
 [tool.ruff]
 line-length = 120
 target-version = "py311"
-exclude = [
-    "tests/",
-    "**/tests/",
-]
 
 [tool.ruff.lint]
+# Tests are formatted (via `ruff format`) but excluded from lint rules, which
+# are too noisy for test code (unused imports/vars, import ordering).
+exclude = [
+    "tests/**",
+    "**/tests/**",
+]
 select = [
     "E",      # pycodestyle errors
     "W",      # pycodestyle warnings
     "F",      # Pyflakes
     "I",      # isort
+    "B021",   # flake8-bugbear: f-string used as docstring (leaves __doc__ None)
 ]
 ignore = [
     "E501",   # line too long (handled by formatter)
     "E402",   # module import not at top of file
-    "F401",   # unused import (too noisy during development)
-    "F841",   # unused variable (too noisy during development)
     "F811",   # redefined while unused
     "F821",   # undefined name (forward references in type hints)
 ]

hindsight-api-slim/tests/conftest.py

@@ -1,6 +1,7 @@
 """
 Pytest configuration and shared fixtures.
 """
+
 import asyncio
 import os
 from pathlib import Path
@@ -15,6 +16,27 @@ from hindsight_api.engine.cross_encoder import LocalSTCrossEncoder
 from hindsight_api.engine.query_analyzer import DateparserQueryAnalyzer
 from hindsight_api.engine.task_backend import SyncTaskBackend
 from hindsight_api.pg0 import EmbeddedPostgres
+from hindsight_api.tracing import unregister_span_recorder
+
+
+async def _teardown_memory_engine(mem: MemoryEngine) -> None:
+    """Tear down a test MemoryEngine, guaranteeing its span recorder is unregistered.
+
+    LLM-trace recorders live in a process-global registry; ``MemoryEngine.close()`` is
+    the only thing that removes the engine's recorder from it. If close() is skipped
+    (pool already closing) or raises before that step, the recorder leaks and a later
+    test's LLM calls get recorded into the shared DB — the flaky
+    test_llm_trace::test_disabled_writes_no_rows (#2229). Unregister unconditionally;
+    it's a no-op when close() already did it.
+    """
+    try:
+        if mem._pool and not mem._pool._closing:
+            await mem.close()
+    except Exception:
+        pass
+    finally:
+        unregister_span_recorder(mem._llm_recorder)
+
 
 # Default pg0 instance configuration for tests
 DEFAULT_PG0_INSTANCE_NAME = "hindsight-test"
@@ -76,6 +98,7 @@ def pg0_db_url(db_url, tmp_path_factory, worker_id):
     if db_url and not _parse_pg0_url(db_url)[0]:
         # Plain postgresql:// URL - use it directly but still run migrations
         from hindsight_api.migrations import run_migrations
+
         run_migrations(db_url)
         return db_url
 
@@ -127,6 +150,7 @@ def pg0_db_url(db_url, tmp_path_factory, worker_id):
     # Run migrations - uses PostgreSQL advisory lock internally,
     # so safe to call from multiple workers (only one will actually run migrations)
     from hindsight_api.migrations import run_migrations
+
     run_migrations(url)
 
     # Clean up stale test data from previous sessions. Per-bank vector indexes
@@ -157,8 +181,7 @@ def _cleanup_stale_test_data(db_url: str) -> None:
         conn = await asyncpg.connect(db_url)
         try:
             idx_rows = await conn.fetch(
-                "SELECT indexname FROM pg_indexes "
-                "WHERE schemaname = 'public' AND indexname LIKE 'idx_mu_emb_%'"
+                "SELECT indexname FROM pg_indexes WHERE schemaname = 'public' AND indexname LIKE 'idx_mu_emb_%'"
             )
             if idx_rows:
                 for row in idx_rows:
@@ -166,10 +189,20 @@ def _cleanup_stale_test_data(db_url: str) -> None:
 
             # Truncate test data in dependency order
             for table in [
-                "entity_cooccurrences", "unit_entities", "memory_links",
-                "entities", "memory_units", "chunks", "documents",
-                "mental_models", "directives", "async_operations",
-                "audit_log", "webhooks", "file_storage", "banks",
+                "entity_cooccurrences",
+                "unit_entities",
+                "memory_links",
+                "entities",
+                "memory_units",
+                "chunks",
+                "documents",
+                "mental_models",
+                "directives",
+                "async_operations",
+                "audit_log",
+                "webhooks",
+                "file_storage",
+                "banks",
             ]:
                 try:
                     await conn.execute(f"TRUNCATE {table} CASCADE")
@@ -252,8 +285,7 @@ def oracle_db_url(_oracle_admin_dsn):
         # Create test user (idempotent — skip if already exists)
         try:
             cursor.execute(
-                f'CREATE USER {test_user} IDENTIFIED BY "{test_pass}" '
-                f"DEFAULT TABLESPACE USERS QUOTA UNLIMITED ON USERS"
+                f'CREATE USER {test_user} IDENTIFIED BY "{test_pass}" DEFAULT TABLESPACE USERS QUOTA UNLIMITED ON USERS'
             )
         except oracledb.DatabaseError as e:
             if hasattr(e.args[0], "code") and e.args[0].code == 1920:
@@ -331,10 +363,7 @@ async def oracle_memory(oracle_db_url, embeddings, cross_encoder, query_analyzer
         )
         await mem.initialize()
         yield mem
-        try:
-            await mem.close()
-        except Exception:
-            pass
+        await _teardown_memory_engine(mem)
     finally:
         # Restore original env var and clear config cache
         if old_backend is None:
@@ -420,13 +449,12 @@ def cross_encoder(tmp_path_factory, worker_id):
 
     return ce
 
+
 @pytest.fixture(scope="session")
 def query_analyzer():
     return DateparserQueryAnalyzer()
 
 
-
-
 @pytest_asyncio.fixture(scope="function")
 async def memory(pg0_db_url, embeddings, cross_encoder, query_analyzer):
     """
@@ -453,11 +481,7 @@ async def memory(pg0_db_url, embeddings, cross_encoder, query_analyzer):
     )
     await mem.initialize()
     yield mem
-    try:
-        if mem._pool and not mem._pool._closing:
-            await mem.close()
-    except Exception:
-        pass
+    await _teardown_memory_engine(mem)
 
 
 @pytest_asyncio.fixture(scope="function")
@@ -486,11 +510,7 @@ async def memory_real_llm(pg0_db_url, embeddings, cross_encoder, query_analyzer)
     )
     await mem.initialize()
     yield mem
-    try:
-        if mem._pool and not mem._pool._closing:
-            await mem.close()
-    except Exception:
-        pass
+    await _teardown_memory_engine(mem)
 
 
 @pytest_asyncio.fixture(scope="function")
@@ -517,8 +537,22 @@ async def memory_no_llm_verify(pg0_db_url, embeddings, cross_encoder, query_anal
     )
     await mem.initialize()
     yield mem
-    try:
-        if mem._pool and not mem._pool._closing:
-            await mem.close()
-    except Exception:
-        pass
+    await _teardown_memory_engine(mem)
+
+
+@pytest_asyncio.fixture
+async def api_client(memory):
+    """General-purpose HTTP test client over the `memory` fixture's app.
+
+    Use for any integration test that exercises the FastAPI surface without
+    needing audit-logging side effects. See `audit_api_client` for the
+    audit-enabled variant.
+    """
+    import httpx
+
+    from hindsight_api.api import create_app
+
+    app = create_app(memory, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        yield client

hindsight-api-slim/tests/e2e_oracle_smoke.py

@@ -23,6 +23,7 @@ from urllib.parse import urlparse
 # Helpers
 # ---------------------------------------------------------------------------
 
+
 def _log(step: int, total: int, msg: str) -> None:
     print(f"  [{step}/{total}] {msg}")
 
@@ -64,8 +65,7 @@ def _bootstrap_test_user(admin_dsn: dict[str, str]) -> str:
         # Create user (skip if already exists - ORA-01920)
         try:
             cursor.execute(
-                f'CREATE USER {test_user} IDENTIFIED BY "{test_pass}" '
-                f"DEFAULT TABLESPACE USERS QUOTA UNLIMITED ON USERS"
+                f'CREATE USER {test_user} IDENTIFIED BY "{test_pass}" DEFAULT TABLESPACE USERS QUOTA UNLIMITED ON USERS'
             )
         except oracledb.DatabaseError as e:
             if hasattr(e.args[0], "code") and e.args[0].code == 1920:
@@ -100,6 +100,7 @@ def _bootstrap_test_user(admin_dsn: dict[str, str]) -> str:
 # Main
 # ---------------------------------------------------------------------------
 
+
 async def _run() -> None:
     total_steps = 8
 
@@ -290,6 +291,7 @@ def main() -> int:
     except Exception as exc:
         print(f"\nFAILED: {exc}", file=sys.stderr)
         import traceback
+
         traceback.print_exc()
         return 1
 

hindsight-api-slim/tests/llm_judge.py

@@ -87,7 +87,7 @@ async def _judge_once(
             "content": (
                 "You are a test evaluation judge. Given a response and evaluation criteria, "
                 "determine whether the response meets the criteria. "
-                "Respond with JSON: {\"meets_criteria\": true/false, \"reasoning\": \"brief explanation\"}"
+                'Respond with JSON: {"meets_criteria": true/false, "reasoning": "brief explanation"}'
             ),
         },
         {
@@ -163,9 +163,7 @@ async def evaluate(
 
     if met > not_met:
         agreeing = next(v for v in verdicts if v.meets_criteria)
-        logger.info(
-            f"Judge: primary 'not met' overruled by majority ({met}/{len(verdicts)} met). Criteria: {criteria}"
-        )
+        logger.info(f"Judge: primary 'not met' overruled by majority ({met}/{len(verdicts)} met). Criteria: {criteria}")
         return JudgeVerdict(
             meets_criteria=True,
             reasoning=f"Majority of {len(verdicts)} judges met criteria (primary verdict overruled as noise). {agreeing.reasoning}",

hindsight-api-slim/tests/test_agents_api.py

@@ -1,6 +1,7 @@
 """
 Tests for agent management API (profile, disposition).
 """
+
 import pytest
 import uuid
 from hindsight_api import MemoryEngine, RequestContext
@@ -17,9 +18,7 @@ class TestAgentProfile:
     """Tests for agent profile management."""
 
     @pytest.mark.asyncio
-    async def test_get_bank_profile_no_auto_create_returns_none(
-        self, memory: MemoryEngine, request_context
-    ):
+    async def test_get_bank_profile_no_auto_create_returns_none(self, memory: MemoryEngine, request_context):
         """When create_if_missing=False is passed, a missing bank returns None
         rather than being silently auto-created. This is what read-only
         endpoints (HTTP GET, polling, etc.) must use to avoid creating banks
@@ -27,28 +26,20 @@ class TestAgentProfile:
         bank_id = unique_agent_id("test_no_auto_create")
 
         # First call with create_if_missing=False on a non-existent bank
-        result = await memory.get_bank_profile(
-            bank_id, request_context=request_context, create_if_missing=False
-        )
+        result = await memory.get_bank_profile(bank_id, request_context=request_context, create_if_missing=False)
         assert result is None, "Expected None for missing bank with create_if_missing=False"
 
         # Verify the bank was NOT created as a side effect
-        result_again = await memory.get_bank_profile(
-            bank_id, request_context=request_context, create_if_missing=False
-        )
+        result_again = await memory.get_bank_profile(bank_id, request_context=request_context, create_if_missing=False)
         assert result_again is None, "Bank must not exist after read-only call"
 
         # And explicit auto-create still works
-        created = await memory.get_bank_profile(
-            bank_id, request_context=request_context, create_if_missing=True
-        )
+        created = await memory.get_bank_profile(bank_id, request_context=request_context, create_if_missing=True)
         assert created is not None
         assert created["disposition"]["skepticism"] == 3
 
         # Now read-only call sees it
-        seen = await memory.get_bank_profile(
-            bank_id, request_context=request_context, create_if_missing=False
-        )
+        seen = await memory.get_bank_profile(bank_id, request_context=request_context, create_if_missing=False)
         assert seen is not None
         assert seen["disposition"]["skepticism"] == 3
 
@@ -122,11 +113,7 @@ class TestAgentEndpoint:
         bank_id = unique_agent_id("test_put_create")
 
         request = CreateBankRequest(
-            disposition=DispositionTraits(
-                skepticism=4,
-                literalism=5,
-                empathy=2
-            ),
+            disposition=DispositionTraits(skepticism=4, literalism=5, empathy=2),
         )
 
         profile = await memory.get_bank_profile(bank_id, request_context=request_context)
@@ -155,7 +142,7 @@ class TestAgentDispositionIntegration:
         disposition = {
             "skepticism": 5,  # Very skeptical
             "literalism": 4,  # High literalism
-            "empathy": 2,     # Low empathy
+            "empathy": 2,  # Low empathy
         }
         await memory.update_bank_disposition(bank_id, disposition, request_context=request_context)
 
@@ -163,7 +150,7 @@ class TestAgentDispositionIntegration:
             bank_id=bank_id,
             contents=[
                 {"content": "Traditional painting techniques have been used for centuries"},
-                {"content": "Modern digital art is changing the art world"}
+                {"content": "Modern digital art is changing the art world"},
             ],
             request_context=request_context,
         )

hindsight-api-slim/tests/test_bank_attribution.py

@@ -0,0 +1,331 @@
+"""
+Tests for per-bank provider cost attribution.
+
+Covers the opt-in `HINDSIGHT_API_LLM_SEND_BANK_AS_USER` plumbing that tags
+outbound OpenAI-compatible LLM and embedding calls with `user=<bank_id>`, the
+`_current_bank_id` engine ContextVar that carries the bank across the async call
+chain, and its propagation into the embedding executor thread.
+
+All deterministic — no network, stdlib/pytest only.
+"""
+
+import os
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, patch
+
+import pytest
+from pydantic import BaseModel
+
+from hindsight_api.engine.bank_attribution import apply_bank_attribution
+from hindsight_api.engine.embeddings import OpenAIEmbeddings
+from hindsight_api.engine.memory_engine import (
+    _bind_bank_id,
+    _current_bank_id,
+    get_current_bank_id,
+)
+from hindsight_api.engine.providers.openai_compatible_llm import OpenAICompatibleLLM
+from hindsight_api.engine.retain.embedding_utils import generate_embeddings_batch
+
+
+@pytest.fixture(autouse=True)
+def restore_send_bank_env():
+    """Save/restore the attribution env var and clear the cached config."""
+    from hindsight_api.config import clear_config_cache
+
+    original = os.environ.get("HINDSIGHT_API_LLM_SEND_BANK_AS_USER")
+    clear_config_cache()
+    yield
+    if original is None:
+        os.environ.pop("HINDSIGHT_API_LLM_SEND_BANK_AS_USER", None)
+    else:
+        os.environ["HINDSIGHT_API_LLM_SEND_BANK_AS_USER"] = original
+    clear_config_cache()
+
+
+def _set_flag(enabled: bool) -> None:
+    from hindsight_api.config import clear_config_cache
+
+    os.environ["HINDSIGHT_API_LLM_SEND_BANK_AS_USER"] = "true" if enabled else "false"
+    clear_config_cache()
+
+
+# ── ContextVar lifecycle ──────────────────────────────────────────────────────
+
+
+class TestBankContextVar:
+    def test_default_is_none(self):
+        assert get_current_bank_id() is None
+
+    def test_set_and_reset(self):
+        token = _current_bank_id.set("user-42")
+        try:
+            assert get_current_bank_id() == "user-42"
+        finally:
+            _current_bank_id.reset(token)
+        assert get_current_bank_id() is None
+
+    def test_reset_runs_even_on_exception(self):
+        """A finally-based reset must unwind the binding even when the body raises."""
+        token = _current_bank_id.set("user-boom")
+        try:
+            with pytest.raises(ValueError):
+                try:
+                    assert get_current_bank_id() == "user-boom"
+                    raise ValueError("boom")
+                finally:
+                    _current_bank_id.reset(token)
+        finally:
+            pass
+        assert get_current_bank_id() is None
+
+
+class TestBindBankIdDecorator:
+    """The engine binds the bank via @_bind_bank_id on recall/retain/batch/task methods."""
+
+    async def test_binds_named_arg_positional_and_keyword(self):
+        @_bind_bank_id()
+        async def op(bank_id: str, query: str) -> str | None:
+            return get_current_bank_id()
+
+        assert await op("user-pos", "q") == "user-pos"
+        assert await op(bank_id="user-kw", query="q") == "user-kw"
+        assert get_current_bank_id() is None
+
+    async def test_extracts_dict_key(self):
+        @_bind_bank_id("task_dict", key="bank_id")
+        async def op(task_dict: dict) -> str | None:
+            return get_current_bank_id()
+
+        assert await op({"bank_id": "user-task", "type": "consolidation"}) == "user-task"
+        assert await op({"type": "consolidation"}) is None
+        assert get_current_bank_id() is None
+
+    async def test_resets_on_exception(self):
+        @_bind_bank_id()
+        async def op(bank_id: str) -> None:
+            assert get_current_bank_id() == "user-boom"
+            raise ValueError("boom")
+
+        with pytest.raises(ValueError):
+            await op("user-boom")
+        assert get_current_bank_id() is None
+
+    async def test_non_string_value_binds_none(self):
+        @_bind_bank_id()
+        async def op(bank_id: object) -> str | None:
+            return get_current_bank_id()
+
+        assert await op(12345) is None
+
+
+# ── LLM provider: user injection ──────────────────────────────────────────────
+
+
+class _SimpleJson(BaseModel):
+    ok: bool
+
+
+def _llm() -> OpenAICompatibleLLM:
+    return OpenAICompatibleLLM(
+        provider="openai",
+        api_key="test-key",
+        base_url="https://example.test/v1",
+        model="gpt-4o-mini",
+    )
+
+
+def _chat_response(content: str = '{"ok": true}'):
+    choice = SimpleNamespace(
+        finish_reason="stop",
+        message=SimpleNamespace(content=content, tool_calls=None, refusal=None),
+    )
+    return SimpleNamespace(choices=[choice], usage=None, error=None)
+
+
+async def _call(llm: OpenAICompatibleLLM, create: AsyncMock):
+    llm._client.chat.completions.create = create
+    with patch("hindsight_api.engine.providers.openai_compatible_llm.get_metrics_collector"):
+        return await llm.call(
+            messages=[{"role": "user", "content": "ping"}],
+            max_retries=0,
+        )
+
+
+async def test_user_injected_when_flag_on_and_bank_set():
+    _set_flag(True)
+    llm = _llm()
+    create = AsyncMock(return_value=_chat_response())
+    token = _current_bank_id.set("user-7")
+    try:
+        await _call(llm, create)
+    finally:
+        _current_bank_id.reset(token)
+    assert create.call_args.kwargs["user"] == "user-7"
+
+
+async def test_user_not_injected_when_flag_off():
+    _set_flag(False)
+    llm = _llm()
+    create = AsyncMock(return_value=_chat_response())
+    token = _current_bank_id.set("user-7")
+    try:
+        await _call(llm, create)
+    finally:
+        _current_bank_id.reset(token)
+    assert "user" not in create.call_args.kwargs
+
+
+async def test_user_not_injected_when_bank_unset():
+    _set_flag(True)
+    llm = _llm()
+    create = AsyncMock(return_value=_chat_response())
+    # No bank bound in context.
+    assert get_current_bank_id() is None
+    await _call(llm, create)
+    assert "user" not in create.call_args.kwargs
+
+
+async def test_caller_set_user_is_not_overridden():
+    """The helper never clobbers a `user` the caller already placed in call_params."""
+    _set_flag(True)
+    # Simulate a caller-provided user via the centralized helper directly.
+    params = {"user": "explicit-user"}
+    token = _current_bank_id.set("user-7")
+    try:
+        apply_bank_attribution(params)
+    finally:
+        _current_bank_id.reset(token)
+    assert params["user"] == "explicit-user"
+
+
+async def test_user_injected_in_tool_calling_path():
+    """call_with_tools() builds its own call_params; attribution must reach it too."""
+    _set_flag(True)
+    llm = _llm()
+    tool_response = SimpleNamespace(
+        choices=[
+            SimpleNamespace(
+                finish_reason="stop",
+                message=SimpleNamespace(content="done", tool_calls=None, refusal=None, reasoning_content=None),
+            )
+        ],
+        usage=None,
+        error=None,
+    )
+    create = AsyncMock(return_value=tool_response)
+    llm._client.chat.completions.create = create
+    token = _current_bank_id.set("user-tools")
+    try:
+        with patch("hindsight_api.engine.providers.openai_compatible_llm.get_metrics_collector"):
+            await llm.call_with_tools(
+                messages=[{"role": "user", "content": "ping"}],
+                tools=[{"type": "function", "function": {"name": "noop", "parameters": {}}}],
+                max_retries=0,
+            )
+    finally:
+        _current_bank_id.reset(token)
+    assert create.call_args.kwargs["user"] == "user-tools"
+
+
+# ── Embeddings: user injection ─────────────────────────────────────────────────
+
+
+def _openai_embeddings() -> OpenAIEmbeddings:
+    emb = OpenAIEmbeddings(api_key="sk-test", model="text-embedding-3-small", batch_size=100)
+    emb._dimension = 1536
+    return emb
+
+
+def _fake_embed_client(captured: list[dict]):
+    def fake_create(**kwargs):
+        captured.append(kwargs)
+        n = len(kwargs["input"])
+        return SimpleNamespace(data=[SimpleNamespace(index=i, embedding=[0.0] * 1536) for i in range(n)])
+
+    return SimpleNamespace(embeddings=SimpleNamespace(create=fake_create))
+
+
+def test_embeddings_user_injected_when_flag_on_and_bank_set():
+    _set_flag(True)
+    emb = _openai_embeddings()
+    captured: list[dict] = []
+    emb._client = _fake_embed_client(captured)
+    token = _current_bank_id.set("user-emb")
+    try:
+        emb.encode(["hello"])
+    finally:
+        _current_bank_id.reset(token)
+    assert captured[0]["user"] == "user-emb"
+
+
+def test_embeddings_user_not_injected_when_flag_off():
+    _set_flag(False)
+    emb = _openai_embeddings()
+    captured: list[dict] = []
+    emb._client = _fake_embed_client(captured)
+    token = _current_bank_id.set("user-emb")
+    try:
+        emb.encode(["hello"])
+    finally:
+        _current_bank_id.reset(token)
+    assert "user" not in captured[0]
+
+
+def test_embeddings_user_not_injected_when_bank_unset():
+    _set_flag(True)
+    emb = _openai_embeddings()
+    captured: list[dict] = []
+    emb._client = _fake_embed_client(captured)
+    assert get_current_bank_id() is None
+    emb.encode(["hello"])
+    assert "user" not in captured[0]
+
+
+# ── Executor context propagation ──────────────────────────────────────────────
+
+
+class _BankCapturingBackend:
+    """Embeddings backend whose encode records the bank id visible at call time.
+
+    The real `generate_embeddings_batch` offloads encode to a thread via
+    run_in_executor; this verifies the bank ContextVar survives that thread hop.
+    """
+
+    dimension = 1
+
+    def __init__(self) -> None:
+        self.seen_bank_id: str | None = "UNSET"
+
+    def encode_documents(self, texts: list[str]) -> list[list[float]]:
+        self.seen_bank_id = get_current_bank_id()
+        return [[0.0] for _ in texts]
+
+    def encode_query(self, texts: list[str]) -> list[list[float]]:
+        return self.encode_documents(texts)
+
+
+async def test_executor_propagates_bank_contextvar_into_worker_thread():
+    backend = _BankCapturingBackend()
+    token = _current_bank_id.set("user-thread")
+    try:
+        vectors = await generate_embeddings_batch(backend, ["a", "b"], input_type="document")
+    finally:
+        _current_bank_id.reset(token)
+    assert backend.seen_bank_id == "user-thread"
+    assert len(vectors) == 2
+
+
+async def test_executor_length_validation_preserved():
+    """The 1:1 alignment guard must still fire after the context-aware offload."""
+
+    class _ShortBackend:
+        dimension = 1
+
+        def encode_documents(self, texts: list[str]) -> list[list[float]]:
+            return [[0.0]]  # one vector for two inputs
+
+        def encode_query(self, texts: list[str]) -> list[list[float]]:
+            return self.encode_documents(texts)
+
+    with pytest.raises(Exception, match="expected exact 1:1 alignment"):
+        await generate_embeddings_batch(_ShortBackend(), ["a", "b"], input_type="document")

hindsight-api-slim/tests/test_bank_attribution_config.py

@@ -0,0 +1,132 @@
+"""
+Config wiring for per-bank attribution and the configurable OpenRouter rerank URL.
+
+- HINDSIGHT_API_LLM_SEND_BANK_AS_USER (default off, opt-in bool)
+- HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL (default = previously hardcoded URL)
+
+Deterministic, no network.
+"""
+
+import os
+from dataclasses import fields
+from unittest.mock import patch
+
+from hindsight_api.config import DEFAULT_RERANKER_OPENROUTER_BASE_URL, HindsightConfig
+from hindsight_api.engine.cross_encoder import create_cross_encoder_from_env
+
+
+def _restore_env(saved: dict[str, str | None]) -> None:
+    from hindsight_api.config import clear_config_cache
+
+    for key, value in saved.items():
+        if value is None:
+            os.environ.pop(key, None)
+        else:
+            os.environ[key] = value
+    clear_config_cache()
+
+
+def _make_full_config(**overrides):
+    """Build a complete HindsightConfig from type-based defaults plus overrides.
+
+    Mirrors the helper in test_reranker_timeouts.py so we can exercise the
+    factory without touching real env/config.
+    """
+    defaults: dict = {}
+    for f in fields(HindsightConfig):
+        if f.type == "str":
+            defaults[f.name] = ""
+        elif f.type == "str | None":
+            defaults[f.name] = None
+        elif f.type == "int":
+            defaults[f.name] = 0
+        elif f.type == "int | None":
+            defaults[f.name] = None
+        elif f.type == "float":
+            defaults[f.name] = 0.0
+        elif f.type == "float | None":
+            defaults[f.name] = None
+        elif f.type == "bool":
+            defaults[f.name] = False
+        else:
+            defaults[f.name] = None
+    defaults.update(overrides)
+    return HindsightConfig(**defaults)
+
+
+class TestSendBankAsUserConfig:
+    def test_default_is_false(self):
+        from hindsight_api.config import clear_config_cache
+
+        saved = {"HINDSIGHT_API_LLM_SEND_BANK_AS_USER": os.environ.get("HINDSIGHT_API_LLM_SEND_BANK_AS_USER")}
+        os.environ.pop("HINDSIGHT_API_LLM_SEND_BANK_AS_USER", None)
+        clear_config_cache()
+        try:
+            assert HindsightConfig.from_env().llm_send_bank_as_user is False
+        finally:
+            _restore_env(saved)
+
+    def test_true_enables(self):
+        from hindsight_api.config import clear_config_cache
+
+        saved = {"HINDSIGHT_API_LLM_SEND_BANK_AS_USER": os.environ.get("HINDSIGHT_API_LLM_SEND_BANK_AS_USER")}
+        os.environ["HINDSIGHT_API_LLM_SEND_BANK_AS_USER"] = "true"
+        clear_config_cache()
+        try:
+            assert HindsightConfig.from_env().llm_send_bank_as_user is True
+        finally:
+            _restore_env(saved)
+
+    def test_one_enables(self):
+        from hindsight_api.config import clear_config_cache
+
+        saved = {"HINDSIGHT_API_LLM_SEND_BANK_AS_USER": os.environ.get("HINDSIGHT_API_LLM_SEND_BANK_AS_USER")}
+        os.environ["HINDSIGHT_API_LLM_SEND_BANK_AS_USER"] = "1"
+        clear_config_cache()
+        try:
+            assert HindsightConfig.from_env().llm_send_bank_as_user is True
+        finally:
+            _restore_env(saved)
+
+
+class TestRerankerOpenRouterBaseUrlConfig:
+    def test_default_matches_previously_hardcoded_url(self):
+        from hindsight_api.config import clear_config_cache
+
+        saved = {
+            "HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL": os.environ.get("HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL")
+        }
+        os.environ.pop("HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL", None)
+        clear_config_cache()
+        try:
+            config = HindsightConfig.from_env()
+            assert config.reranker_openrouter_base_url == DEFAULT_RERANKER_OPENROUTER_BASE_URL
+            assert config.reranker_openrouter_base_url == "https://openrouter.ai/api/v1/rerank"
+        finally:
+            _restore_env(saved)
+
+    def test_env_override_is_read(self):
+        from hindsight_api.config import clear_config_cache
+
+        saved = {
+            "HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL": os.environ.get("HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL")
+        }
+        os.environ["HINDSIGHT_API_RERANKER_OPENROUTER_BASE_URL"] = "https://gateway.internal/v1/rerank"
+        clear_config_cache()
+        try:
+            assert HindsightConfig.from_env().reranker_openrouter_base_url == "https://gateway.internal/v1/rerank"
+        finally:
+            _restore_env(saved)
+
+    def test_factory_threads_configured_base_url_into_cross_encoder(self):
+        """create_cross_encoder_from_env() honors the configured OpenRouter rerank URL."""
+        config = _make_full_config(
+            reranker_provider="openrouter",
+            reranker_openrouter_api_key="k",
+            reranker_openrouter_model="cohere/rerank-v3.5",
+            reranker_openrouter_base_url="https://gateway.internal/v1/rerank",
+            reranker_openrouter_timeout=60.0,
+        )
+        with patch("hindsight_api.config.get_config", return_value=config):
+            encoder = create_cross_encoder_from_env()
+        assert encoder.base_url == "https://gateway.internal/v1/rerank"